Troubleshooting the FUSE-Based POSIX Client

Explains how to enable and collect the stack trace to troubleshoot POSIX client issues.

This section contains information for troubleshooting the FUSE-based POSIX client.

Enabling Traces

To enable traces at system startup, set the property fs.mapr.trace in the core-site.xml file. For example:

<property>
   <name>fs.mapr.trace</name>
   <value>DEBUG | INFO | WARN | ERROR | CRITICAL | OFF</value>
   <description> </description>
</property>

Collecting the Stack Trace

If the mountpoint is not responding or if the file system operations are taking too much time, collect the stack trace of all the threads to debug. To collect the stack trace of all threads, run the following command:

gstack <fuse-process-id> > ./gstack.log

If the file system commands fail, repeat the file system command with strace and collect the log file:

strace <file system command> > ./strace.log

Issue: FUSE POSIX client does not honor the MAPR_SUBNETS environment variable

The FUSE POSIX client may not honor the MAPR_SUBNETS environment variable on client nodes. When you set MAPR_SUBNETS on a client node to restrict FUSE traffic to a specific network interface (for example, 192.168.4.0/24), the POSIX client may continue to use both NICs to communicate with data nodes, instead of restricting to the specified subnet only.

Cause

The MAPR_SOURCE_HONOUR_SUBNETS flag controls source‑side filtering of client connections based on the subnets configured in MAPR_SUBNETS. When this flag is set to true on the client, the FUSE/POSIX client attempts to use only those local interfaces whose IPs fall within the CIDR ranges listed in MAPR_SUBNETS. If MAPR_SUBNETS is incomplete or incorrect, or if MAPR_SOURCE_HONOUR_SUBNETS is not set as expected, source IP selection may not match the intended storage network design.

Resolution:

A fix has been implemented at the RPC layer to enforce source IP selection on the client side based on MAPR_SUBNETS. The logic works as follows:
  • When MAPR_SUBNETS is not set (default), it uses INADDR_ANY (the kernel selects the interface).
  • When one or more subnets are specified in MAPR_SUBNETS, it uses the source IP of the client node matching that subnet mask.
NOTE

  • Every subnet listed in MAPR_SUBNETS must be routable from all cluster nodes. If any data node lacks a route to the client's source subnet, write operations may hang or fail. Verify routing on all nodes before enabling.

  • When an interface listed in MAPR_SUBNETS goes down on a FUSE client, ongoing I/O may stall for several minutes before failing over to another interface. The delay occurs because existing TCP connections continue retrying until kernel timeouts expire, which is controlled largely by net.ipv4.tcp_retries2, and additional time is required for RPC reconnect and FUSE cidcache refresh. Users requiring faster failover should tune TCP retry settings appropriately.

Configuration Required

To enable the fix, export these environment variables on the client node using the FUSE init script:
Variable Purpose
MAPR_SUBNETS Specifies the subnet(s) the client should use, comma separated (for example, "196.171.4.0/24,172.23.46.0/24")
MAPR_SOURCE_HONOUR_SUBNETS Must be enabled (set to true) to activate client-side subnet enforcement. By default it is disabled which means the kernel selects the interface for communication regardless of MAPR_SUBNETS.
Example:
# vim /opt/mapr/initscripts/mapr-fuse
# Function to mount the filesystem, if not already mounted.
do_start()
{
  export MAPR_SUBNETS="192.168.4.0/24"
  export MAPR_SOURCE_HONOUR_SUBNETS="true"