Troubleshooting the Development Environment for HPE Data Fabric

This section describes problems you might encounter when deploying, running, and accessing the Development Environment for HPE Data Fabric. It also includes steps to troubleshoot and resolve the problems.

Cluster Does Not Come Up

Problem
The cluster does not come up, and maprcli dump cldbstate -json might return an error saying that CID1 is waiting to become master.
Possible Cause
Stale processes. The error can be seen on Linux nodes if any mapr- processes are running when you run the mapr_dev_sandbox_container.sh script.
Solution
Before running the mapr_dev_sandbox_container.sh script, kill any running mapr- processes.

MAPR_EXTERNAL Error

Problem
Running maprcli commands can return an error such as MAPR_EXTERNAL: Empty string found in the output of line 1 on the Docker container.
Possible Cause
The default host network interface of en() doesn't exist on the node where the script is run.
Solution
Specify a host network interface other than en() by using the -nwinterface option as described in Running the Development Environment Script.

Docker Login Problems

Problem
Attempting to log in to your Docker container returns the following error:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!    @
                @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Possible Cause

You have an old ssh key in your .ssh/known_hosts file.

Solution
Replace the old ssh key with the correct key:
ssh-keygen -R [localhost]:2222

Docker Failures

Problem

Docker fails to run the container.

Possible Cause

Docker encounters problems starting ZooKeeper or Warden.

Solution
  1. Determine your Docker <container-id> by examining the output of the docker ps command.
  2. Examine the Docker log files by running:
    docker logs <container-id>
  3. Examine the HPE Data Fabric log files specified in the output for further diagnostics. You need to log in to the container to see those files.
Problem
Docker completes its startup as shown by the following output from Docker logs:
This container IP : 172.17.0.2

But Docker is killed before the cluster processes are running.

Possible Cause

You have not allocated enough memory to Docker.

Solution
Make sure you have configured Docker with at least 25 GB of memory as described at Step 2c at Prerequisites to Running the Development Environment for HPE Data Fabric.

Connection Problems

Problem

Unable to connect to Control System in your browser.

Possible Cause Solution
The AdminApplication process is not running yet. Run jps and wait for AdminApplication to appear in the list of running java processes.
You are accessing an older, cached copy of the Control System URL. Clear your browser cache and retry connecting to the URL.

Unable to Access HPE Data Fabric Database Table

Problem
You cannot access a HPE Data Fabric Database table.
Possible Cause
You do not have permissions on the volume where the table is stored.
Solution
When creating a volume, make sure you set up the user access controls appropriately. See Creating a Volume for details.

Unable to run OJAI Queries Due to Query Service Errors

Problem
When running an OJAI query, you encounter an error indicating that the Query Service is not enabled.
Possible Cause
The Container for Developers is setup with only users mapr and root. You are running as some other user and your query requires the OJAI Distributed Query Service.
Solution
Add your user name and group to the container by following the instructions at Setting up New Users.

NFS server not coming up

Problem
The NFS server does not come up when the sand box was configured.
Possible Cause
CLDB is not completely up and running when the NFS server is coming up.
Solution
Start the NFS server manually by running the following command:
maprcli node services -nfs start -nodes `hostname -f` -json

CLDB takes a long time to come up initially and shuts down intermittently

Problem
CLDB takes a long time to come up initially and shuts down intermittently, typically on Mac machines.
Possible Cause
Sufficient memory is not allotted to Docker.
Solution
Restart warden and wait for about 10 minutes.Run the following command to restart warden:
service mapr-warden restart