Troubleshooting Object Store
Provides methods for troubleshooting issues in Object Store.
Before You Troubleshoot
Verify that Object Store is properly installed and enabled, as described in Installing HPE Ezmeral Data Fabric Object Store and Enabling the HPE Ezmeral Data Fabric Object Store. Enabling the HPE Ezmeral Data Fabric Object Store includes several important steps required to use Object Store successfully, including steps for setting up certificates. If certificates are not properly configured, applications cannot access Object Store.
You can also perform the following pre-troubleshooting verification checks:
- Check access to the Object Store UI
- The Object Store UI is the Object Store entry point. If Object Store is installed
and running, you should be able to access the Object Store UI from the MCS (management
control system). Go to
https://<node-ip-address>:8443/app/mcs/#/app/login
and log in. Click on the Data tab and look for Object Store in the dropdown. If you see Object Store in the dropdown, Object Store is installed and running. If you do not see Object Store in the dropdown, the CLDB S3server quorum is not properly set up or the quorum has not finished setting up. - Check the status of the CLDB and S3 server quorum
- To check the status of the CLDB and S3 server quorum, run
maprcli dump cldbstate -json
. In the output, look fors3Info
. S3Info contains the status of all S3 servers.- When the status of all S3 servers is running, you should be able to access Object Store through the Object Store UI. If you followed all the instructions in Enabling the HPE Ezmeral Data Fabric Object Store, it may just take a bit more time for the status to change.
- If the
s3State
isAWAITING_FEATURE_ENABLE
, restart the CLDB service. See node services.
- Verify that users have permission to log in to Object Store.
- Before a user that is listed in LDAP/AD can access Object Store, the cluster
administrator (typically the
mapr
user) must first give the user permission to log in. In the MCS go to Admin > User Settings and click on the Permissions tab. Add the user and assign Login permission to the user. Click Save Changes when done.
Logging
- MOSS
- CLDB S3 server module
- MSI (interface module between MOSS and the file system)
Log File | Description |
---|---|
moss.log |
|
moss.fileclient.log |
|
moss.out |
|
cldb.log |
|
mfs.log |
|
Debugging
You can debug MOSS with the mc admin profile
command or DNU/GDB debugger.
mc
commands, the
/opt/mapr/conf/ca/chain-ca.pem
file must be copied to
~/.mc/certs/CAs/
on the node running mc
. Also, a
symbolic link must be created in the user directory. To create the symbolic link for a user,
run:su - <user>
mkdir -p ~/.mc/certs/CAs
ln -s /opt/mapr/conf/ca/chain-ca.pem ~/.mc/certs/CAs/chain-ca.pem
- MOSS Profile
- The
mc admin profile
command returns information about the MOSS thread activities. Run thestart
command, wait a few seconds and then run thestop
command. The command outputs a zip file that you can unzip to access text files. View (vim or cat) the text files to see the activity of the MOSS threads.Run the
mc admin profile
command, as shown:/opt/mapr/bin/mc admin profile start --type goroutines mapralias /opt/mapr/bin/mc admin profile stop mapralias
- Debug with DNU/GDB Debugger
- Running the debugger is helpful if MOSS crashes. Run the debugger, as shown in the following example:
gdb /opt/mapr/server/moss <moss/core/path>
Debugging Bucket Metrics
Bucket metrics provide you with account-level and bucket-level statistics, such as the total size of an account, the total object count, historical usage of buckets, and so on. The MOSS server includes an SRM (storage recovery metrics) component that automatically recovers metrics for buckets, updates statistics, and reclaims space when any issues occur; for example, if a put operation does not complete. SRM loads metrics into the BucketList table.
The BucketList table is the source of truth for statistics; it provides the last time a
bucket recovery occurred. You can access the BucketList table through the mc
lb
and mc stat
commands or in the Object Store UI. You can also
get stats from olt statsfid
when you run the mrconfig s3
bucketstats
command.
Interface | Description |
---|---|
Object Store UI |
|
mc
stat |
Returns bucket-level statistics. |
mc lb
|
|
mrconfig
s3 bucketstats |
Returns statistics from OLT StatsFid for a given bucket. |
- Resolving Issues with Bucket Metrics
- The following issues could result in inaccurate metrics. If the solutions provided do not resolve the issue, you can manually trigger recovery. Recovery is performed by the MOSS server on the node where the master copy of the OLT table's first tablet is hosted.
Debugging Volumes
Every account is associated with a volume and every bucket is associated with a volume. An account is associated with a root volume, which stores metadata for the account, including users, groups, and policies. Bucket volumes store data and metadata.
- If a container is not accessible.
- You need to run fsck.
- An offload fails, which would trigger an offload failure in the UI.
- Find the Volume Name and Details
- Working with Bucket Volumes provides instructions for finding the name of a volume and viewing volume details.
- Delete a Volume to Reclaim Space
- If you want to reclaim space, you can remove objects; however, deleting objects is a
time-consuming process. Instead, you may prefer to delete the volume if deleting all
buckets in the volume is feasible.
To delete the root volume for an account, you must delete the account. When the account is deleted, the root volume is automatically deleted.
To delete a bucket volume, run:maprcli volume remove -deletes3bucket true -name <volumeName>
CAUTIONThis command only works on volumes where all buckets are non-worm buckets. Running this command against a volume with worm buckets fails.
Additional Tips
- Check if a bucket exists
- To see if a bucket exists, check in the MOSS file client log by
running:
grep -nira "<bucketName> /opt/mapr/logs/moss.fileclient.log | more
- Check if a bucket is created
- To see if a bucket is created, you can check the mfs log by
running:
The output will state S3BucketCreateFinish.grep -nir "<bucketName>" /opt/mapr/logs/mfs.log-3
- Get the container ID for a bucket
- The request to create a bucket goes to the CLDB and the CLDB creates a volume for
the bucket. You can look at the CLDB log to gather data for troubleshooting, such as
the container ID, by
running:
grep -nira "<volumeName>" /opt/mapr/logs/cldb.log
You can dump the container to get additional details, such as the file server node IP address, for example:TIPWorking with Bucket Volumes provides instructions for finding the name of a volume associated with a bucket.maprcli dump containerinfo -ids <containerID> -json
- Issues running S3 cmd
- If the system returns an error similar to the following when you run S3cmd, such as
S3cmd ls s3://
:ERROR: SSL certificate verification failure: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
Run the following command to resolve the error:s3cmd -ca-certs=/opt/mapr/conf/ca/chain-ca.pem ls s3://
When you run this command, the system returns an error about the access key and prompts you for the keys. To add an access key, run:s3cmd -configure
And follow the instructions at the prompt.
- Issues running AWS
- If you try to run the following AWS
command:
aws s3 ls s3:// --endpoint-url https://<hostname>:9000
You may get an error similar to the following:SSL validation failed for https://<hostname>:9000/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)
To resolve the error, point the aws configuration file to the/opt/mapr/conf/ca/chain-ca.pem
directory or export it, as shown:export AWS_CA_BUNDLE=/opt/mapr/conf/ca/chain-ca.pem
If that does not work, run:aws configure