HPE Ezmeral Data Fabric Monitoring Tips and Troubleshooting
Lists the nuances of monitoring clusters.
Monitoring a Secure Cluster
- After regenerating the HPE Ezmeral Data Fabric user ticket, service failures
occur for
collectd
and OpenTSDB - If you delete or regenerate the HPE Ezmeral Data Fabric user ticket, the running
collectd
and OpenTSDB services will fail. After updating the HPE Ezmeral Data Fabric user ticket, restartcollectd
and OpenTSDB services.
Monitoring Logs
- I notice a sudden increase in
fluentd
logs. What can I do? - A sudden increase in the log file for
fluentd
could mean that a feedback loop is occurring wherefluentd
logs an error in the log file for afluentd
issue and that log entry causes yet another error whenfluentd
tries to parse it. In this case, consider disabling the index offluentd
logs. See Configure Logs to Index.
- I see "400 - Rejected by Elasticsearch" messages in the fluentd logs. What can I do?
- Messages such as the following can accumulate in the
fluentd
log when a process does not produce logs with valid UTF-8 output:2019-04-25 17:00:11 -0700 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Elasticsearch ErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil after setting this option in es_config.conf
Monitoring Metrics
- Where should I store the Elasticsearch index?
- Elasticsearch requires a lot of disk space. Also, when you upgrade Elasticsearch, the
default index directory is removed along with the package update. Therefore, it is
recommended to configure a separate filesystem for the index data. It is not recommended
to store index data under the
/
or the/var
filesystem.NOTEIf you store the Elasticsearch index on a filesystem that is locally hosted, you will be able to access logs in the event that the HPE Ezmeral Data Fabric cluster is not available. - I see a "Bad Request" error message for my HPE Ezmeral Data Fabric Database metrics? What can I do?
- If you have more than 1000 active tables in HPE Ezmeral Data Fabric Database and
the HPE Ezmeral Data Fabric monitoring request size to OpenTSDB is more than 4 KB, you may see
the following error message:
You can increase the maximum request size of OpenTSDB to up to 64 KB by setting the following parameters in the"Sorry but your request was rejected as being invalid. The reason provided was: Chunked request not supported."
opentsdb.conf
file:tsd.http.request.enable_chunked=true tsd.http.request.max_chunk=65536