HPE Ezmeral Data Fabric Streams Telemetry Metrics

This section describes the available HPE Ezmeral Data Fabric Streams telemetry metrics.

Starting with HPE Ezmeral Data Fabric Release 7.10, HPE Ezmeral Data Fabric Streams supports telemetry metrics. Telemetry data can be collected and analyzed to gain insights into system performance. You can monitor and analyze streams performance in real-time to prevent potential performance bottle necks and suboptimal resource utilization. With detailed insights of specific topics/partitions and streams, you can detect anomalies and promptly resolve issues.

Every 10 seconds, the collectd service uses a plugin to gather the Streams Telemetry metrics on the cluster, node, streams and topic-partition level.

Streams telemetry is integrated with the Spyglass metrics workflow. See Spyglass on Streams. The Streams telemetry metrics can be queried by a visualization tool, for example Grafana.
NOTE
Integration with Open Telemetry is planned for a future release. At that time an export plugin will be added.

The following table lists HPE Ezmeral Data Fabric Streams telemetry metrics:

Metric Type Name Description
Producer Metrics - Topic-partition level1 mapr.stream.producer.nfputs The number of failed puts.
mapr.stream.producer.nputs The number of put requests.
mapr.stream.producer.nrecs The number of records.
mapr.stream.producer.psize_avg The average put size.
mapr.stream.producer.psize_max The maximum put size.
mapr.stream.producer.psize_min The minimum put size.
mapr.stream.producer.psize_tot The put size total.
mapr.stream.producer.ravgsize The average record size.
Producer Metrics - Node level mapr.stream.producer.ibytes In bytes.
mapr.stream.producer.iotime_avg The average wait time in reading the put request data from the socket.
mapr.stream.iotime_max The maximum wait time in reading the put request data from the socket.
mapr.stream.iotime_min The minimum wait time in reading the put request data from the socket.
mapr.stream.producer.latency_avg The average latency of put requests.
mapr.stream.producer.latency_max The maximum latency of put requests.
mapr.stream.producer.latency_min The minimum latency of put requests.
mapr.stream.producer.nbatches The number of batches in the put request.
mapr.stream.producer.nfreqs The number of failed put requests.
mapr.stream.producer.nreqs The number of put requests.
Consumer Metrics - Topic-Partition level1 mapr.stream.consumer.nfetches The number of fetches.
mapr.stream.consumer.nrecs The number of records.
mapr.stream.consumer.ravgsize The average record size.
mapr.stream.consumer.fsize_avg The average fetch size.
mapr.stream.consumer.fsize_max The maximum fetch size.
mapr.stream.consumer.fsize_min The minimum fetch size.
mapr.stream.consumer.fsize_tot The fetch size total.
Consumer Metrics - Node level mapr.stream.consumer.obytes Out bytes.
mapr.stream.consumer.iotime_avg The average time to write the data to the socket.
mapr.stream.consumer.iotime_max The maximum time to write the data to the socket.
mapr.stream.consumer.iotime_min The minimum time to write the data to the socket.
mapr.stream.consumer.latency_avg The average latency of fetch requests.
mapr.stream.consumer.latency_max The maximum latency of fetch requests.
mapr.stream.consumer.latency_min The minimum latency of fetch requests.
mapr.stream.consumer.nfresps The number of fetches.
mapr.stream.consumer.nresps The number of responses.
1 Contains the following tags:
  • hostname
  • stream_fid
  • producer_id
  • topic_name
  • partition_id
NOTE
In Release 7.10, producer_id and partition_id are not added as tags.

Enabling Metrics Collection

To enable metrics collection, set the configuration value for mfs.enable.streams.metrics. 
maprcli config save -values {"mfs.enable.streams.metrics":"1"} -json