Kafka JMX Metrics

Starting in EEP 9.0.0, you can enable metrics collection for Kafka consumers and producers. When enabled, JMX collects Kafka producer and consumer metrics from client applications. You can view the metrics through the JConsole UI or JMXTerm CLI.

Metrics Collected

When you enable Kafka JMX metrics, you can view the following producer and consumer metrics through the JConsole UI or JMXTerm CLI :
Producer Metrics Collected
Global metrics collected:
  • record-send-rate
  • record-send-total
  • record-error-rate
  • record-error-total
  • record-size-max
  • record-size-avg
    NOTE
    Record size is computed as the sum of topic length, key size, value size, and sizes of keys and values of all headers if any exist.
Per-topic metrics collected:
  • record-send-rate
  • record-send-total
  • byte-rate
  • byte-total
  • record-error-rate
  • record-error-total
Consumer Metrics Collected
  • fetch-size-avg
  • fetch-size-max
  • bytes-consumed-rate
  • bytes-consumed-total
  • records-consumed-rate
  • records-consumed-total
NOTE
This list of supported metrics is a subset of all the Kafka JMX metrics. The unsupported metrics are either not registered or have default values such as 0 or NaN. A full list of Kafka JMX metrics is available here.

Enabling Metrics Collection

To enable metrics collection, set metrics.enabled=true in the producer and consumer configuration.

The following sections provide steps to enable metrics collection and view metrics in the JConsole UI and JMXTerm CLI.

Using the JConsole UI to View Metrics

Complete the following steps to enable metrics collection and view metrics in the JConsole UI:
  1. Start the console producer with metrics collection enabled:
    bin/kafka-console-producer.sh --topic /s:t --producer-property metrics.enabled=true
  2. Enter and send data.
  3. In another terminal, run the following command to start JConsole:
    jconsole
  4. In JConsole, select the console producer process and connect to it.
  5. Select Insecure Connection.
  6. In the MBeans section, look at kafka.producer or kafka.producer domain, and select the MBean of interest to view its attributes.
  7. Click Refresh to update attribute values.

Using the JMXTerm CLI to View Metrics

JMXTerm is the CLI analog of JConsole. To download JMXTerm, go to https://docs.cyclopsgroup.org/jmxterm.

Complete the following steps to enable metrics collection and view metrics through the JMXTerm CLI:

View Producer Metrics
  1. Start the console producer with metrics collection enabled:
    bin/kafka-console-producer.sh  --topic /s:t --producer-property metrics.enabled=true
  2. Run the following command to start JMXTerm in interactive mode:
    java -jar jmxterm-1.0.2-uber.jar
  3. Run the jvms command to identify which process is running the Kafka console producer:
    jvms  

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.cyclopsgroup.jmxterm.utils.WeakCastUtils$2 (file:/home/mapr/jmxterm-1.0.2-uber.jar) to method sun.tools.jconsole.LocalVirtualMachine.getAllVirtualMachines()
WARNING: Please consider reporting this to the maintainers of org.cyclopsgroup.jmxterm.utils.WeakCastUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
4113     ( ) - com.mapr.admin.AdminApplication
18946    (m) - org.apache.hadoop.yarn.server.nodemanager.NodeManager
3045     (m) - org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
23286    ( ) - org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/mapr/zookeeper/zookeeper-3.5.6/conf/zoo.cfg
29545    (m) - com.mapr.fs.cldb.CLDB /opt/mapr/conf/cldb.conf
4090     ( ) - jmxterm-1.0.2-uber.jar
26491    ( ) - com.mapr.warden.WardenMain /opt/mapr/conf/warden.conf
13003    (m) - org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
2286     (m) - kafka.tools.ConsoleProducer --topic /s:t --producer-property metrics.enabled=true
    NOTE
    In this example, the process is 2286. On your system, the process number may differ. Use the process number provided by your system.
  4. Run the open command with the process number to open the connection:
    open 2286
  5. Run the following command to set the domain to kafka.producer:
    domain kafka.producer
  6. Run the beans command to see the beans available in the kafka.producer domain:
    beans

#domain = kafka.producer:
kafka.producer:client-id=console-producer,topic="/s:t",type=producer-topic-metrics
kafka.producer:client-id=console-producer,type=kafka-metrics-count
kafka.producer:client-id=console-producer,type=producer-metricseans
  7. Set the bean to kafka.producer:client-id=console-producer,type=producer-metrics:
    bean kafka.producer:client-id=console-producer,type=producer-metrics
  8. Run the following command to get all the metrics for the kafka.producer:client-id=console-producer,type=producer-metrics bean:
    get *

#mbean = kafka.producer:client-id=console-producer,type=producer-metrics:
record-send-rate = 0.0;
record-retry-total = 0.0;
record-size-avg = NaN;
batch-split-total = 0.0;
record-queue-time-avg = NaN;
request-latency-avg = NaN;
record-error-total = 0.0;
batch-split-rate = 0.0;
record-error-rate = 0.0;
record-send-total = 3.0;
batch-size-max = NaN;
compression-rate-avg = NaN;
record-queue-time-max = NaN;
record-retry-rate = 0.0;
request-latency-max = NaN;
record-size-max = NaN;
batch-size-avg = NaN;
records-per-request-avg = NaN;
View Consumer Metrics

  1. Start the console consumer with metrics collection enabled:

    bin/kafka-console-consumer.sh  --topic /s1:t1 --consumer-property metrics.enabled=true  --from-beginning
  2. Run the following command to start JMXTerm in interactive mode:
    java -jar jmxterm-1.0.2-uber.jar
  3. Run the jvms command to identify which process is running the Kafka console consumer:
    jvms

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.cyclopsgroup.jmxterm.utils.WeakCastUtils$2 (file:/root/jmxterm-1.0.2-uber.jar) to method sun.tools.jconsole.LocalVirtualMachine.getAllVirtualMachines()
WARNING: Please consider reporting this to the maintainers of org.cyclopsgroup.jmxterm.utils.WeakCastUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
1408085  ( ) - com.mapr.data.DataAccessGatewayApplication
1404215  ( ) - com.mapr.fs.gateway.Gateway /opt/mapr/conf/gateway.conf
1328215  (m) - com.mapr.fs.cldb.CLDB /opt/mapr/conf/cldb.conf
1454674  (m) - org.apache.hadoop.yarn.server.nodemanager.NodeManager
1454739  (m) - org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
753251   ( ) - org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/mapr/zookeeper/zookeeper-3.8.3/conf/zoo.cfg
1385949  ( ) - com.mapr.fs.mastgateway.MASTGateway /opt/mapr/conf/mastgateway.conf
1310589  ( ) - com.mapr.warden.WardenMain /opt/mapr/conf/warden.conf
1536163  ( ) - jmxterm-1.0.2-uber.jar
1389379  (m) - org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
1387245  (m) - org.apache.drill.exec.server.Drillbit
1404556  ( ) - com.mapr.admin.AdminApplication
1531018  ( ) - kafka.tools.ConsoleConsumer --topic /s1:t1 --consumer-property metrics.enabled=true --from-beginning
  4. Run the open command with the process number to open the connection:
    NOTE
    In this example, the process is 1531018. On your system, the process number may differ. Use the process number provided by your system.
    open 1531018
  5. Run the following command to set the domain to kafka.consumer:
    domain kafka.consumer
  6. Run the beans command to see the beans available in the kafka.consumer domain:

    #domain = kafka.consumer:
 kafka.consumer:client-id=console-consumer,type=consumer-fetch-manager-metrics
 kafka.consumer:client-id=console-consumer,type=consumer-metrics
 kafka.consumer:client-id=console-consumer,type=kafka-metrics-count
  7. Set the bean to kafka.consumer:client-id=<clientid>,type=consumer-fetch-manager-metrics: 
     bean kafka.consumer:client-id=console-consumer,type=consumer-fetch-manager-metrics

  8. Run the following command to get all the metrics for the kafka.consumer:client-id=consumer-console,type=consumer-fetch-manager-metrics bean:

    get *
    #mbean = kafka.consumer:client-id=console-consumer,type=consumer-fetch-manager-metrics:
    bytes-consumed-rate = 0.0;
    fetch-latency-max = NaN;
    fetch-rate = 0.0;
    fetch-total = 0.0;
    fetch-throttle-time-max = NaN;
    fetch-size-max = NaN;
    records-consumed-total = 0.0;
    records-lead-min = NaN;
    bytes-consumed-total = 0.0;
    fetch-latency-avg = NaN;