Apache Kafka Wire Protocol Service

HPE Ezmeral Data Fabric Streams supports Apache Kafka Wire Protocol Service. Apache Kafka Wire Protocol Service is a TCP/IP service that emulates a Kafka cluster backed by HPE Ezmeral Data Fabric Streams. The service makes it possible for Apache Kafka clients written in any programming language to access topics in HPE Ezmeral Data Fabric Streams.

Before Apache Kafka Wire Protocol Service was introduced, you had to use a modified version of the open-source libraries (kafka-clients.jar and librdkafka.so) for user applications to connect and publish/subscribe messages to topics in HPE Ezmeral Data Fabric Streams. With Apache Kafka Wire Protocol Service, you can simply use the Apache Kafka client libraries.

Apache Kafka Wire Protocol Service provides the following benefits:
  • User applications can connect, publish, and subscribe to HPE Ezmeral Data Fabric Streams topics using standard Apache Kafka client libraries. Starting with data access gateway 6.0, only topics created through Apache Kafka Wire Protocol Service are supported.
  • User applications developed using Apache Kafka clients do not require any modification to work with this new system, including recompilation, reconfiguration, or dependency management.
  • Supports available Apache Kafka clients in multiple programming languages, both from the Apache Software Foundation and the community.

For more information, see Getting Started with Apache Kafka Wire Protocol Service.

Architecture

The Apache Kafka Wire Protocol Service works within DAG (data access gateway) to act as a Kafka Broker (server in an Apache Kafka cluster). For all application purposes, the HPE Ezmeral Data Fabric cluster (with DAG and Apache Kafka Wire Protocol Service) runs like an Apache Kafka cluster.

The following image shows the communication paths between Apache Kafka clients and HPE Ezmeral Data Fabric with DAG and Apache Kafka Wire Protocol Service acting as brokers:

Monitoring

DAG is not a Kafka Broker; therefore, Broker metrics are not available. However, you can monitor Data Fabric core services, as described in Using HPE Ezmeral Data Fabric Monitoring (Spyglass Initiative).

Limitations

The following list describes unsupported functionality in the initial release of Apache Kafka Wire Protocol Service:
  • Quotas
  • Transactions
  • Idempotent Producer
  • Kafka ACL-based security
  • Topic-level security using ACEs.
  • Group ACEs for Consumer Groups
  • Delegation tokens
  • Log directories