Home
7.3 Development
This section contains information related to application development for Ezmeral ecosystem components and HPE Ezmeral Data Fabric products, including the file system, Database (Key-Value and JSON), and Event Streams.
Apache Kafka Wire Protocol Service
Starting in EEP 9.0.0, HPE Ezmeral Data Fabric Streams supports Apache Kafka Wire Protocol Service. Apache Kafka Wire Protocol Service is a TCP/IP service that emulates a Kafka cluster backed by HPE Ezmeral Data Fabric Streams. The service makes it possible for Apache Kafka clients written in any programming language to access topics in HPE Ezmeral Data Fabric Streams.
Configuring Apache Kafka Wire Protocol Service
Describes the Apache Kafka Wire Protocol Service configuration files and instructions.

HPE Ezmeral Data Fabric – Customer-Managed 7.3 Documentation

7.3 Development
This section contains information related to application development for Ezmeral ecosystem components and HPE Ezmeral Data Fabric products, including the file system, Database (Key-Value and JSON), and Event Streams.
- Application Development Process
  Before you start developing applications on the HPE Ezmeral Data Fabric platform, consider how you will get the data into the platform, the storage format of the data, the type of processing or modeling that is required, and how the data will be accessed.
- File Store and Apps
  The following sections provide information about accessing the File Store with C and Java applications.
- HPE Ezmeral Data Fabric Database and Apps
  This section contains information about developing client applications for JSON and key-value tables.
- Apache Kafka Wire Protocol Service
  Starting in EEP 9.0.0, HPE Ezmeral Data Fabric Streams supports Apache Kafka Wire Protocol Service. Apache Kafka Wire Protocol Service is a TCP/IP service that emulates a Kafka cluster backed by HPE Ezmeral Data Fabric Streams. The service makes it possible for Apache Kafka clients written in any programming language to access topics in HPE Ezmeral Data Fabric Streams.
  - Getting Started with Apache Kafka Wire Protocol Service
    Describes the steps required to install, configure, and use Apache Kafka Wire Protocol Service.
  - Configuring Apache Kafka Wire Protocol Service
    Describes the Apache Kafka Wire Protocol Service configuration files and instructions.
    - Apache Kafka Wire Protocol Service Settings
      Lists and describes parameters that you can configure in kafka-server.conf.
  - Securing Apache Kafka Wire Protocol Service
    Describes the security mechanisms that Apache Kafka Wire Protocol Service does and does not support.
  - Enabling SSL for Apache Kafka Wire Protocol Service
    Describes the security protocol values that are needed to enable SSL for Apache Kafka Wire Protocol Service.
  - Supported Apache Kafka RPCs
    Lists the Apache Kafka RPCs that Apache Kafka Wire Protocol Service supports and provides links to the correlating API request documentation in the Kafka Protocol Guide.
- HPE Ezmeral Data Fabric Streams and Apps
  HPE Ezmeral Data Fabric Streams brings integrated publish and subscribe messaging to HPE Ezmeral Data Fabric.
- MapReduce and Apps
  This section contains information associated with developing YARN applications.
- Kubernetes Interfaces for Data Fabric
  This section describes how to leverage the capabilities of the Kubernetes Interfaces for Data Fabric.
- Ecosystem Components
  The following sections provide information about each open-source project that is supported by the HPE Ezmeral Data Fabric.
- Maven and the HPE Ezmeral Data Fabric
  This section discusses topics associated with Maven and the HPE Ezmeral Data Fabric.
- Developer's Reference
  This section contains in-depth information for the developer.
- API Documentation
  HPE Ezmeral Data Fabric supports public APIs for file system, HPE Ezmeral Data Fabric Database, and HPE Ezmeral Data Fabric Streams. These APIs are available for application-development purposes.

Configuring Apache Kafka Wire Protocol Service

Describes the Apache Kafka Wire Protocol Service configuration files and instructions.

The following sections provide configuration information for DAG (Data Access Gateway) and Apache Kafka clients and applications:

Configuring Apache Kafka Wire Protocol Service

DAG packages include the following configuration files for Apache Kafka Wire Protocol Service:

kafka-server.conf

This file is stored in /opt/mapr/data-access-gateway/conf and loaded from the local file system.
This file contains node-specific options, such as the port number and security protocol. Set the options in this file on a per-node basis.
By default, the Apache Kafka Wire Protocol Service listens on port 9092.
If the DAG service instances are behind a firewall, the firewall rules must be configured to allow this traffic.

For a list of configurable parameters, see Apache Kafka Wire Protocol Service Settings.

kafka-cluster.conf

This file is stored in /opt/kafka-wire-protocol/default-cluster/conf and loaded from the distributed file system.
This file contains cluster-wide settings.
A template of this file is located in /opt/mapr/data-access-gateway/conf as kafka-cluster.conf.src.
You must copy the template file (kafka-cluster.conf.src) to the distributed file system under /opt/kafka-wire-protocol/default-cluster/conf and save it as kafka-cluster.conf.

Configuring Apache Kafka Clients and Applications

No special configuration is required on the client side; however, an application must modify the connection string (bootstrap server parameter) to point to the DAG hostname/IP (Apache Kafka Wire Protocol Service). This is similar to pointing to another Kafka cluster when using Apache Kafka.

(Topic last modified: 2023-04-08)