Running a Kafka Streams Java App

Describes how to set up and run a Kafka Streams Java application.

Setup

To set up your project, add the required dependencies to the pom.xml file, as shown in the example. Note that the versions you use may differ from the versions shown in the example. Version numbers typically change with new releases.

Maven artifacts are published to https://repository.mapr.com/maven/. You can also refer to Maven Artifacts for the HPE Ezmeral Data Fabric for dependency versions.

Example pom.xml

The following example shows the dependencies you must add to pom.xml. 
<repository>
	<id>mapr-releases</id>
<url>https://repository.mapr.com/maven/</url>
</repository>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.1.1.200-mapr-710</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams</artifactId>
    <version>2.1.1.200-mapr-710</version>
</dependency>
<dependency>
    <groupId>com.mapr.streams</groupId>
    <artifactId>mapr-streams</artifactId>
    <version>6.2.0.0-mapr</version>
</dependency>
<dependency>
     <groupId>org.rocksdb</groupId>
     <artifactId>rocksdbjni</artifactId>
     <version>5.7.3</version>
</dependency>
<dependency>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
     <version>1.1.1</version>
</dependency>
IMPORTANT: For Kafka 2.6.1, enter 5.18.4 as the rocksdbjni version, and use kafka-eventstreams instead of mapr-streams, as shown:
<dependency>
     <groupId>org.rocksdb</groupId>
     <artifactId>rocksdbjni</artifactId>
     <version>5.18.4</version>
</dependency>
<dependency>
    <groupId>com.mapr.kafka</groupId>
    <artifactId>kafka-eventstreams</artifactId>
    <version>0.1.0.0-eep-800</version>
</dependency>

Running a Kafka Streams App on a Cluster

To run a Kafka Streams Java application on a cluster:

  1. Copy the <Kafka Streams Java application>.jar file to an arbitrary folder on your cluster.
  2. Run the following shell command on your cluster:
    • For Kafka 2.1.1, run:
      java -cp "$(mapr clientclasspath):<Kafka Streams Java application>.jar" <Kafka Streams Application Main Class Name>
    • For Kafka 2.6.1, run:
      java cp "$(mapr clientclasspath):/opt/mapr/kafka/kafka<version>/libs/kafka-eventstreams<version>.jar:<Kafka Streams Java application>.jar" <Kafka Streams Application Main Class Name>