Configuration Properties for HPE Ezmeral Data Fabric Streams Python Client
In the instance constructor of a HPE Ezmeral Data Fabric Streams Python application, you can use a dictionary to set the following configuration properties. HPE Ezmeral Data Fabric Streams Python client supports a superset of the configuration properties supported by the HPE Ezmeral Data Fabric Streams C client.
Global Configuration Properties
| Property Name | Behavior |
|---|---|
| client.id | Same as librdkafka |
| default.topic.config | A dictionary of topic-level configuration properties that are applied to all used topics for the instance. |
| message.max.bytes | Supports a value less than or equal to 10MB (10000000). If this property is set to a value that is higher than 10MB, the client issues a warning and sets the configuration to 10MB. Produce calls fail when the message size is greater than 10MB. |
| receive.message.max.bytes | Same as librdkafka |
| topic.blacklist | Same as librdkafka |
| error_cb | A callback for generic/global error events. This callback is served by poll(). |
| opaque | Same as librdkafka. |
Consumer Configuration Properties
| Property Name | Behavior |
|---|---|
| group.id | Same as librdkafka. |
| enable.auto.commit | Same as librdkafka. |
| auto.commit.interval.ms | Same as librdkafka. |
| rebalance_cb | Same as librdkafka. |
| offset_commit_cb | Same as librdkafka. |
| delivery.report.only.error | Same as librdkafka. |
| dr_msg_cb | Same as librdkafka. |
| on_commit | A callback used to indicate success or failure of commit requests. |
Topic Configuration Properties
| Property Name | Behavior |
|---|---|
| partitioner_cb | Same as librdkafka. |
| auto.offset.reset | Supports the following values: earliest, latest, none, smallest, and largest. librdkafka also supports biggest, end and error. |
Producer Configuration Properties
| Property Name | Behavior |
|---|---|
| on_delivery(kafka.KafkaError, kafka.Message) | A Python function reference that is called once for each produced message to indicate the final delivery result (success or failure). This property may also be set per-message by passing callback=callable (or on_delivery=callable) to the confluent_kafka.Producer.produce() function. |
HPE Ezmeral Data Fabric-Specific Configuration Properties
| Property Name | Behavior |
|---|---|
| streams.consumer.default.stream | Specifies the path and name of the stream that the consumer subscribes to if,
when subscribing to a topic, the consumer does not specify a stream. For example,
the consumer can specify the name of a stream together with the name of a topic to
write to, like this: /<stream>:<topic>. NOTE rd_kafka_list groups API uses
this consumer configuration to obtain the consumer groups. |
| streams.parallel.flushers.per.partition | Enables the producer may have multiple parallel send requests to the server for each topic partition. If this setting is set to true, it is possible for messages to be sent out of order. |
| streams.producer.default.stream | Specifies the stream that the producer will use by default if the producer does not provide the name of a stream when specifying a topic to write to. For example, the producer can specify the name of a stream together with the name of a topic to write to, like this: /<stream>:<topic>. However, if the stream is not specified, the value of this configuration parameter is assumed to be the stream in which the topic is located. If the producer specifies the name of a topic without also providing the path and name of the stream, and there is no value for this configuration parameter, HPE Ezmeral Data Fabric Streams assumes that the topic specified is in Apache Kafka and does nothing. |
Additional Information
Here is a consumer configuration
example:
conf = {'group.id': 'mygroup',
'session.timeout.ms': 6000,
'on_commit': my_commit_callback,
'default.topic.config': {'auto.offset.reset': 'smallest'}}
consumer = mapr_streams_python.Consumer(**conf)