librdkafka APIs Supported by HPE Data Fabric Streams C Client

This topic lists the librdkafka APIs supported by the HPE Data Fabric Streams C Client. It also describes behavior differences between librdkafka and the HPE Data Fabric Streams C Client.

Table 1. Supported Apache Kafka librdkafka versions
Core release EEP Release Kafka librdkafka version
As of HPE Data Fabric 7.10.0 As of 9.4.0 2.6.0**
As of HPE Data Fabric 7.9.0 As of 9.3.1 2.0.2*
As of HPE Data Fabric 6.0.1 As of 5.0 0.11.3
As of HPE Data Fabric 5.2.1 through 6.0.0 As of 3.0 0.9.0

*Librdkafka 2.0.2 in core 7.9 is not compatible with HPE Data Fabric Stream clients for Python1 or C#2 applications. Nodes running Python or C# applications should not be upgraded to HPE Data Fabric 7.9.

**Librdkafka 2.6.0 in core 7.10 is not compatible with HPE Data Fabric Stream clients for C#2 applications. Nodes running C# applications should not be upgraded to HPE Data Fabric 7.10.

1HPE Data Fabric Streams Python Applications

2HPE Data Fabric Streams C#/.NET Applications

This topic contains the following supported APIs:

Producer APIs

API Behavior
rd_kafka_flush Same as librdkafka.
rd_kafka_poll Same as librdkafka.
rd_kafka_produce When this API is called with NULL payload, an invalid argument error is sent to the callback. librdkafka creates a message with NULL payload and key value instead.
rd_kafka_producev Same as librdkafka. This API should be used with either RD_KAFKA_V_TOPIC or RD_KAFKA_V_RKT.
rd_kafka_produce_batch When this API is called with NULL payload, an invalid argument error is sent to the callback. librdkafka creates a message with NULL payload and key value instead.
rd_kafka_outq_len This API returns a positive number to indicate that messages are waiting to be produced to a streams topic but the value does not indicate the actual number of messages. librdkafka returns the actual number of messages that are waiting to be sent to or acknowledged by the broker.

Consumer APIs

API Behavior
rd_kafka_assign If this API is called for a consumer that is already subscribed to topics, no operation is performed.
rd_kafka_assignment This API returns the number of topic partitions the consumer is assigned to. However, it returns 0 when topic partitions have yet to be created by the producer. librdkafka returns the number of partitions assigned to a consumer even when the partitions have not been created.
NOTE
For this API to work, the argument partitions must be explicitly allocated or initialized with either rd_kafka_topic_partition_list_t *parts = NULL or rd_kafka_topic_partition_list_new(0);. For example: 
RD_EXPORT rd_kafka_resp_err_t
rd_kafka_assignment (rd_kafka_t *rk,
                     rd_kafka_topic_partition_list_t **partitions);
rd_kafka_commit Same as librdkafka.
rd_kafka_commit_message Same as librdkafka.
rd_kafka_committed Same as librdkafka.
NOTE
The HPE Data Fabric Streams offset starts at 1.
rd_kafka_consumer_close Same as librdkafka.
rd_kafka_consumer_poll Same as librdkafka.
rd_kafka_group_list_destroy Same as librdkafka.
rd_kafka_list_groups This API can only be used by consumers that are subscribed to at least one stream on the cluster and have a default stream configured with the streams.consumer.default.stream parameter. It returns the group list of subscribed consumers associated with the default stream. librdkafka returns all consumer groups from the cluster instead.
NOTE
This API returns RD_KAFKA_RESP_ERR__TIMED_OUT when the querying consumer is not subscribed to any topic.
rd_kafka_message_destroy Same as librdkafka.
rd_kafka_poll_set_consumer Same as librdkafka.
rd_kafka_position This API returns 0 when the messages have not yet been consumed from partitions. librdkafka returns -1001 instead.
rd_kafka_seek Same as librdkafka.
rd_kafka_subscribe Same as librdkafka.
rd_kafka_subscription This API allows either a list of topics from one or more streams or a regex expression for topics from a single streams. For example, regex expression /streamA:^t*a,/streamA:^t*b is supported but /streamA:^t*a,/streamB:^t*a is not supported. librdkafka accepts both options in the same call.
NOTE
You cannot use the rd_kafka_subscribe API to subscribe a consumer to topics when that consumer is already assigned to topics. If you call this API for an assigned consumer, error RD_KAFKA_RESP_ERR__CONFLICT is returned.
rd_kafka_unsubscribe Same as librdkafka.

Producer/Consumer Common APIs

API Behavior
rd_kafka_conf_destroy Same as librdkafka.
rd_kafka_conf_dup Same as librdkafka.
rd_kafka_conf_get Same as librdkafka.
rd_kafka_conf_new Same as librdkafka.
rd_kafka_conf_set Same as librdkafka.
rd_kafka_conf_set_consume_cb Same as librdkafka.
rd_kafka_conf_set_dr_msg_cb Same as librdkafka.
rd_kafka_conf_set_error_cb Same as librdkafka.
rd_kafka_conf_set_offset_commit_cb Same as librdkafka.
rd_kafka_conf_set_opaque Same as librdkafka.
rd_kafka_conf_set_rebalance_cb Same as librdkafka.
rd_kafka_conf_set_throttle_cb Same as librdkafka.
rd_kafka_destroy Same as librdkafka.
rd_kafka_new Same as librdkafka.
rd_kafka_opaque Same as librdkafka.
rd_kafka_wait_destroyed Same as librdkafka.
rd_kafka_yield Same as librdkafka.

Topic APIs

API Behavior
rd_kafka_conf_set_default_topic_conf Same as librdkafka.
rd_kafka_topic_conf_destroy Same as librdkafka.
rd_kafka_topic_conf_dup Same as librdkafka.
rd_kafka_topic_conf_new Same as librdkafka.
rd_kafka_topic_conf_set Same as librdkafka.
rd_kafka_topic_conf_set_opaque Same as librdkafka.
rd_kafka_topic_conf_set_partitioner_cb Same as librdkafka.
rd_kafka_topic_destroy Same as librdkafka.
rd_kafka_topic_name Same as librdkafka.
rd_kafka_topic_new Same as librdkafka..
rd_kafka_topic_partition_list_add Same as librdkafka.
rd_kafka_topic_partition_list_add_range Same as librdkafka.
rd_kafka_topic_partition_list_copy Same as librdkafka.
rd_kafka_topic_partition_list_del Same as librdkafka.
rd_kafka_topic_partition_list_del_by_idx Same as librdkafka.
rd_kafka_topic_partition_list_destroy Same as librdkafka.
rd_kafka_topic_partition_list_find Same as librdkafka.
rd_kafka_topic_partition_list_new Same as librdkafka.
rd_kafka_topic_partition_list_set_offset Same as librdkafka.

Queue APIs

API Behavior
rd_kafka_queue_new Same as librdkafka.
rd_kafka_queue_destroy Same as librdkafka.
rd_kafka_queue_get_main For this API, produce events are batched as well as the APIs that use this API, such as, rd_kafka_event_message_count and rd_kafka_event_message_next. The messages produce events can be consumed together in batches, wheras, opensource librdkafka events are obtained one at a time.
rd_kafka_queue_get_consumer Same as librdkafka.
rd_kafka_queue_length Same as librdkafka.
rd_kafka_queue_poll Same as librdkafka.
rd_kafka_queue_poll_callback Same as librdkafka.
rd_kafka_commit_queue Same as librdkafka.
rd_kafka_queue_forward Same as librdkafka.
rd_kafka_set_log_queue Same as librdkafka.
rd_kafka_queue_length Same as librdkafka.
rd_kafka_queue_io_event_enable Same as librdkafka.
rd_kafka_queue_get_partition Same as librdkafka.

Event APIs

API Behavior
rd_kafka_event_type Same as librdkafka.
rd_kafka_event_name Same as librdkafka.
rd_kafka_event_destroy Same as librdkafka.
rd_kafka_event_message_array Same as librdkafka.
rd_kafka_event_message_count Same as librdkafka.
rd_kafka_event_error Same as librdkafka.
rd_kafka_event_error_string Same as librdkafka.
rd_kafka_event_topic_partition_list Same as librdkafka.
rd_kafka_event_message_next Same as librdkafka.
rd_kafka_event_opaque Same as librdkafka.

Timestamp APIs

API Behavior
rd_kafka_offsets_for_times Same as librdkafka.
rd_kafka_message_timestamp Same as librdkafka.

Interceptors APIs

ATTENTION
Modifying the message in interceptors is not supported and can result in undefined behavior.
API Behavior
rd_kafka_interceptor_f_on_conf_set_t Same as librdkafka.
rd_kafka_interceptor_f_on_conf_dup_t Same as librdkafka.
rd_kafka_interceptor_f_on_conf_destroy_t Same as librdkafka.
rd_kafka_interceptor_f_on_new_t Same as librdkafka.
rd_kafka_interceptor_f_on_destroy_t Same as librdkafka.
rd_kafka_interceptor_f_on_send_t Same as librdkafka.
rd_kafka_interceptor_f_on_acknowledgement_t Same as librdkafka.
rd_kafka_interceptor_f_on_consume_t Same as librdkafka.
rd_kafka_interceptor_f_on_commit_t Same as librdkafka.
rd_kafka_conf_interceptor_add_on_conf_set Same as librdkafka.
rd_kafka_conf_interceptor_add_on_conf_dup Same as librdkafka.
rd_kafka_conf_interceptor_add_on_conf_destroy Same as librdkafka.
rd_kafka_conf_interceptor_add_on_new Same as librdkafka.
rd_kafka_conf_interceptor_add_on_destroy Same as librdkafka.
rd_kafka_interceptor_add_on_send Same as librdkafka.
rd_kafka_interceptor_add_on_acknowledgement Same as librdkafka.
rd_kafka_interceptor_add_on_consume Same as librdkafka.
rd_kafka_interceptor_add_on_commit Same as librdkafka.

Cluster Configuration APIs

API Behavior
rd_kafka_brokers_add

This API has no impact on HPE Data Fabric Streams since HPE Data Fabric Streams does not utilize Kafka brokers. When this API is called, the HPE Data Fabric Streams client may print a brokers are down error message to the console.

Miscellaneous API

API Behavior
rd_kafka_err2name Same as librdkafka.
rd_kafka_err2str Same as librdkafka.
rd_kafka_errNo Same as librdkafka.
rd_kafka_errNo2err Same as librdkafka.
rd_kafka_get_err_descs Same as librdkafka.
rd_kafka_last_error Same as librdkafka.
rd_kafka_log_print Same as librdkafka.
rd_kafka_log_syslog Same as librdkafka.
rd_kafka_message_errstr Same as librdkafka.
rd_kafka_set_logger Same as librdkafka.
rd_kafka_query_watermark_offsets When you are querying or retrieving a topic that is non-existent topic/partition (using rd_kafka_query_watermark_offsets() and rd_kafka_get_watermark_offsets() APIs), the timeout is honored even though you still receive the correct error message.
rd_kafka_get_watermark_offsets When you are querying or retrieving a topic that is non-existent topic/partition (using rd_kafka_query_watermark_offsets() and rd_kafka_get_watermark_offsets() APIs), the timeout is honored even though you still receive the correct error message.

Additional Information

For more information and API signatures, see rdkafka.h.