Synchronous Consumer Partition Assignment for Streams

Describes synchronous consumer partition assigment for various applications that use Data Fabric Streams.

Synchronous Consumer Partition Assignment

As of HPE Ezmeral Data Fabric 7.8, synchronous consumer partition assignment is supported. For streams created with HPE Ezmeral Data Fabric 7.8 and later, this feature is enabled by default. To disable this feature, set streams.async.subscription.callback.enabled=true in core-site.xml. By default, this value is set to false.

With this feature enabled, consumer assignment updates occur synchronously within the consumer.poll() call in the user thread.

When setting a new consumer partition assignment, consumer.assignment() does not update until you invoke consumer.poll().
NOTE
For the call to update, you might have to invoke consumer.poll() multiple times, or with a long enough timeout for assignment to be completed.
IMPORTANT
If you have an application that is set to sleep until consumer partition assignment is updated, you must perform one of the following actions to ensure the application continues to work as intended:
  • Disable synchronous consumer partition assignment with streams.async.subscription.callback.enabled=true in core-site.xml.
  • Enable the application to begin polling without the new user assignment.