Parallelism when Consuming Messages

For parallelism when reading messages from topics, you can create consumer groups. These groups consist of consumers that are associated with an ID that you set for each of the participating consumers with the group.id configuration parameter. The partitions in each topic to which all of the consumers are subscribed, are assigned dynamically to the consumers in round-robin fashion.

These groups consist of consumers that are associated with an ID that you set for each of the participating consumers with the group.id configuration parameter. The partitions in each topic to which all of the consumers are subscribed, are assigned dynamically to the consumers in round-robin fashion.

For example, suppose that there are three consumers in a group and each consumer is subscribed to the same topic. There are five partitions in the topic. HPE Ezmeral Data Fabric Streams assigns each partition to a consumer, with two consumers both being assigned two partitions.

If one of the consumers goes offline, the partitions are reassigned dynamically among the remaining consumers in the group.

If the offline consumer comes back online or a different consumer is added to the group, again the partitions are redistributed among the consumers in the group.

This parallelism and dynamic reassignment is possible only when none of the consumers in a consumer group subscribe to individual partitions.

For example, suppose that from three consumers in a consumer group:
  • Two subscribe to the same topic.
  • One subscribes to a single partition within that topic.
If the topic has five partitions, HPE Ezmeral Data Fabric Streams assigns four of them via round robin to two of the consumers. Only the remaining partition is read from the third consumer.

If that third consumer fails, HPE Ezmeral Data Fabric Streams does not reassign its partition to either of the other consumers.

Now that you understand how partitions are assigned when the number of partitions is equal to or greater than the number of consumers in a consumer group, you might be wondering what happens if the number of partitions in a topic is less than the number of consumers in a consumer group. The answer is simply that one or more consumers in the consumer group will not be assigned any partitions from the topic.

That does not necessarily mean those consumers will be idle. There could be other topics to which the consumer group is subscribed, and those consumers could be assigned partitions from those other topics.

For example, in this diagram there is a consumer group with four consumers. Topic A has only three partitions, and those are assigned to the first three consumers shown in the group. However, the fourth consumer is not idle. The consumer group also subscribes to Topic B, which has more partitions than there are consumers. Each of the consumers in the group is assigned at least one partition from Topic B.

Moreover, if a consumer that is assigned a partition from Topic A happens to fail, its partition will be reassigned to the fourth consumer.