Consuming Messages

Describes the process by which consumers consume messages.

Consumers request the HPE Ezmeral Data Fabric Streams consumer client library to check whether any new messages have been published in the topics or partitions to which they are subscribed, or the partitions that they are assigned. Consumers can do this at any time.

If a minimum number of bytes worth of messages is waiting across a consumer's subscription, HPE Ezmeral Data Fabric Streams sends those messages to the consumer, up to a maximum number of bytes. You can configure this minimum and maximum in the configuration parameters for each consumer.

The HPE Ezmeral Data Fabric Streams consumer client library sends the consumer messages that have been published by producers but not yet flushed to disk. If a consumer is able to consume data at the rate at which a producer publishes messages, the consumer client library continuously sends messages to consumers from its memory, increasing the speed of throughput from producer to consumer.

Time-based Consumption

As of data-fabric 6.0.1, HPE Ezmeral Data Fabric Streams supports the consumption of messages based on the message's timestamp. When a consumer wants to search for messages based on a timestamp, the consumer provides the topic-partition and the timestamp, and then, HPE Ezmeral Data Fabric Streams locates the message and returns the offset for that message. The returned message offset corresponds to the earliest message in a topic-partition whose timestamp is equal to or greater than (>=) the consumer-provided timestamp.

For example, with the following topic-partition, if your consumer-provided timestamp is 1522195205, then offset 1 would be returned because it is the earliest message with a timestamp that is greater than or equal to the consumer-provided timestamp. In this case, greater than (>).

topic:partition0
        offset 0: 1522195200
        offset 1: 1522195210
        offset 2: 1522195205
        offset 3: 1522195215
TIP
The consumer-provided timestamp and the returned message offset is in seconds since a Epoch Unix timestamp is used. In this example, the consumer-provided timestamp is March 26th, 2018 @ 12:00:05am and the message offsets are timestamped March 28th @ 12:00:00, 12:00:010, 12:00:05, and 12:00:15 in that order.

Resources

For information about HPE Ezmeral Data Fabric Streams streams or topics, see: