Time-to-Live for Messages

The time-to-live (TTL) for messages means that messages persist in the partitions of a stream topic for a specific time period. During that time, messages can be read or re-read by consumers. Once the TTL for a message expires, the message is marked for deletion.

Setting TTL for Message

Set the TTL for topic messages when you create or edit a stream. Since the TTL setting is specified at the stream-level, all messages in all topics associated with the stream will have the same TTL. The default TTL is 604,800 seconds (7 days).

Deleting Expired Messages

Whenever there are any consumer or producer operations on any stream tablet, the partitions in that tablet qualify for purge only if

  • it has been more than 24 hours since the last purge on this tablet and,

  • more than TTL/10 secs have expired since the last purge on this tablet and,

  • reclaimed disk space percentage at the tablet partition level is greater than or equal to 10.

For example, if the reclaimed disk space at the tablet partition level is greater than or equal to 10% and:

  • If the TTL is set to 24 hours, the expired messages are deleted once every 24 hours.
  • If the TTL is set to 7 days (168 hours, which is the default), the expired messages are deleted once every 24 hours because 24 is greater than 168/10.
  • If the TTL is set to 20 days (480 hours), the expired messages are deleted once every 48 hours because 48 is greater than 24.
ATTENTION
The messages from active streams that have an expired TTL are deleted using the aforementioned mechanism. Deleted messages with an expired TTL from idle streams are not purged until producer or consumer operations are performed on such streams.

You must monitor disk space utilization and manually delete messages from streams, as needed, to reclaim disk space.

To manually delete expired messages, run the maprcli command stream purge.

For More Information