Time-to-Live for Messages
The time-to-live (TTL) for messages means that messages persist in the partitions of a stream topic for a specific time period. During that time, messages can be read or re-read by consumers. Once the TTL for a message expires, the message is marked for deletion.
Setting TTL for Message
Set the TTL for topic messages when you create or edit a stream. Since the TTL setting is specified at the stream-level, all messages in all topics associated with the stream will have the same TTL. The default TTL is 604,800 seconds (7 days).
Deleting Expired Messages
Whenever there are any consumer or producer operations on any stream tablet, the partitions in that tablet qualify for purge only if
-
it has been more than 24 hours since the last purge on this tablet and,
-
more than TTL/10 secs have expired since the last purge on this tablet and,
-
reclaimed disk space percentage at the tablet partition level is greater than or equal to 10.
For example, if the reclaimed disk space at the tablet partition level is greater than or equal to 10% and:
- If the TTL is set to 24 hours, the expired messages are deleted once every 24 hours.
- If the TTL is set to 7 days (168 hours, which is the default), the expired messages are deleted once every 24 hours because 24 is greater than 168/10.
- If the TTL is set to 20 days (480 hours), the expired messages are deleted once every 48 hours because 48 is greater than 24.
You must monitor disk space utilization and manually delete messages from streams, as needed, to reclaim disk space.
To manually delete expired messages, run the maprcli
command stream purge.
For More Information
- See maprcli stream create for information about creating streams.
- See maprcli stream edit for information about editing streams.
- See maprcli stream purge for information about purging expired topic messages.
- See HPE Ezmeral Data Fabric Streams Java API Library for the methods used to create and edit streams