Stream Design
Streams are created in volumes and contain topics, which in turn, contain messages. Security, replication, retention, and compression policies are applied at the stream-level.
- Security Permissions (using access-control expressions)
Security permissions are set at the stream-level and, subsequently, topics inherit the stream permissions. See Stream Security for more information.
- Data Replication
HPE Ezmeral Data Fabric Streams streams can be replicated to other streams in the same or different data-fabric clusters. For example, you can create a backup copy of a stream so that producers and consumers fail over to the backup if the original stream goes offline. See Stream Replication
- Data Retention
The time-to-live of a message is the elapse time (in seconds) between the publication of a message in a topic in this stream and the expiration of that message. See Time-to-Live for Messages for more information.
- Data Compression
Topic messages can be sent to the server compressed. When compression is implemented, the messages are stored compressed, replicated to other containers compressed, and (if stream replication is configured) replicated to replica streams compressed. See Managing Compression for more information.
Pollution Monitor Example
Suppose that you plan to create the stream pollution_monitors
to collect
various measurements about pollution levels in European cities. However, during a planning
session, the representative from Amsterdam says that her country wants to analyze the data
for its cities and would like your company to replicate the data to its own data-fabric cluster, where its own
consumers can read the replicated messages.
- Create a stream dedicated to the Netherland's pollution data or even for every country
you are monitoring. For example, create streams named
pollution_monitors_netherlands
,pollution_monitors_sweden
,pollution_monitors_france
, and so on. - Within each stream, create topics for each city in that county. For example, create a
topic named
amsterdam
that contains data from Amsterdam's pollution sensors. - Since, in this scenario, the Amsterdam representative also requested stream replication to their own data-fabric cluster, you would set up stream replication from your data-fabric cluster to Amsterdam's data-fabric cluster. See Managing Stream Replication for information about setting up and managing replication.
Alternatively, consider that the Netherlands did not request replication to their own data-fabric cluster. However, you want to restrict access to the pollution data where consumers could read only pollution data for their respective country.
- Create streams for each country.
- Create topics for each city in that country.
- Set each stream's
consumeperm
permission for consumers associated with that country. See Stream Security for more information about security permissions used with HPE Ezmeral Data Fabric Streams streams. For general information about access-control expressions, see ACE Syntax.