Creating a Stream
Explains how to create a stream using the Control System and the CLI.
About this task
Your decision about what streams to create should take into account whatever topics you want to replicate. Replication is between streams, not individual topics.
For example, suppose that you plan to create the stream
pollution_monitors
to collect various measurements about
pollution levels in cities in Europe. However, during a planning session, the
representative from Amsterdam says that her country wants to perform analyses of the
data for its cities, and would like your company to replicate the data to its own
HPE Ezmeral Data Fabric cluster,
where its own consumers can read the replicated messages.
You would create a separate stream of topics that contain data from only the
pollution sensors in the cities in that country. You might even decide to do the
same for each center, in case other centers eventually want to perform their own
analyses, too. The streams you might decide to create could be
pollution_monitors_netherlands
, pollution_monitors_sweden
,
and so on.
Creating a Stream Using the Control System
About this task
To create a stream:
Procedure
-
Log in to the Control System and click Create Stream
under .
NOTEThis option is not available in the Kubernetes version of the Control System.The Create New Stream page displays.
-
Specify the following properties.
Property Description Stream Path The path and name of the stream to create. The path to the stream can include any character allowed by HPE Ezmeral Data Fabric. For example,
/my/path/with:/to/mystream:topic1
is valid, but/my/path/with:/to/mystream:withcolon:topic1
is invalid.The name of the stream cannot include a colon (
:
) or a forward slash (/
).Time To Live The amount of time to elapse between the publication of a message in a topic in this stream and the expiration of that message. Choose: - Forever to retain messages indefinitely
- Seconds to specify the number of seconds. A value of 0 causes messages to be retained indefinitely.
Compression The compression setting to use for the stream. Producer client libraries can bundle messages that are to be published on the same partition and compress them. The messages are sent to the server compressed, are stored compressed, are replicated to other containers compressed, and (if stream replication is configured) replicated to replica streams compressed. Consumer client libraries receive compressed data, decompresses it, and passes it to client applications. Choose one of the following:- Inherited (to inherit from the directory where the stream is stored), which is the default setting
- OFF (to disable compression)
- LZF
- LZF4
- ZLIB
Auto Create Topics Whether (Yes) or not (No) to create a topic automatically when a producer tries to write the first message to it. Default Partitions The default number of partitions to allocate to new topics in the stream. When deciding how many partitions to create by default for new topics in a stream, consider the expected volume of messages that will be published to the topics in the stream. High message volumes can be handled more efficiently by multiple consumers in consumer groups reading from multiple partitions than by individual consumers reading from a single partition. NOTEYou can override the default and specify a different number of partitions for each topic in the stream at the time of creating the topic or after creating the topic.Use for Change Log Whether (Yes) or not (No) to create the stream for changed data records (as a result of inserts, updates, and deletes) in a HPE Ezmeral Data Fabric Database table. -
Set up access to streams for users, groups, and roles.
For each user, group, and/or role, you can grant (by selecting the associated check box) or block (by deselecting the associated checkbox) the following types of access:
By default, all permissions are given to the user creating the stream. To grant or block access to other users, groups, and/or roles, choose one of the following:Administer Can modify the access-control expressions for the stream, set up replication from the stream, and modify attributes of the stream. This permission includes the topic permission.
Copy Stream Can copy data from one HPE Ezmeral Data Fabric stream to another HPE Ezmeral Data Fabric stream (using the mapr copystream
utility) and compare the message IDs, metadata, and data in two HPE Ezmeral Data Fabric streams (using themapr diffstreams
utility).Topic Can create, edit, or remove topics in the stream. Producer Can publish messages to topics in the stream. Consumer Can listen to topics in the stream. - Basic Settings: Select the type — public, (OR) user, group, or role
— from the drop-down list and grant read and/or write
permissions.Click Add Another to add permissions for another user, group, or role.TIPClick to create a copy of the associated access control setting. Click to remove the associated access control expression.
- Advanced Settings: Within empty strings (""), specify user (u),
group (g), role (r), or public (p) who have and do not have read
and/or write access using the following boolean expressions and subexpressions:
!
— Negation operator.&
— AND operation.|
— OR operation.
()
, parentheses, for subexpressions.Alternatively, click associated with the type of access to use the Create Access Control Expression window to define access for public or users, group, and/or role.
To add access control expressions for another user, group, or role, click Add Another and repeat this step.NOTEIf you switch from Basic to Advanced, the basic settings, if any, will be carried over to the Advanced settings. If you switch from Advanced to Basic, all the settings will be lost because the subexpressions and AND (&) and negation (!) operations that are supported by Advanced settings are not supported in the Basic settings. - Basic Settings: Select the type — public, (OR) user, group, or role
— from the drop-down list and grant read and/or write
permissions.
- Click Create Stream to create the stream.
Creating a Stream Using the CLI or REST API
About this task
The basic command to create a stream is:
maprcli stream create -path <Stream Path>
readAce
andwriteAce
on the volumelookupdir
on directories in the path
For complete reference information, see
stream create
.