Producers
Producers are data-generating applications, such as sensors in automobiles or activity loggers in servers. Producers create messages with the collected data and publish the messages to HPE Ezmeral Data Fabric Streams topics, specifically, to HPE Ezmeral Data Fabric Streams topic-partitions.
Permissions
Before a producer can publish to topics, the user ID running the producer needs these
permissions:
- The
writeAce
permission on the volume where the streams are located. For information about how to set permissions on volumes, see Setting Whole Volume ACEs. - The
produceperm
permission on the streams where the topics are located. Users with theadminperm
permission on those streams can grant theproduceperm
permission.
Producing Messages
Producers create messages about the collected data and send the collected data to a HPE Ezmeral Data Fabric Streams producer client
library. In addition to the actual message, the producer specifies the topic that the
message is intended for and an optional partition ID. The producer client buffers incoming
messages and sends them (in batches) to the HPE Ezmeral Data Fabric Streams server.
NOTE
In case of server failure, the producer client automatically continues to retry
sending messages.ATTENTION
As of data-fabric 6.1, the HPE Ezmeral Data Fabric Streams API enforces a
maximum of 4096 partitions for a topic. That is, when you create an application with the
API, the maximum number of partitions is 4096. If you previously created an application with
HPE Ezmeral Data Fabric Streams 6.0.1 API (or older) and you have upgraded,
the original number of partitions can be used. For example, if you were using more than 4096
partitions in data-fabric 6.0.1 or
earlier, you will be able to continue with the same number of partitions after
upgrading.Event-time Timestamp
As of data-fabric 6.0.1, HPE Ezmeral Data Fabric Streams supports an event-time timestamp. The timestamp type
can be either createtime
(default) or logappendtime
. See
the maprcli
stream create and stream edit for more information about these
parameters.
TIP
Since each message is automatically published into a topic-partition with an
event-time timestamp as part of the message record, this allows the Consumer application to
seek records based on the timestamp.Idempotent (exactly once) Producers
An "exactly-once" message delivery semantic produces messages without duplication. Each message is delivered once and only once. Exactly-once is insured by uniquely identifying a group of messages that are atomically persisted. Exactly-once message delivery is set with the producer idempotence option. See Modes of Publishing for more information.The following failure scenarios are addressed with idempotence:
- The stream processor might take input from multiple source topics and the ordering across these source topics is not deterministic across multiple runs. So if you re-run your stream processor that takes input from multiple source topics, it might produce different results.
- The stream processor might produce output to multiple destination topics. If the producer cannot do an atomic write across multiple topics, then the producer output can be incorrect if writes to some (but not all) partitions fail.
- The stream processor might aggregate or join data across multiple inputs. If one of the instances of the stream processor fails, then you need to be able to rollback the state materialized by that instance of the stream processor. On restarting the instance, you also need to be able to resume processing and recreate its state.
- The stream processor might look up enriching information in an external database or by calling out to a service that is updated out of band. By depending on an external service, the stream processor can be fundamentally non-deterministic. For example, if the external service changes its internal state between two runs of the stream processor, it can lead to incorrect results downstream.
For More Information
For more information about creating and editing streams or
topics:
- maprcli
- See maprcli stream create for information about creating streams.
- See maprcli stream edit for information about editing streams.
- See maprcli stream info for information about streams.
- See maprcli stream topic create for information about creating topics.
- See maprcli stream topic edit for information about modifying topics.
- See maprcli stream topic info for information about topic data.
- HPE Ezmeral Data Fabric Streams Java API
- See the HPE Ezmeral Data Fabric Streams Java API Library for the methods used to create and edit streams and to create and edit topics.