Logical Schema of Messages

Each message has the same logical schema: _id, topic, partition, offset, timestamp, producer, key, and value.

As the logical schema of each message is the same, analytics applications can run queries on these fields. See HPE Data Fabric Streams Java API Library for information about querying messages and mapr streamanalyzer for a sample application used to query and count messages in topics.

{
          "_id":<STRING>,
          "topic":<STRING>,
          "partition":<SHORT>,
          "offset":<LONG>,
          "timestamp":<LONG>,
          "producer":<VARCHAR>,
          "key":<BINARY>,
          "value":<VARBINARY>
          }
Field Description
_id A STRING value that represents the ID of the topic in which the message is located.
topic A STRING value that represents the name of the topic in which the message is located.
partition A SHORT value that represents the index of the partition in the topic.
offset A LONG value that represents the position of the message within a partition.
timestamp A LONG value that represents the date and time of the message. As of Data Fabric 6.0.1, HPE Data Fabric Streams supports an event-time timestamp. The timestamp type can be either createtime (default) or logappendtime.

A createtime value (default) is the time defined by the user or application (when creating the message). If user or application does not define this value (or passes null), the client uses the current system timestamp.

A logappendtime value is the time when the message (log) was appended to the server.

TIP
Because each message is automatically produced into a topic-partition with an event-time timestamp as part of the message record, this allows the Consumer to seek based on the timestamp.
producer A VARCHAR value that represents the value of the client.id configuration parameter for the producer that published the message. HPE Data Fabric Streams does not require a value for this configuration parameter, so the value for this field could be empty.
key A BINARY value that represents the key of the message. HPE Data Fabric Streams does not require each message to have a key, so this value could be empty. The configuration parameter key.serializer for the producer that published the message specifies the means by which the key was serialized.
value A VARBINARY value that represents the value of the message. The configuration parameter value.serializer for the producer that published the message specifies the means by which the value was serialized.

Resources

For more information about creating and editing streams or topics: