Logical Schema of Messages
Each message has the same logical schema: _id, topic, partition, offset, timestamp, producer, key, and value.
As the logical schema of each message is the same, analytics applications can run queries on these fields. See HPE Ezmeral Data Fabric Streams Java API Library for information about querying messages and mapr streamanalyzer for a sample application used to query and count messages in topics.
{
"_id":<STRING>,
"topic":<STRING>,
"partition":<SHORT>,
"offset":<LONG>,
"timestamp":<LONG>,
"producer":<VARCHAR>,
"key":<BINARY>,
"value":<VARBINARY>
}
Field | Description |
---|---|
_id |
A STRING value that represents the ID of the topic in which the message is located. |
topic |
A STRING value that represents the name of the topic in which the message is located. |
partition |
A SHORT value that represents the index of the partition in the topic. |
offset |
A LONG value that represents the position of the message within a partition. |
timestamp |
A LONG value that represents the date and time of the message. As of data-fabric 6.0.1, HPE Ezmeral Data Fabric Streams supports an event-time timestamp. The timestamp
type can be either createtime (default) or
logappendtime . A A TIP Because each message is automatically produced into a topic-partition with an
event-time timestamp as part of the message record, this allows the Consumer to
seek based on the timestamp. |
producer |
A VARCHAR value that represents the value of the
client.id configuration parameter for the
producer that published the message. HPE Ezmeral Data Fabric Streams does not require a
value for this configuration parameter, so the value for this field
could be empty. |
key |
A BINARY value that represents the key of the message. HPE Ezmeral Data Fabric Streams
does not require each message to have a key, so this value could be
empty. The configuration parameter key.serializer
for the producer that published the message specifies the means by
which the key was serialized.
|
value |
A VARBINARY value that represents the value of the message. The
configuration parameter value.serializer for the
producer that published the message specifies the means by which the
value was serialized.
|
Resources
- maprcli
- See maprcli stream create for information about creating streams.
- See maprcli stream edit for information about editing streams.
- See maprcli stream info for information about streams.
- See maprcli stream topic create for information about creating topics.
- See maprcli stream topic edit for information about modifying topics.
- See maprcli stream topic info for information about topic data.
- HPE Ezmeral Data Fabric Streams Java API
- See the HPE Ezmeral Data Fabric Streams Java API Library for the methods used to create and edit streams and to create and edit topics.