mapr diffstreamswithcrc
This utility uses a cyclic redundancy check to detect differences between sets of messages in the specified HPE Ezmeral Data Fabric Streams. Then, for each set of non-identical messages, it performs a detailed comparison. Finally, it generates one or more directories of sequence files.
Sequence files are binary flat files. You can learn more about them here. To convert a sequence file into a format that you can read, use the HPE Ezmeral Data Fabric Database JSON FormatResult utility.
This utility requires less network bandwidth than the mapr diffstreams
utility because it performs a
detailed table comparison only on the sets of messages where the CRC algorithm detected a
difference. Therefore, consider using this utility when the streams you compare are very
similar and you are concerned about the data transfer rate.
This utility considers both the source stream and the destination stream to be a master stream. Therefore, it generates two directories with sequence files. These sequence files contain the puts required to update each stream so that each stream can contain a superset of the messages in both streams at the time at which the utility was run.
- opsForDst
- A directory containing sequence files that correspond to each put and delete required to make the destination stream identical to the source stream.
- opsForSrc
- A directory containing sequence files that correspond to each put and delete required to make the source stream identical to the destination stream.
Requirements
- When the cluster runs YARN, it must also use zero configuration failover for the ResourceManager.
- To use this utility, you must have the following permissions:
- The permission
readAce
on the volumes where the tables are located. - On the source stream: either
consumeperm
orcopyperm
. - On the destination stream: either
consumeperm
orcopyperm
.
For information about how to set permissions on volumes, see Setting Whole Volume ACEs.
For information about how to set permissions on streams, see Enabling Table and Stream Authorizations with ACEs.
- The permission
mapr
user is not treated as a
superuser. HPE Ezmeral Data Fabric Streams does not allow the
mapr
user to run this utility unless that user is given the relevant
permission or permissions with access-control expressions.Run the mapr importstream
command to implement the puts and deletes
specified in the sequence files.
Syntax
mapr diffstreamswithcrc
-src <srcStream>
-dst <dstStream>
-outdir <output directory>
[-first_exit] Exit when first difference is found
Parameters
Parameter | Description |
---|---|
src |
The path of the first stream to include in the comparison. |
dst |
The path of the second stream to include in the comparison. |
outdir |
The path to a directory in which to place the generated sequence files. The utility creates the specified directory. If the specified directory already exists, the command fails. |
first_exit |
By default, the utility compares all the data in the specified streams. Use this parameter if you want to exit after the first difference is identified between the streams. The parameter takes no value. |