och-instance create

Creates a copier.

You can use the och-instance create command to create an object copier hub instance.

The security certificate CA chain and the replication rules file are required for auto load-balancing of copier tasks on the cluster.

If the -iscontentinline parameter is set to false while creating a copier, you must maintain a copy of the security certificate CA chain and the replication rules file on on all cluster nodes that run the mapr-och service. If you set the -iscontentinline parameter to true, you do not need to maintain the copies of security certificate CA chain and replication rules file on all cluster nodes that run the mapr-och service.

Syntax

CLI
maprcli och-instance create
 -name copier name
                [ -node FQDN of the OCH service node; auto-assigned via load balancing across active OCH nodes if omitted ]
                [ -sticky pin this copier to -node and exclude it from automatic load rebalancing. default: false. default: false ]
                 -isSQSListener true if using SQS listener, false if using Kafka. default: false
                [ -enableMultipart true to enable multipart copy for large objects. default: false. default: false ]
                [ -multipartThreshold minimum object size in bytes to trigger multipart copy. default: 524288000 (500MB) ]
                [ -multipartSize size of each part in bytes during multipart copy. default: 104857600 (100MB) ]
                [ -multipartConcurrency number of concurrent part uploads. default: 10 ]
                [ -isAuditEnabled true to record one audit entry per copy attempt made by this copier, under /var/mapr/local/mapr.och.audit. default: false. default: false ]
                [ -sqsQueueUrl SQS queue URL, e.g. https://sqs.us-east-1.amazonaws.com/123456789/queue-name ]
                [ -sqsAwsRegion AWS region for SQS, e.g. us-east-1 ]
                [ -sqsAwsAccessKey AWS access key ID for SQS access ]
                [ -sqsAwsSecretKey AWS secret access key for SQS access ]
                [ -sqsAwsSessionToken AWS STS session token for SQS access; required when -sqsAwsAccessKey/-sqsAwsSecretKey are temporary credentials ]
                [ -sqsRoleArn IAM role ARN to assume for SQS access ]
                [ -kafkaTopics comma-separated Kafka topic list, e.g. /var/objstore/event-stream:event-topic ]
                [ -srcIsAWS true if source is AWS S3, false if source is DataFabric. default: false ]
                [ -srcAwsRegion AWS region for source S3 bucket ]
                [ -srcAwsAccessKey AWS access key ID for source S3 access ]
                [ -srcAwsSecretKey AWS secret access key for source S3 access ]
                [ -srcAwsSessionToken AWS STS session token for source S3 access; required when -srcAwsAccessKey/-srcAwsSecretKey are temporary credentials ]
                [ -srcRoleArn IAM role ARN to assume for source S3 access ]
                [ -srcAccessKey DataFabric access key for source ]
                [ -srcSecretKey DataFabric secret key for source ]
                [ -srcEndpoint source DataFabric endpoint, e.g. https://hostname:9000 ]
                [ -srcChainCA path to CA certificate chain file for source endpoint, or the PEM chain itself with -isContentInline true ]
                [ -dstAccessKey DataFabric access key for destination ]
                [ -dstSecretKey DataFabric secret key for destination ]
                [ -dstEndpoint destination DataFabric endpoint, e.g. https://hostname:9000 ]
                [ -dstChainCA path to CA certificate chain file for destination endpoint, or the PEM chain itself with -isContentInline true ]
                [ -bucketMap comma-separated <sourceBucket>:<destBucket>[:<rules>] entries, e.g. src1:dst1,src2:dst2:/path/rules2.json; <rules> is a file path unless -isContentInline true, when it is the rules JSON itself ]
REST
Request Type POST
Request URL
http[s]://<host>:<port>/rest/och-instance/create?<parameters>

Parameters

Parameter Description
name

(Required) The name of the OCH instance.

node

(Optional) The host name on which the object copier hub is installed/running

sticky (Optional) Pin this copier to -node and exclude it from automatic load rebalancing. .Default: false.
isSQSListener (Optional) Set to true if you are using SQS listener, false if using Kafka. Default: false.
-enableMultipart (Optional) Set to true to enable multipart copy for large objects. Default: false.
multipartThreshold (Optional) The minimum object size, in bytes, to trigger multipart copy. Default: 524288000 (500MB).
multipartSize (Optional) The size of each part in bytes during multipart copy. Default: 104857600 (100MB)
multipartConcurrency The number of concurrent part uploads. Default: 10
sqsQueueUrl (Optional) The SQS queue URL. For example, https://sqs.us-east-1.amazonaws.com/123456789/queue-name
sqsAwsRegion (Optional). The AWS region for SQS, For example, us-east-1
sqsAwsAccessKey (Optional). The AWS access key ID for SQS access
sqsAwsSecretKey (Optional). The AWS secret access key for SQS access
sqsAwsSessionToken (Optional). The AWS STS session token for SQS access; required when -sqsAwsAccessKey/-sqsAwsSecretKey are temporary credentials.
sqsRoleArn (Optional). The IAM role ARN to assume for SQS access.
kafkaTopics (Optional). A comma-separated Kafka topic list. For example, /var/objstore/event-stream:event-topic
srcRoleArn (Optional). The IAM role ARN to assume for source S3 access
srcSecretKey (Optional). The source Data Fabric secret key.
srcAccessKey (Optional). The source Data Fabric access key.
srcChainCA (Optional). The path to CA certificate chain file for the source endpoint. You must have acccess rights on the folder containing the CA certificate chain file.
srcEndpoint (Optional). The source Data Fabric endpoint, For example, https://hostname:9000
dstAccessKey (Optional). The destination Data Fabric access key
dstSecretKey (Optional). The destination Data Fabric secret key
dstEndpoint (Optional). The destination Data Fabric endpoint. For example, https://hostname:9000
dstChainCA (Optional). The CA certificate chain file path for the destination endpoint. You must be able to access the folder containing the CA certificate chain.
bucketMap (Optional). Comma-separated <sourceBucket>:<destBucket>[:<rulesFile>] entries for bucket mapping. The source bucket is the bucket from where to copy the objects and the destination bucket is the destination to which the objects are to be copied. The replication rules file forms the basis to filter the objects to copy from the source bucket to the destination bucket.For example, src1:dst1,src2:dst2:/path/rules2.json ]
isContentInline Set to true if you wish to maintain only a single copy of the CA certificate chain and the replication rules file on a cluster instead of having copies of the CA certificate chain and the replication rules file on each node of the cluster where the mapr-och service is running. Default: false.

Examples

Example 1: Create a copier by the name ochtest106 on a node with FQDN as node.fqqddn.net for fabric to fabric object copying from one fabric to another fabric

CLI
maprcli och-instance create -name ochtest106 -node node.fqqddn.net \
  -isSQSListener false \
  -kafkaTopics "/var/objstore/event-stream:event-topic" \
  -srcAccessKey <source fabric access key> \
  -srcSecretKey <source fabric secret key> \
  -srcEndpoint <source fabric endpoint>:9000 \
  -srcChainCA /tmp/<security certificate file path for source fabric> \
  -dstAccessKey <destination fabric access key> \
  -dstSecretKey <destination fabric secret key> \
  -dstEndpoint <destination fabric endpoint>:9000 \
  -dstChainCA /tmp/<security certificate file path for destination fabric> \
  -bucketMap srcbkt106:destbkt106 \
  -json

REST

curl -u <username> -X POST https://server.sj.us:8443/rest/och-instance/create?name="ochtest106"&node="node.fqqddn.net"isSQSListener="false"&kafkaTopics="/var/objstore/event-stream:event-topic"&
srcAccessKey="<source fabric access key>>"&srcSecretKey="<srcSecretKey>"&srcEndpoint="<source fabric endpoint>"&srcChainCA="/tmp/<security certificate file path for source fabric>"&dstAccessKey="<destination fabric access key>"&dstSecretKey="<destination fabric secret key>"
&dstEndpoint="<destination fabric endpoint>:9000"&dstChainCA="/tmp/<security certificate file path for destination fabric>"&dstAccessKey="<dstAccessKey>"&dstSecretKey="<dstSecretKey>"&dstChainCA="/tmp/chain-ca-dest.pem"&bucketMap="<DestinationBucketsrcbkt106:destbkt106>"

Example 2: Create copier with S3 ARN

CLI
mmaprcli och-instance create
 -name nochawsinst2 \
 -isSQSListener true \
 -sqsQueueUrl https://<sqsQueueUrl>  \
 -sqsAwsRegion <sqsAwsRegion>  \
 -sqsAwsAccessKey <sqsAwsAccessKey> \
 -sqsAwsSecretKey <sqsAwsSecretKey> \
 -srcIsAWS true \
 -srcAwsRegion <sqsAwsRegion>  \
 -srcAwsAccessKey <sqsAwsAccessKey> \
 -srcAwsSecretKey <sqsAwsSecretKey> \
 -srcRoleArn <srcRoleArn>\
 -dstEndpoint https://<Destination host>:9000 \
 -dstAccessKey <dstAccessKey> \
 -dstSecretKey <dstSecretKey> \
 -dstChainCA /tmp/chain-ca-dest.pem \
 -bucketMap <AWSBucket>:<DestinationBucket> \
 -json

REST

curl -u <username> -X POST https://server.sj.us:8443/rest/och-instance/create?name="nochawsinst2"&isSQSListener="true"&sqsQueueUrl="https://<sqsQueueUrl>"&
sqsAwsRegion="<sqsAwsRegion>"&sqsAwsAccessKey="<sqsAwsAccessKey>"&sqsAwsSecretKey="<sqsAwsSecretKey>"&srcIsAWS="true"&srcAwsRegion="<sqsAwsRegion>"&srcAwsAccessKey="<sqsAwsAccessKey>"
&srcAwsSecretKey="<sqsAwsSecretKey>"&srcRoleARN"<srcRoleArn>"&dstEndpoint="https://<Destination host>:9000"&dstAccessKey="<dstAccessKey>"dstSecretKey="<dstSecretKey>"&dstChainCA="/tmp/chain-ca-dest.pem"&bucketMap="<AWSBucket>:<DestinationBucket>"

Example 3: Create copier for object replication/copying from AWS S3 to Data Fabric

CLI

maprcli och-instance create
        -name ochawsinst1 \
 -isSQSListener true \
 -sqsQueueUrl https://<sqsQueueUrl> \
        -sqsAwsRegion <sqsAwsRegion> \
 -sqsAwsAccessKey <sqsAwsAccessKey> \
 -sqsAwsSecretKey
        <sqsAwsSecretKey> \
 -srcIsAWS true \
 -srcAwsRegion <sqsAwsRegion>  \
 -srcAwsAccessKey <sqsAwsAccessKey> \
        -srcAwsSecretKey <sqsAwsSecretKey> \
 -dstEndpoint https://<Destination host>:9000 \
        -dstAccessKey <dstAccessKey> \
 -dstSecretKey <dstSecretKey> \
 -dstChainCA
        /tmp/chain-ca-dest.pem \
 -bucketMap <AWSBucket>:<DestinationBucket> \
 -json

REST

curl -u <username> -X POST https://server.sj.us:8443/rest/och-instance/create?name="ochawsinst1"&isSQSListener="true"&sqsQueueUrl="https://<sqsQueueUrl>"&
sqsAwsRegion="<sqsAwsRegion>"&sqsAwsAccessKey="<sqsAwsAccessKey>"&sqsAwsSecretKey="<sqsAwsSecretKey>"&srcIsAWS="true"&srcAwsRegion="<sqsAwsRegion>"&srcAwsAccessKey="<sqsAwsAccessKey>"
&srcAwsSecretKey="<sqsAwsSecretKey>"&dstEndpoint="https://<Destination host>:9000"&dstAccessKey="<dstAccessKey>"dstSecretKey="<dstSecretKey>"&dstChainCA="/tmp/chain-ca-dest.pem"&bucketMap="<AWSBucket>:<DestinationBucket>"

Example 4: Create copier for object replication/copying from AWS S3 to Data Fabric(between SQS ARN and S3 ARN)

maprcli och-instance create
        -name ochawsinst4 \
 -isSQSListener true \
 -sqsQueueUrl https://<sqsQueueUrl> \
 -sqsAwsRegion <sqsAwsRegion>  \
 -sqsAwsAccessKey <sqsAwsAccessKey> \
 -sqsAwsSecretKey <sqsAwsSecretKey> \
 -sqsRoleArn <sqsRoleArn>\
 -srcIsAWS true \
 -srcAwsRegion <sqsAwsRegion> \
 -srcAwsAccessKey <sqsAwsAccessKey> \
 -srcAwsSecretKey <sqsAwsSecretKey>\
 -srcRoleArn <srcRoleArn> \
 -dstEndpoint https://<Destination host>:9000 \
 -dstAccessKey <dstAccessKey> \
 -dstSecretKey <dstSecretKey> \
 -dstChainCA /tmp/chain-ca-dest.pem \
 -bucketMap <AWSBucket>:<DestinationBucket> \
 -json

REST

curl -u <username> -X POST https://server.sj.us:8443/rest/och-instance/create?name="ochawsinst4"&isSQSListener="true"&sqsQueueUrl="https://<sqsQueueUrl>"&
sqsAwsRegion="<sqsAwsRegion>"&sqsAwsAccessKey="<sqsAwsAccessKey>"&sqsAwsSecretKey="<sqsAwsSecretKey>"&sqsRoleARN="<sqsRoleArn>"&srcIsAWS="true"&srcAwsRegion="<sqsAwsRegion>"&srcAwsAccessKey="<sqsAwsAccessKey>"
&srcAwsSecretKey="<sqsAwsSecretKey>"&srcRoleARN="<srcRoleArn>"&dstEndpoint="https://<Destination host>:9000"&dstAccessKey="<dstAccessKey>"dstSecretKey="<dstSecretKey>"&dstChainCA="/tmp/chain-ca-dest.pem"&bucketMap="<AWSBucket>:<DestinationBucket>"