Prerequisites to use Object Copier Hub to move data from Fabric to Fabric

Lists the prerequisite settings to use the Object Copier Hub service to move data from a bucket in one fabric to a bucket in another fabric.

Overview

You can install the object copier hub service on a cluster node or an edge node. For the purpose of this topic, an edge node is any node on the network other than a cluster node. In other words, an edge node is any node on the network that does not have HPE Data Fabric installed.

ATTENTION
When the mapr-och service is installed on the edge node, the service communicates with the cluster by using a ticket. The prerequisites would vary depending on whether you are installing the mapr-och package on a cluster node or an edge node. Step 1 and step 2 given below are required only when you are installing the mapr-och service on an edge node. Step 3 - step 5 are required irrespective of the type of node on which you install the mapr-och service.

The following steps must be performed on the respective fabric node or edge node, to be able to use the Object Copier Hub, after installing the mapr-och package on the fabric node or edge node. See Installing the mapr-och package to install the package.

NOTE
Copy the clusters.conf from the Data Fabric cluster to the edge node where the object copier hub service is running.
scp /opt/mapr/conf/mapr-clusters.conf root@<och-node-ip>:/opt/mapr/conf/mapr-clusters.conf
  1. Copy source cluster mapr-clusters.conf to the node on which the mapr-och service is installed.
  2. Copy source cluster ticket maprticket_0 to the /tmp/ folder on the object copier hub node where the OCH is installed. Alternatively, export the ticket as MAPR_TICKETFILE_LOCATION variable. This is to enable listening on Data Fabric (Kafka) Streams.
    export MAPR_TICKETFILE_LOCATION=</path/to/ticketfile>
  3. Configure enviroment and proxy settings for Data Fabric.
  4. Enable streams events for Data Fabric.
  5. Change the consumer ACEs for default Data Fabric streams.

Configure environment and proxy settings

Run the following commands to configure the proxy settings for the mapr-och service.
vi /etc/systemd/system/mapr-och.service
[Service]
Environment="https_proxy=<enter https proxy host FQDN>:80"
Environment="http_proxy=<enter http proxy host FQDN>:80"
Environment="no_proxy=localhost,127.0.0.1"

Reload the systemd manager configuration and restart the mapr-och service.

systemctl daemon-reload
service mapr-och restart

Configure the port and log level settings in the OCH configuration file opt/mapr/conf/och.conf

//log level settings
och.log.level=info
och.port=8080

//port level settings
etcd.peer.port=2380
etcd.client.port=2379

Enable Streams events

To enable Streams events on Data Fabric, run the following commands:

maprcli config save -values {"s3.events.disable":"0"} -json

/opt/mapr/bin/mc event add <account alias>/<bucket name> arn:aws:sqs:us-west-2:444455556666:event-topic --event put
/opt/mapr/bin/mc event add <account alias>/<bucket name> arn:aws:sqs:us-west-2:444455556666:event-topic --event delete

Change the Consumer ACEs

To change the consumer ACEs for Data Fabric streams, run the following command as the mapr user:

maprcli stream edit -path /var/objstore/event-stream -consumeperm u:root -json