Adding an Object Copier Hub Rule

Describes the procedure to add an Object Copier Hub rule by using the Data Fabric UI.

Prerequisites

The JSON file for rules exist at the specified path on all fabric nodes running the Object Copier Hub service.

About this task

You can define an Object Copier Hub (OCH) rule to move data from a bucket on a source S3 object store to a bucket on a destination S3 object store. You can configure events to copy objects (data) from one or more source buckets on the source S3 object store to one or more destination buckets on the source and destination S3 object stores.

The rules to copy data from the bucket on the source S3 object store (that is, Amazon S3 or fabric) to the bucket on the destination S3 object store (that is, a fabric) must be written in JSON format.

See Replication Rules JSON Format for a sample JSON rule file for data movement from a Amazon S3 bucket or a Data Fabric bucket to a bucket on another fabric.

IMPORTANT
When you specify the Rule file path for OCH bucket mappings on the Data Fabric UI, ensure that the replication rules JSON file exists in the /tmp folder on all cluster nodes, along with the security CA certificate chain file. The Data Fabric UI does not verify the existence of the specified file at the path mentioned by you. If the rules file path does not contain the rules JSON file, no rule would be applied to the object movement from the source bucket to the destination bucket.

Procedure

  1. Click Menu > Data Movement.
  2. Click the Management tab and click Add Rule.
  3. On General Settings, enter the Rule name.
  4. Select the fabric that is running the OCH service from the OCH fabric dropdown.
  5. Click Next.
  6. Enter the Source details: Select fabric to move objects from a bucket on a fabric. Alternatively, select AWS S3 to move objects from an AWS S3 bucket.
  7. Based on your selection of source type, do either of the following:
    • If you have selected fabric as the type, enter the following credentials:
      Table 1. Source fabric credentials
      Field Description
      Access key Fabric access key
      Secret key Fabric secret key
      Source endpoint The endpoint URL for the source fabric
      Source CA path absolute path to the security certificate file for the source fabric
      Kafka topics The Kafka topics on which to
    • If you have selected AWS S3 as the Type, enter the following source credentials:
      Table 2. Source AWS S3 credentials
      Field Description
      Access key AWS access key
      Secret key AWS secret key
      Region AWS region containing the bucket
      Role ARN unique Amazon Resource Name (ARN) that identifies an IAM role across all AWS accounts and regions. This is an optional field.
      Enter the SQS listener details:
      Table 3. SQS listener details
      Field Description
      SQS Queue URL The URL or the Amazon SQS queue that stores the data
      Access key AWS access key
      Secret key AWS secret key
      Region AWS region containing the bucket
      Role ARN (optional) unique Amazon Resource Name (ARN) that identifies an IAM role across all AWS accounts and regions. This is an optional field.
  8. click Next.
  9. Select the Destination fabric and enter the destination fabric details:
    Table 4. Destination fabric details
    Field Description
    Access key Access key for destination fabric
    Secret key Secret key for destination fabric
    Source endpoint Destination fabric endpoint URL.
    Source CA path absolute path to the certificate .pem file for the source fabric
    Kafka topics Comma-separated list of Kafka topics for data movement.
  10. Click Next.
  11. To add bucket mapping, use the following steps:
    1. Select the source bucket and the destination bucket under source and destination respectively.
    2. In the Rules file path, specify the path of the JSON file containing the rules to apply.
    3. Click Add to add the bucket mapping.
      The bucket mapping is added to the rule under the list of bucket mappings on the Bucket Mapping page.
      NOTE
      Click the bin icon to delete any bucket mapping that you may have erroneously entered or may not require. The bucket mapping deletion is an optional step.
    4. Click Add to add more such source and destination bucket mapping from the selected source and destination S3 object stores (AWS/fabric to fabric).
  12. Click Next.
  13. Review the rule on the Review page. You can click Back to navigate backward for the purpose of making corrections.
  14. Click Submit to save the rule.

Results

The rule is added to the list of Object Copier Hub rules on the Data Movement page. For the data movement to take place based on the bucket mappings that you have configured in the Object Copier Hub rule, you must enable the Object Copier Hub rule. See Enabling an Object Copier Hub Rule to enable the Object Copier Hub rule.