Object Copier Hub

Event-based object copier supporting data transfer and deletion between any S3-compatible sources and destinations.

Key capabilities

Following are the key capabilities of the Object Copier Hub service:

  • Consumes notifications from AWS SQS for cloud S3 buckets or MapR Streams/Kafka for Data Fabric buckets.

    Supports standard queues to move objects from AWS S3 to Data Fabric.

  • Copies objects between Data Fabric buckets that reside on the same fabric or between buckets across fabrics.

  • Copier tasks are auto load-balanced.

  • Supports delete operations on destination buckets.

  • Large object multipart copy is supported.

  • Allows multiple independent copiers to run concurrently. Each copier has its own source and destination bucket mapping, listener, and worker pool.

  • Listens and operates on an HTTPS with mutual TLS (mTLS) port (default 8080). Both server and client authenticate using PKCS12 certificates. All data stored is encrypted with AES-256-GCM

  • Uses REST APIs and maprcli commands to carry out the operations related to data movement across buckets

Things to note

  • Set EnableMultipart: true in the create-copier request to enable large object multipart copy.

  • Each copier runs a single event listener consumer and a single copier worker pool. To consume from multiple queues, create a separate copier configuration for each queue.

  • Each event queue must be configured with exactly one copier such that there is one consumer per producer. One copier can manage multiple source-destination bucket pairs.

  • The nodes running the mapr-och service register with CLDB.

  • CLDB performs load balancing by auto-placing new copiers on the least-loaded active OCH node, and rebalances the existing copiers. CLDB automatically distributes copiers across available nodes to maintain a balanced workload. Automatic placement and rebalancing apply only to copiers that are not explicitly assigned to a node.
  • Warden manages OCH like any other Data Fabric service ny using warden.och.conf and the standard mapr-och init script.
  • You must have fc (full control) and a (admin) permissions to be able to use the Object Copier Hub service. You must have access to the /tmp directory to be able to access the security CA certificate chain and the replicate rule file.