Migrating S3 buckets to Inline Erasure Coding (IEC) buckets

Describes how to convert existing scheduled Erasure Coding (EC) S3 buckets to Inline Erasure Coding (IEC) buckets using mossdb and manual volume offload.

Starting with Data Fabric 8.2.0, you can migrate existing scheduled Erasure Coding (EC) buckets to Inline Erasure Coding (IEC) buckets without deleting or re-creating the buckets.

After a bucket is converted to IEC, objects uploaded to the bucket are written and erasure-coded inline during ingestion rather than waiting for scheduled tiering and compaction jobs.

Prerequisites and important considerations

  • Data Fabric version: This procedure is supported on Data Fabric 8.2.0 and later.
  • User privileges: All commands must be run as the mapr user (or executed with su mapr -c.
  • Shared-volume constraint: If an underlying volume hosts multiple S3 buckets, all buckets residing on that volume must be migrated simultaneously. You cannot migrate a subset of buckets on a shared volume.
  • Service impact: After initiating the migration, this procedure requires restarting the s3server service across all S3 server nodes in the cluster to update the running configuration (step 2 below). Because the s3server service may be unavailable during the restart, schedule this operation during this operation during a period of low production activity or an approved maintenance window.

Procedure

Migrating buckets to Inline Erasure Coding is a multi-step process that involves an initial preparation command, executing a few cluster commands, and a finalization command.

Step 1: Initiate the migration with mossdb.

Run the migrateToInstantEC command against the target bucket (or comma-separated list of buckets sharing the same volume):
su mapr -c "/opt/mapr/server/tools/mossdb manage migrateToInstantEC -buckets <bucket_name_or_list>"
  • For a single bucket on a dedicated volume:
    su mapr -c "/opt/mapr/server/tools/mossdb manage migrateToInstantEC -buckets bk1"
  • For multiple buckets sharing a volume:
    su mapr -c "/opt/mapr/server/tools/mossdb manage migrateToInstantEC -buckets \"bk22,bk23,bk24,bk25,bk26\""
NOTE

The command outputs the underlying volume name (for example, mapr.s3bucketVol.00000001) and provides the exact manual steps required for Steps 2–5.

Step 2: Restart the S3 server service.

Restart the s3server service across all S3 nodes in your cluster so that nodes pick up the inline EC configuration:
maprcli node services -name s3server -action restart -nodes <node1,node2,...>

Step 3: Trigger a manual volume offload.

Run a manual offload on the underlying volume identified in Step 1:
maprcli volume offload -name <volume_name> -json
Example:
maprcli volume offload -name mapr.s3bucketVol.00000001 -json

Step 4: Monitor offload completion.

Check the offload job status and ensure the operation completes with a status of "Success:
maprcli volume tierjobstatus -name <volume_name> -json

Example

maprcli volume tierjobstatus -name mapr.s3bucketVol.00000001 -json

Do not proceed to Step 5 until the offload state reports Success.

Step 5: Finalize and confirm the migration.

Re-run the migrateToInstantEC command with the postmigrationstepexecuted=true parameter:
su mapr -c "/opt/mapr/server/tools/mossdb manage migrateToInstantEC -buckets <bucket_name_or_list> -postmigrationstepexecuted=true"
  • Single bucket example:
    su mapr -c "/opt/mapr/server/tools/mossdb manage migrateToInstantEC -buckets bk1 -postmigrationstepexecuted=true"
  • Shared volume example:
    su mapr -c "/opt/mapr/server/tools/mossdb manage migrateToInstantEC -buckets bk22,bk23,bk24,bk25,bk26 -postmigrationstepexecuted=true"
Expected output:
Volume <volume_id> migration successful.

Service restart, event recording, and background offload activation

Pausing client ingestion is not required during migration. Understanding how the service restart and finalization steps interact ensures continuous ingestion without stranded data:
  • What restarting s3server does:
    Restarting s3server causes the S3 server to reload configuration and begin recording event metadata for newly written objects into the event tracking database so they can be processed by the background offloader.
    • Newly ingested objects are still written using standard replication (such as 3-way replication / RF3) per the volume policy - they are not immediately converted to Erasure Coding inline.
    • The automated background offloader process itself is not activated until the final migration step (re-running mossdb manage migrateToInstantEC with -postmigrationstepexecuted=true).
  • Continuous ingestion during a rolling restart:
    If clients ingest data through a load balancer while you perform a rolling restart across S3 server nodes:
    • Writes routed to already restarted nodes have their object events recorded for future background offload processing.
    • Writes routed to nodes awaiting restart continue writing via standard volume replication without event recording.
  • Handling non-offloaded data:

    Any data written to nodes before they are restarted—along with pre-existing data in the volume—is swept up and moved to backend Erasure Coding storage by the manual offload step (maprcli volume offload). Once that offload succeeds and you execute the final mossdb manage migrateToInstantEC ... -postmigrationstepexecuted=true command, the background offloader is fully activated to process all recorded data going forward.

Command syntax reference: mossdb manage migrateToInstantEC

Usage: /opt/server/tools/mossdb manage migrateToInstantEC -buckets="" [-postmigrationstepexecuted=true|false]
Parameter Type Required? Description
-buckets String Yes Comma-separated list of bucket names to migrate. Bucket names should be enclosed in double quotes if multiple buckets are listed. All buckets specified must reside on the same underlying volume.
-postmigrationstepexecuted Boolean Optional (Default: false) Set to true (using =true) after completing the s3server restart, manual offload, and tierjobstatus verification to finalize the migration.