Data Reads, Writes, and Recalls

Provides a synopsis of how data is read and written to a warm or cold tier.

Once offloaded to the storage tier, data is considered to be warm or cold on the storage tier, but the data can still be accessed (read, written, and recalled).

Read of Tiered Data

Depending on whether the standard volume data is outside the data-fabric cluster and in the cloud (cold tiering) or on the data-fabric cluster (warm tiering), data-fabric processes the request to read standard volume data and mirror volume data.

Data Fabric processes client requests to read standard volume data on the warm tier and the cold tier differently.
Warm Tier

When a client attempts to read, the read request is first sent to the front-end volume and if the data exists in the front-end volume, the data is returned from the front-end volume. If data is not in the front-end volume, the data is returned from the erasure coded volume.

Cold Tier

When a client attempts to read, the read request is first sent to the volume on the data-fabric cluster and if the data exists in the volume on the cluster, the data is returned from the volume. On the other hand, if the data was offloaded, the MAST Gateway recalls the data from the cold-tier to process the read request. See Recall of Tiered Data for more information on recalled data.

The following illustration shows a client sending the data read request first (#1) to the tiering-enabled volume and the response (#2) being served from the volume on the data-fabric cluster. Then (#3), data is offloaded to the back-end erasure coded volume (for Volume A) and to the cloud (for Volume B). When the client next sends a read request to the volume on the data-fabric cluster (#4), for:
  • Volume A, the MAST Gateway forwards the request to the back-end erasure-coded volume (#5) from where data is returned (#6) to the client.
  • Volume B, the MAST Gateway recalls the data (#5 and #6) from the cloud to the volume on the data-fabriccluster, from where data is returned (#7) to the client.


Data Fabric processes client requests to read mirror volume data on the warm tier and the cold tier differently.
Warm Tier

When a client attempts to read, the read request is first sent to the front-end volume and if the data exists in the front-end mirror volume, the data is returned from the front-end volume. If data is not in the front-end volume, the data is returned from the erasure coded volume.

Cold Tier

When a client attempts to read, the read request is first sent to the volume on the data-fabric cluster and if the data exists in the volume on the cluster, the data is returned from the volume. On the other hand, if the data was offloaded, the MAST Gateway recalls or fetches a copy of the data (from the tier) into an associated cache-volume, from where data is returned to the client. See Recall of Tiered Data for more information on recalled data.

The following illustration shows a client sending the read request first (#1) to the tiering-enabled mirror volume and the response (#2) being served from the volume on the data-fabric cluster. Then (#3), data is offloaded to the back-end erasure coded volume (for warm tier) and to the cloud (for cold tier). When the client next sends a read request to the volume on the data-fabric cluster (#4), for:
  • Warm tier, data is returned from the back end erasure-coded volume (#5).
  • Cold tier, data is recalled in the associated cache-volume (#5 and #6), from where data is returned to the client (#7).


The cache-volume named mapr.internal.cv.<volume-name>_<id> is created when the tiering-enabled mirror volume is created. Although it may not hold any data initially, a copy of the tiered data is fetched into the cache-volume whenever there is a read of the cold-tiered mirror volume data or explicit recall of (cold or warm) tiered mirror volume data. You can use the volume info command on the tiering-enabled mirror volume to get the offload and recall statistics, which are from the cache-volume, for the mirror volume.

The cache-volume has the same replication factor as the mirror volume (at the time of volume creation); changes to the mirror volume replication factor do not trigger a change to the replication factor of the associated cache-volume.

If the tiering-enabled mirror volume is deleted, the cache-volume is also deleted. If the tiering-enabled mirror volume is promoted to a read-write volume, the associated cache-volume is deleted.

Write on Tiered Data

When writes happen, if the write is:

  • An append, new data is offloaded when the data meets the criteria in the rule (associated with the volume) for offload.
  • A change to existing data (overwrite), the data is recalled to the data-fabric filesystem to allow the write to succeed and then offloaded when the data meets the criteria in the rule (associated with the volume) for offload. See Recall of Tiered Data for more information on recalled data.
NOTE
If cold data is accessed (read/written) frequently, I/O to that file may suffer large latencies. In such scenarios, recall the whole volume or the corresponding files.

Recall of Tiered Data

Offloaded data is automatically recalled when a client performs a read or overwrite on the data in the cold-tier, or when a client performs an overwrite on the data in the warm-tier. The MAST Gateway fetches a copy of the data to allow the operations to succeed. You can also manually trigger a recall of:

Based on the expiration time period set at the volume level for recalled data, recalled data is:
  • Offloaded again based on the rules if there are changes to the data.
  • Purged when the compactor runs if there are no changes to the data.

For a cold tiering volume, explicitly recall the volume before running any analytics jobs.

For mirror volumes, when you recall tiered data, data from the tier is recalled into an associated cache-volume, which is created at the time of the creation of the tiering-enabled mirror volume. For all explicit recall of warm-tiered data and explicit and automatic recalls of cold-tiered data, the MAST Gateway recalls data into the associated cache-volume. The data in the cache-volume is "hot" in the cluster, or available for reads, for the duration of the expiry-period. The recalled data is purged by the compactor when the expiration time that is set at the volume level is reached or has passed.

If the recall fails, CLDB retries the operation after some time. See Retrying Failed Operation for more information.