S3 Federation in Global Namespace

Provides an overview of S3 federation in the global namespace

Data Fabric provides a native implementation of the S3 object store for object management. You can access data from multiple fabrics with the native S3 object store implementation.

Additionally, Data Fabric provides a mechanism to access data that is stored on S3-compliant object stores from vendors other than HPE such as Amazon Web Services (AWS), Google Cloud Platform (GCP), , Scality, WEKA, VAST, to name a few . A third-party object store that is managed by vendors other than HPE is referred to as an external S3 object store (external S3 server).

S3 federation is a federation of one or more Data Fabric native S3 servers and/or one or more external S3 servers in the global namespace.

You can access remote S3 servers imported into your global namespace from any fabric in the same global namespace, by obtaining the access points to the object stores.

All buckets and objects from your native and external S3 servers can be accessed on the global namespace via the Data Fabric UI.

Data Fabric acts as the intermediary between a S3 client and an external S3 server that is imported into the global namespace. Data Fabric forwards requests coming from S3 clients to the external S3 server. The responses from the external S3 server are transported back to the requesting S3 clients via Data Fabric.

By default, Data Fabric works in forwarding mode with external S3 servers in their communication with S3 clients via Data Fabric. Data Fabric forwards the S3 client request to the external S3 server by using the secret key-access key pair that has been used to import the S3 server into the global namespace. The forwarding mechanism that happens during the S3 client-external S3 server communication consumes CPU and memory resources on Data Fabric. This can impact the Data Fabric performance, depending on the use case for frequency of communication between the S3 client and external S3 server.

Data Fabric can work in redirection mode for S3 client-native S3 server communication with the clustergroup s3gns command. Data Fabric redirects an incoming S3 client request made to a native S3 server on another fabric in the same global namespace. Once the redirection is successful,the communication between the S3 client and the native S3 server on the other fabric takes place directly. In case of redirection, there is no impact on Data Fabric performance.