External KMIP Keystore Support

Both bare-metal and container-based HPE Ezmeral Data Fabric implementations include external KMIP Key Store (see External KMIP Keystore Overview; link opens in a new browser tab/window), but there are some key differences:

Bare-metal configurations configure KMIP using either configure.sh with the new HSM options (for a fresh installation) or the mrhsm utility (to modify or upgrade existing installations). The configure.sh script calls the mrhsm utility behind the scenes and will not generate CLDB and DARE keys when HSM is used. The encrypted KMIP configuration is created and then copied to all the CLDB and Zookeeper nodes in the cluster.
Kubernetes pods are ephemeral, and anything written to the ${MAPR_HOME}/conf/tokens directory does not persist after the pods are destroyed. The encrypted KMIP configuration must therefore be stored as a Kubernetes secret and mounted as a volume on the CLDB and Zookeeper nodes. The encrypted KMIP configuration is stored in CRs with appropriate labels so that Kubernetes will know where to mount the volume.

KMIP Deployment Workflow

To deploy the External KMIP Key Store with HPE Ezmeral Data Fabric:

Configure kubectl to point to your Kubernetes environment.
Bootstrap your Kubernetes environment by executing the following commands:
```
$ cd bootstrap
$ ./bootstrap.sh install
```
Set up the KMIP-enabled HSM by following the instructions in the appropriate KMIP Integration Guide (link opens in a new browser tab/window). You must obtain the HSM IP addresses and port numbers. You must also download the CA certificate, client certificate, and private client key. The currently supported HSMs are:
- Utimaco ESKM
- Vormetric DSM
- Gemalto SafeNet Keysecure
Use kubectl to install and configure container permissions on the Mac or Linux machine. You need permissions to access the container images in gcr.io/mapr-252711. Click here for information about the gcloud command (link opens an external website in a new browser tab/window).
Run the HSM setup script to configure KMIP for the cluster:
```
$ cd tools
$ ./setup-hsm.sh <cluster-name>
```
Deploy the generated CR from the tools/ directory to push the KMIP secret to the hpe-secure namespace via the interface (not manually):
```
$ kubectl apply -f hsm_config/<cluster-name>/hsmconfig-<cluster-name>.yaml
```

Deploy the dataplatform CR:

$ kubectl apply -f <path-to-dataplatform-CR>

The remaining steps follow the standard deployment workflow.

Example Setup

The following sample session illustrates a two-node Gemalto SafeNet KeySecure KMIP configuration for a cluster named demo.cluster.com:

tools % ./setup-hsm.sh demo.cluster.com                                 
Configuring HSM for cluster demo.cluster.com
latest: Pulling from mapr-252711/hsmsetup-6.2.0
Digest: sha256:22451ee67f8d15c083410d288298a90ec9cf138f0456ece60f559752f0521fc9
Status: Image is up to date for gcr.io/mapr-252711/hsmsetup-6.2.0:latest
gcr.io/mapr-252711/hsmsetup-6.2.0:latest
Configuring HSM for demo.cluster.com. When you are done, type "exit" to
return to the rest of the HSM setup.
[root@88df33691bff mapr]# /opt/mapr/server/mrhsm info -slots
Slot 0
    Slot info:
        Description:          MapRHSM slot ID 0x0
        Manufacturer ID:      HPE MapR-HSM
        Token present:        yes
    Token info:
        Manufacturer ID:      HPE MapR-HSM
        Model:                MapRHSM
        Serial number:
        Initialized:          no
        User PIN initialized: no
        Label:
[root@88df33691bff mapr]# /opt/mapr/server/mrhsm init -label "Gemalto SafeNet KeySecure"
Enter SO PIN (4-255 characters): ****
Please reenter SO PIN: ****
[root@88df33691bff mapr]# /opt/mapr/server/mrhsm set -ip "10.10.30.129,10.10.30.182" -cacert /opt/mapr/hsmsetup/CA.pem -clientcert /opt/mapr/hsmsetup/client.pem -clientkey /opt/mapr/hsmsetup/key.pem
Enter SO PIN: ****
[root@88df33691bff mapr]# /opt/mapr/server/mrhsm enable -dare
Enter SO PIN: ****
Obtained cluster name demo.cluster.com from mapr-clusters.conf
Enabling MapR HSM on cluster demo.cluster.com
Successfully generated Core KEK, UUID A39276162C3BFCFD972AF9ED354CE53A8351932EA6772B5790939BC339E8C139
SHA-256 checksum for Core KEK is 5122D496285E1A768D201727521C028E938E9606D54C26529ECF07AF8307B789
Successfully generated Common KEK, UUID 1C7A23C33D6797EFC35040A4C01646E4350C2C986FD24678C47C8EAAAA0C8FA7
SHA-256 checksum for Common KEK is 5E579DE1029486FE1A09190C77374AE1C034ADD0A22F2F6C5439854A68D8980D
No CLDB key found on host
SHA-256 checksum for CLDB key is 3AC9E80BFEA952FAEC181D837D5B83E5C38A712A913742D35A4B9E0AEA17177F
Successfully set encrypted CLDB key in KMIP configuration
SHA-256 checksum for DARE key is 672B1545C131B7B1C76E5776A18E2BBEECFE318534AAE1892975AD79D077C28B
Successfully set encrypted DARE key in KMIP configuration
##############################################################################
Copy the entire contents of the KMIP token directory /opt/mapr/conf/tokens to
all CLDB and Zookeeper nodes. All files in /opt/mapr/conf/tokens must be owned
by the mapr user and mapr group.
##############################################################################
[root@88df33691bff mapr]# exit
exit
Removing  container 
Continue to create HSM secret? (y/n) [y]:
Generating Kubernetes custom resource for KMIP configuration ...
 
The KMIP configuration generated for this cluster are available at: /Users/testuser/builds/private-kubernetes/tools/./hsm_config/demo.cluster.com/hsmconfig-demo.cluster.com.yaml
Please copy them to a machine where you can run the following command:
  kubectl apply -f /Users/testuser/builds/private-kubernetes/tools/./hsm_config/demo.cluster.com/hsmconfig-demo.cluster.com.yaml
tools % kubectl apply -f /Users/testuser/builds/private-kubernetes/tools/./hsm_config/demo.cluster.com/hsmconfig-demo.cluster.com.yaml
secret/hsmconfig-demo.cluster.com created

Example Notes

In this example:

The hsm_config directory is created in the same directory as the setup-hsm.sh script, which is in the tools/ directory.
The customized container used to run the mrhsm utility to create the encrypted KMIP configuration contains an /opt/mapr/hsmsetup directory that is mapped to the hsm_config/<cluster-name> directory. Certificates and private keys required for the KMIP configuration can be copied to this directory to make it accessible to the container.
If you exit the container prior to creating an enabled KMIP configuration, then the configuration will be saved in hsm_config/<cluster-name>/tokens where you can access it in future sessions. If you need to set new CA or client certificates or private keys, or re-key the core KEK, then you can also use the setup-hsm.sh script.
The Kubernetes secret for the KMIP configuration is in a compressed .tar archive that contains the contents of the ${MAPR_HOME]/conf/tokens directory.
For new installations, the init container does not generate the CLDB and DARE keys if it finds the HSM secret in the hpe-secure namespace. If the HSM secret exists in hpe-secure, then the init container pulls it from the hpe-secure namespace and puts it in the dataplatform namespace. The CLDB and Zookeeper pods retrieve this secret and extract the .tar archive to the ${MAPR_HOME}/conf/tokens directory. This allows the CLDB, MFS, and ZooKeeper services to retrieve the CLDB and DARE keys from the HSM upon startup.
The YAML custom resource used by kubectl to push the Kubernetes secret to the hpe-secure namespace is stored in hsm_config/<cluster-name>/hsmconfig-<cluster-name>.yaml.

This example shows the sample contents of the hsm_config directory for two clusters named new.cluster.com and demo.cluster.com:

tools % find hsm_config -print
hsm_config
hsm_config/new.cluster.com
hsm_config/new.cluster.com/tokens
hsm_config/new.cluster.com/tokens/06e32ec9-9a40-3951-1fd2-23ae6332be79
hsm_config/new.cluster.com/tokens/06e32ec9-9a40-3951-1fd2-23ae6332be79/token.lock
hsm_config/new.cluster.com/tokens/06e32ec9-9a40-3951-1fd2-23ae6332be79/token.object
hsm_config/new.cluster.com/tokens/06e32ec9-9a40-3951-1fd2-23ae6332be79/generation
hsm_config/new.cluster.com/tokens/mrhsm.conf
hsm_config/demo.cluster.com
hsm_config/demo.cluster.com/hsmconfig-demo.cluster.com.tgz
hsm_config/demo.cluster.com/key.pem
hsm_config/demo.cluster.com/README
hsm_config/demo.cluster.com/client.pem
hsm_config/demo.cluster.com/CA.pem
hsm_config/demo.cluster.com/req.pem
hsm_config/demo.cluster.com/hsmconfig-demo.cluster.com.yaml
hsm_config/demo.cluster.com/tokens
hsm_config/demo.cluster.com/tokens/5f2356db-1201-859e-b316-66a512b25fbb
hsm_config/demo.cluster.com/tokens/5f2356db-1201-859e-b316-66a512b25fbb/token.lock
hsm_config/demo.cluster.com/tokens/5f2356db-1201-859e-b316-66a512b25fbb/token.object
hsm_config/demo.cluster.com/tokens/5f2356db-1201-859e-b316-66a512b25fbb/generation
hsm_config/demo.cluster.com/tokens/mrhsm.conf

Rekeying and Modifying the KMIP Configuration

You can use the setup-hsm.sh script and mrhsm rekey command to re-key the Core KEK for an existing configuration. After exiting the setup-hsm.sh script, modify the custom resource definition to change the namespace from hpe-secure to dataplatform:

$ more hsmconfig-demo.cluster.com.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: hsmconfig-demo.cluster.com
  namespace: demo-cluster.com
type: Opaque
data:
  MAPR_KMIP:     "H4sIAIJHfV8AA+2cXainW13HJ0uyJ6TsQrKQJkrqwjOu9xehi/WqRzuj6BHMGxnn7JPmOWfwOJKCVEYUdCWUl5ZIlBQYYRB001XSRTcVCUFgFxlYBhlhYBf1+f5n+8JonrOPzpzy7AUzs/d////Ps9bv5fuy1rPn9q13nj3x7ldcuZfDGJNjvHr6N6XTv8aFO/+ej6vWZ59d8NHlq8YG49
…

NOTE In the above example, change the namespace from demo-cluster.com to the actual cluster namespace.

Next, execute the following command to update the secret in the dataplatform namespace:

$ kubectl apply -f hsmconfig-demo.cluster.com.yaml

if you need to use the new key immediately, then restart the CLDB and Zookeeper pods. However, the original Core KEK will still be available and you can wait until the next CLDB and Zookeeper restart.

If you change the HSM configuration (such as changing the HSM server IP addresses or replacing an expired client certificate), then the original HSM servers may no longer be accessible or the client certificates may no longer be valid. Restart the CLDB and Zookeeper pods to update the HSM secret in the dataplatform namespace.

Configuring KMIP for an Existing Cluster

To configure KMIP for an existing cluster that already has the CLDB and DARE master key:

Configure the KMIP-enabled HSM per the instructions in the appropriate KMIP Integration Guide (link opens in a new browser tab/window; Utimaco ESKM, Vormetric DSM, and Gemalto SafeNet KeySecure are currently supported).
Obtain the HSM IP addresses and KMIP port number.
Download the CA certificate, client certificate, and private client key.
Use kubectl to install and configure container permissions on the Mac or Linux machine. You need permissions to access the container images in gcr.io/mapr-252711. Click here for information about the gcloud command (link opens an external website in a new browser tab/window).
Run the HSM setup script to configure KMIP for the cluster:
```
$ cd tools
$ ./setup-hsm.sh <cluster-name>
```
As described in Example Notes, copy the client credentials such as the CA certificate, client certificate, and client private key to the shared hsm_config/<cluster-name> folder to allow access by the mrhsm utility when running mrhsm set.
Download the CLDB key ${MAPR_HOME}/conf/cldb.key and DARE master key (if applicable) ${MAPR_HOME}/conf/dare.master.key from one of the CLDB pods in the cluster, and then place then in a location that is accessible to the container that is started within setup-hsm.sh. Be sure to also back up the CLDB and DARE master keys in a safe place.

For example, if you are in the tools/ directory where the setup-hsm.sh script resides:
```
$ kubectl cp -n demo.cluster.com cldb-0:/opt/mapr/conf/cldb.key hsm_config/chyelindarenohsm/cldb.key
$ kubectl cp -n demo.cluster.com cldb-0:/opt/mapr/conf/dare.master.key hsm_config/demo.cluster.com/dare.master.key
```

Copy the CLDB key and DARE master key (if applicable) that you downloaded in Step 7 to /opt/mapr/conf in the container. For example, within the container:

[root@95b57c45e5d6 conf]# ls ../hsmsetup 
CA.pem            cldb.key    dare.master.key   tokens
Client.pem   key.pem
[root@95b57c45e5d6 conf]# cp ../hsmsetup/cldb.key .
[root@95b57c45e5d6 conf]# cp ../hsmsetup/dare.master.key .
[root@95b57c45e5d6 conf]# pwd
/opt/mapr/conf
[root@95b57c45e5d6 conf]# ls
cldb.key  daemon.conf  dare.master.key    mapr-clusters.conf  maprhsm.conf  tokens
[root@95b57c45e5d6 conf]#

Execute the /opt/mapr/server/mrhsm utility in the container within the setup-hsm.sh script.. If the mrhsm utility finds existing CLDB and DARE master keys in /opt/mapr/conf, then the mrhsm enable command will import the existing CLDB and DARE master keys into the HSM instead of generating new keys in the HSM. Be sure to:
- Execute mrhsm enable -dare if the DARE master key exists
- Omit the -dare option if DARE is not enabled.
Modify the generated CR in the tools/hsm_config/<cluster-name> directory to change the namespace from hpe-secure to the cluster name. For example, if the cluster name is demo.cluster.com, then you would modify the CR as follows to push the KMIP secret to the demo.cluster.com namespace:
```
metadata:
  name: hsmconfig-demo.cluster.com
  namespace: demo.cluster.com
```
Deploy the generated CR from the tools/ directory to push the KMIP secret to the Data Fabric cluster namespace. For example, if the cluster name is demo.cluster.com:
```
$ cd hsm_config/demo.cluster.com
$ kubectl apply -f hsmconfig-demo.cluster.com.yaml
```
Verify that the secret is in the cluster namespace. For example, for the demo.cluster.com cluster:
```
% kubectl get secrets -n demo.cluster.com | grep hsmconfig
hsmconfig-demo.cluster.com     Opaque                 1      18m
```
NOTE The KMIP secret will be retrieved from the cluster secret namespace and configured into the ${MAPR_HOME}/conf/tokens directory during the startup procedure so that the KMIP feature will be enabled the next time the CLDB and ZK pods restart.
Restart the CLDB and Zookeeper pods by executing the edf update cluster command in the admincli-0 pod in /usr/bin. For example:
```
kubectl exec -it admincli-0 -n <pod-namespace> /bin/bash
edf update cluster
```

Verify that each of the CLDB and ZK pods are now HSM-enabled by executing the mrhsm info -kmip command on each pod and verifying that the Enabled field is set to Yes. For example, for pod cldb-0:

% kubectl exec cldb-0 -n demo.cluster.com -- /opt/mapr/server/mrhsm info -kmip 
Displaying information for KMIP token with serial ff989992b11f9ed3
KMIP Configuration Version 1
-----------------------------
CLDB:
...
Enabled             : Yes