Deploying the Model Management Service

This topic describes steps to install the Model Management Service for HPE Ezmeral Runtime Enterprise deployments that implement HPE Ezmeral ML Ops. While multiple tenants are able to share a model registry and experiment tracker at the cluster level, the Model Management Service employs rule-based access controls to ensure that users can access only their own models and metadata. The Model Management Service is required for Project Members to run Experiments.

Deploying the Model Management Service

NOTE The Model Management Service is required for Project Members to create Experiments.

Prerequisites:

Required access rights: You must have bucket creation rights.
An object store for storing model artifacts. You can use HPE Ezmeral Runtime Enterprise Data Fabric Object Store, or an object store from another source such as AWS S3.
For information on Data Fabric Object Store, see Object Store (S3 Gateway) Overview.
A MySQL RDBMS for storing experiment runs. For best results, Hewlett Packard Enterprise recommends using the HPE Ezmeral Runtime Enterprise Shared RDBMS feature (see Installing Shared RDBMS).
NOTE HPE Ezmeral Runtime Enterprise does not support the Shared RDBMS feature on CentOS. If you are using CentOS as your host OS, you can use a MySQL RDBMS other than Shared RDBMS to store experiment runs.

Installation steps:

Download the application bundle on the Kubernetes master node:

# wget https://ezml-release.s3.amazonaws.com/5.6.0/model-mgmt-ere560.tar.gz

# tar -xvzf model-mgmt-ere560.tar.gz && cd model-mgmt

(Optional): If you are using Data Fabric Object store, you can create a secure connection to the Object Store by copying the certificates from the Data Fabric master:
```
# scp root@<DF-ClusterMaster-IP>:/opt/mapr/conf/ca/chain-ca.pem ~/model-mgmt/mlflowtrack/
```
Log in to the Kubernetes master node and execute the install script:
```
# sh ezml_model_mgmt_install.sh
```
The following information is required for installation. If the database is hosted on the same cluster you are installing the Model Management Service on, the script fetches the database details automatically.
```
Below Object store configuration required to install the ModelMGMT.
   object_store_host:
   object_store_access_key:
   object_store_secret_key:
Press enter to continue...

Press Enter to continue with installation.
```
Press Enter to continue with installation.
The install script scans for a MySQL endpoint in an existing cluster.
- If the MySQL endpoint is found, the script proceeds with this endpoint, and asks whether or not to use a signed certificate for communication:
```
MYSQL end point identified in ezmysql NameSpace:
  mip-bd-vm184.mip.storage.hpecorp.net:10017

  Proceeding with above mysql endpoint.....
```
- If no MySQL endpoint is found, the script asks for a MySQL hostname and port number. After you have entered the hostname and port number for a MySQL endpoint, the script asks for the database password:
```
Mysql operator is not available in ezmysql namespace. Please 
provide the endpoint. 
Enter the DB host name: 
mip-bd-vm1094.example.net 
Enter the DB port Number: 
10006

Enter the PASSWORD for database user: 
Note: password will be hidden when typing
```
Select the communication type (secure or insecure):
- To create a secure connection to the Data Fabric Object Store with certificates, enter Y:
```
Are you using the self signed certificate for communication (Y/N):
Y
```
  The script prompts you to enter your certificate file path:
```
Then provide absolute path for self signed certificate.
  Please provide the Certificate file(*.pem) with absolute path.
```
  NOTE When creating a secure connection, copy the certificate file from the Data Fabric master (/opt/mapr/conf/ca/chain-ca.pem) to the local path. For example:
```
scp root@<DF-ClusterMaster-IP>:/opt/mapr/conf/ca/chain-ca.pem <local-path>
```
- To create a secure connection to the Data Fabric Object Store without certificates, or to use a different object store type (for example, AWS S3), enter N:
```
Are you using the self signed certificate for communication (Y/N):
N
```

The script asks for the Object Store bucket configuration:

Please enter the Object Store bucket configuration. 
Below is an example for these entries 

object_store_host: https://XXXXXXXXXXXXX:9000 
object_store_access_key: XXXXXXXXXXXXX 
object_store_secret_key: XXXXXXXXXXXXX 

Bucket name: ezmodel-mgmt-k8s-4 
Enter the object_store_host 
URL: https://m2-bd-vm2118.example.net:9000 
Enter the object_store_access_key: 
Note: Access_key will be hidden when typing 
Enter the object_store_secret_key: 
Note: Secret_key will be hidden when typing

After entering the required information, the script begins installation:

Installing the mlflowtrack in namespace ezml-model-mgmt .... 

Installing the MODELMGMT in namespace ezml-model-mgmt .... 

ModelMGMT installation is in Progress ......!!!! 

ModelMGMT installation has been completed successfully. 
Below are endpoint & bucket that will be used to interact with Model Mgmt 
MLFLOW_S3_ENDPOINT_URL: s3://ezmodel-mgmt-k8s-4 
MODELMGMT_BACKEND_URL: http://mip-bd-vm1094.example.net:10035 
*** Please apply the generated secret model-mgmt-secret.yaml to tenant namespace 

Usage: kubectl create -f model-mgmt-secret.yaml -n <tenant-namespace>

The script creates the YAML file model-mgmt-secret.yaml for the secret. Use the following command to create the secrets with tenant-namespace:
```
# kubectl create -f model-mgmt-secret.yaml -n <tenant-namespace>
```