Tutorial: Serving a TensorFlow Model with KServing (Financial Series)

Prerequisites:
  • An Internet connection is required to download the dependencies needed for this tutorial. This tutorial is not available for Air Gapped environments.
  • If you have not done so already: Before beginning this tutorial, download the Kubeflow tutorials zip file, which contains sample files for all of the included Kubeflow tutorials.

Serving TensorFlow Model with KServing

  1. In a KubeDirector Jupyter notebook, create or upload the serving .yaml file financial-series-serving.yaml.
  2. Open the web terminal in the HPE Ezmeral Runtime Enterprise UI, or from the terminal within the KubeDirector notebook.
    NOTE By default, you cannot execute kubectl commands in a newly created KubeDirector notebook. To enable kubectl in a notebook, select one of the following methods:
    • Through the HPE Ezmeral Runtime Enterprise UI:
      1. In the HPE Ezmeral Runtime Enterprise UI, navigate to the Tenant section and initialize a web terminal with the corresponding button.
      2. Start a new Terminal session inside the KubeDirector notebook. Check that the files inside your KubeDirector notebook have the appropriate file permissions that allow you to work with them.
      3. Move all files you want to work with to the following path:
        /bd-fs-mnt/TenantShare
      4. You can now access the files inside the web terminal with kubectl.
    • From inside the KubeDirector notebook:
      1. To authorize your user inside the KubeDirector notebook, execute the following Jupyter code cell:
        from ezmllib.kubeconfig.ezkubeconfig import set_kubeconfig
set_kubeconfig()
      2. A prompt appears below the code cell you executed. Enter your user password in the prompt.
      3. kubectl is now enabled for your KubeDirector notebook. Start a Terminal session in the KubeDirector notebook to work with kubectl.
  3. In the terminal, apply the file:
    $ kubectl apply -f financial-series-serving.yaml
  4. Check that inference service, revision, and relative pod are created:
    $ kubectl get pods | grep finance-sample
NAME                                                              READY   STATUS    RESTARTS   AGE
finance-sample-predictor-default-d45t4-deployment-784457c4wjtl2   3/3     Running   0          63s
    $ kubectl get ksvc
NAME                               URL                                                         LATESTCREATED             LATESTREADY                              READY   REASON
finance-sample-predictor-default   http://finance-sample-predictor-default.nkili.example.com   finance-sample-predictor-default-9lrnm   finance-sample-predictor-default-9lrnm   True
    $ kubectl get revision
NAME                                     CONFIG NAME                        K8S SERVICE NAME                         GENERATION   READY   REASON
finance-sample-predictor-default-9lrnm   finance-sample-predictor-default   finance-sample-predictor-default-9lrnm   1       True
    $ kubectl get inferenceservices
NAME             URL                                                              READY   DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
finance-sample   http://finance-sample.<profile-name>.example.com/v1/models/finance-sample   True    100                     4h52m
  5. Check that virtual service is created:
    $ kubectl get virtualservices | grep finance-sample
finance-sample                          [kubeflow-gateway.kubeflow knative-serving/cluster-local-gateway]   [finance-sample.nkili.example.com finance-sample.nkili.svc.cluster.local]                                                        24m
finance-sample-predictor-default        [knative-serving/cluster-local-gateway kubeflow/kubeflow-gateway]   [finance-sample-predictor-default.nkili finance-sample-predictor-default.nkili.example.com finance-sample-predictor-default.nkili.svc finance-sample-predictor-default.nkili.svc.cluster.local]   24m
finance-sample-predictor-default-mesh   [mesh]                                                              [finance-sample-predictor-default.nkili finance-sample-predictor-default.nkili.svc finance-sample-predictor-default.nkili.svc.cluster.local]                                                      24m
  6. Create or upload the file kserving-request.py.
  7. Install the following Python dependencies:
    $ pip install requests lxml --user
  8. Launch kserving-request.py with the following options:
    $ python kserving-request.py http://finance-sample-predictor-default.$NAMESPACE.svc.cluster.local:80
    For example:
    $ python kserving-request.py http://finance-sample-predictor-default.test1.svc.cluster.local:80
    The output should appear similar to the following:
    200 
{'predictions': [{'model-version': '1', 'prediction': 0}]}

Cleaning the Namespace After Running the Sample

NOTE Make sure finance-sample is the only used name for resources in your cluster.
  1. Delete pods:
    kubectl delete pod -n $(kubectl get pods -n <USED_NAMESPACE> | grep 'finance-sample-predictor' | awk '{ print $1, $2 }' < echo)
    The output should appear similar to the following:
    kubectl delete pod -n $(kubectl get pods -n kfdf-tenant | grep 'finance-sample-predictor' | awk '{ print $1, $2 }' < echo)

pod "finance-sample-predictor-default-00001-deployment-7bbfbd88shd88" deleted
  2. Delete revision:
    kubectl delete revision -n $(kubectl get revision -A | grep 'finance-sample-predictor' | awk '{ print $1, $2 }')
    The output should appear as follows:
    kubectl delete revision -n $(kubectl get revision -A | grep 'finance-sample-predictor' | awk '{ print $1, $2 }')

revision.serving.knative.dev "finance-sample-predictor-default-00001" deleted
  3. Delete kservice:
    kubectl delete ksvc -n $(kubectl get ksvc -A | grep 'finance-sample' | awk '{ print $1, $2 }')
    The output should appear as follows:
    kubectl delete ksvc -n $(kubectl get ksvc -A | grep 'finance-sample' | awk '{ print $1, $2 }')
service.serving.knative.dev "finance-sample-predictor-default" deleted
  4. Delete inferenceservice:
    kubectl delete inferenceservice -n $(kubectl get inferenceservice -A | grep 'finance-sample' | awk '{ print $1, $2 }')
    The output should appear as follows:
    kubectl delete inferenceservice -n $(kubectl get inferenceservice -A | grep 'finance-sample' | awk '{ print $1, $2 }')
inferenceservice.serving.kserve.io "finance-sample" deleted