Tutorial: Kale Extension in Kubedirector Notebook

The Kale extension for HPE Ezmeral Runtime Enterprise enables the automation of Jupyter Notebook deployments to Kubeflow Pipelines.

Kale enables you to deploy local or cloud-based Jupyter Notebooks to Kubeflow Pipelines, automatically converting the Notebook to a valid Kubeflow Pipelines deployment. Kale also resolves data dependencies and manages the lifecycle of the pipeline.

Kale Tutorial

If you have not done so already: Before beginning this tutorial, download the Kubeflow tutorials zip file, which contains sample files for all the included Kubeflow tutorials.

To complete this tutorial:

  1. Log in to the Kubedirector notebook.
  2. Before enabling the Kale extension, execute the steps from examples/kubeflow/kale/README.ipynb in the Kubedirector notebook. Proceed as follows:
    1. Run the following cells:
      from ezmllib.kubeconfig.ezkubeconfig import set_kubeconfig
set_kubeconfig()
      from ezmllib.kubeflow.ezkfp import KfSession
K = KfSession()
client=K.kf_client()
      # List the current kubeflow pipelines
print(client.list_pipelines())
    2. If print(client.list_pipelines()) returns a nonempty array of pipelines, you have successfully created your KFP client.
  3. Upload candies_sharing.ipynb using the Upload Files button in the Kubedirector notebook.
  4. Enable the Kale extension, as follows:
    1. Click the Kale button in the navigation pane. The Kale Deployment Panel opens.
    2. Select Enable.

    For more information on the Kale Deployment Panel, see The Kale Deployment Panel.

  5. Open the candies_sharing.ipynb file.
  6. Run all cells in the notebook using Run > Run All Cells.
  7. At the bottom of the Kale Deployment Panel, click Compile and Run.
  8. On the Kubeflow Dashboard, open the Runs page. Check the status of the pipeline run launched from the Kubedirector notebook.
  9. Perform cleanup steps to remove the pod from your namespace, as described in Cleaning the Namespace After Running the Sample.

Cleaning the Namespace After Running the Sample

After completing the Kale Tutorial, you can perform cleanup steps to remove the pod from your namespace.

The following is an example of Kubernetes pods in the tenant namespace after successful completion of the tutorial: 
```
NAME                                       READY   STATUS      RESTARTS   AGE
candies-sharing-bq9mb-1430592451           0/2     Completed   0          17h
candies-sharing-bq9mb-196233272            0/2     Completed   0          17h
candies-sharing-bq9mb-213010891            0/2     Completed   0          17h
candies-sharing-bq9mb-229788510            0/2     Completed   0          17h
candies-sharing-bq9mb-833381010            0/1     Completed   0          17h
```
To remove pods from the namespace:
  1. Start a new terminal session in the KubeDirector notebook.
  2. Run the following kubectl commands to remove the pipeline's resources:
    • kubectl get wf

```
NAME                         STATUS      AGE
candies-sharing-zshg5        Succeeded   3m44s
```
    • kubectl delete wf <wf_name>
      For example:
      kubectl delete wf candies-sharing-zshg5
  3. The result of the pipeline run remains on the Runs page of the Kubeflow dashboard. If you want to remove the results from the list on the Runs page, you can archive the results instead.

The Kale Deployment Panel

The Kale Deployment Panel contains the following sections:

  • Pipeline Metadata: Define the name of the experiment and pipeline, and provide a description.
  • Run: Enable the Katib feature for this pipeline. Set up appropriate hyperparameters and other variables by clicking the Enable Katib Job button.
  • Click Advanced Settings. In Advanced Settings, you can set a container image. This container image is used for all steps of the current pipeline.
NOTE
The Kale extension does not support the Rok snapshot feature at this time. Instead, you can manually create a volume or use an existing volume for this pipeline.

Editing a Cell

To edit a cell:

  1. Click the pencil icon. Cell information for Kale opens.
  2. Edit the information about this cell.
  3. Click the x button. Cell information closes.