Notebook Magic Functions

Jupyter notebook magic functions, also known as magics, are special commands that provide notebook functions that might not be easy for you to program using Python. HPE Ezmeral Unified Analytics Software supports line magics and cell magics.

Jupyter notebook Magic functions, also known as magic commands or magics, are commands that you can execute within a code cell. Magics are not Python code. They are shortcuts that extends the capabilities of a notebook. Magic commands start with the % character.

HPE Ezmeral Unified Analytics Software supports built-in magic functions and the custom magics that are described in this topic. HPE Ezmeral Unified Analytics Software supports line magics and cell magics.

Line magic commands do not require a cell body and start with a single % character.

Cell magic commands start with %% and require additional lines of input (a cell body).

To use these magic functions, you must create a notebook. See Creating and Managing Notebook Servers.

%commands

The %commands command lists the magic commands and SDKs that are customized by Hewlett Packard Enterprise and are available in this notebook.

%createKernel

The %createKernel command creates a custom Python kernel in the notebook.

The custom Python kernel can be selected as the kernel for a notebook session to work within a specific virtual environment with its own set of dependencies and configurations. By using the custom Python kernel, you can isolate your Python packages and dependencies from other projects or applications such that each project has its own environment and is not affected by changes made to other environments.

To create a custom Python kernel from conda package installation, perform:
  1. Create a notebook with at least 4 Gi memory. See Creating and Managing Notebook Servers.
  2. (Optional) If the cluster is behind proxy, set the following proxy environment variables.
    %env https_proxy=<your-https-proxy>
    %env http_proxy=<your-http-proxy>
    %env no_proxy=<your-no-proxy>
    You can retrieve the proxy settings from the ezua-cluster-config configmap in the ezua-system namespace.
    NOTE
    If you are using the AWS or Azure environment, do not set the proxy environment variables.
  3. Enter the %createKernel command in a notebook.

    You can also directly enter packages as arguments with %createKernel magic function.

    For example:
    %createKernel pigz pandas
  4. Enter the name for your Python kernel in the Name box. In this example, we use MyPython as the custom Python kernel name.
  5. Enter the conda package name in the Package 1 box. To enter additional packages, click the + button.
  6. Click the Create Custom Python Kernel button.

  7. Click the New Launcher button.

  8. You can now see your custom Python kernel -MyPython kernel among the available kernels.

%manage_spark

The %manage_spark command enables you to connect to the Livy server and start a new Spark session. You must use Spark-related kernels such as PySpark, Spark, or SparkR to use sparkmagic. When you run the %manage_spark command in a notebook cell, a new user interface (UI) widget is displayed, which allows you to configure and manage a Spark cluster. You can use different tabs in this UI widget to manage sessions, create sessions, add endpoints, and manage endpoints.

Add Endpoint: To add endpoints, set the following boxes and then click Add endpoint.
  • Auth type: The default authentication for the internal Livy endpoint is Single Sign-On. To connect to other Livy clusters, select the authentication type of your choice.
  • Address: Enter endpoint address.


Create Session: To create sessions, set the following boxes and then click Create Session.
  • Endpoint: Select endpoint for your session.
  • Name: Enter session name.
  • Language: Choose either Scala or Python as a runtime.
  • Properties: Edit the Spark configurations.


%config_spark

You can use the %config_spark magic command to customize the Spark jobs submitted from the PySpark kernel. You can add or delete the Spark configurations when submitting a Livy session.

To customize the Spark configurations, follow these steps:
  1. Run %config_spark.
  2. Click the +Add Spark Configuration Key-Value Pair button.
  3. Enter the key and value for Spark configurations in their respective boxes.
  4. After you have finished adding the key-value pairs, click Submit.
  5. Restart the PySpark kernel and run %manage_spark to see the changes applied in the Properties section of the Create Session tab.


You can also edit the values for other Spark configurations or delete any configurations using this magic command.

For example: To learn more about customizing Spark configurations when enabling the GPU support for Livy sessions, see Enabling GPU Support for Livy Sessions Created Using Notebooks.

%git_clone

The %git_clone magic enables you to clone your private GitHub repository from the notebook.

To clone the repository, enter your GitHub username and GitHub password.

After you have finished cloning the repository, you can use the Git extension in the notebook for version control.

%sql and %%sql

You can use the %sql magic command in Jupyter Notebook to interactively work with SQL databases. To learn more about how to connect to databases, see connecting to a database.

You must use Python kernels to use %sql and %%sql magic commands. You can directly write and execute SQL queries within a notebook cell. When you run the notebook cell containing %sql and your SQL query, the magic command sends the query to the database, runs it, and retrieves the result.

In HPE Ezmeral Unified Analytics Software, you can connect to all SQL databases and submit queries through EzPresto using the %sql magic as follows:
%sql SELECT * FROM cache.information_schema.columns

You can use the %%sql magic command to define and run an entire SQL script or block. This means you can write and run a series of SQL statements in the same cell using %%sql magic.

The results of the SQL query or queries are displayed in the notebook as a table that makes it easy to analyze and visualize the data.

%update_token

The %update_token magic function updates the cached auth token. If you encounter a JWT token expiration error while running cells in the notebbok, you can resolve it by running the %update_token magic function. This function updates the JWT in environment variables and any other locations where the token is utilized.

You can use the %update_token to refresh tokens for the following cases:
  • Authentication when establishing a connection with PrestoDB.
  • Authentication with local s3 minio object storage.
  • Authentication with KServe external API.

Getting Help

To display help about a magic command, enter the command followed by a ? (question mark). For example:
%manage_spark?