Release Notes

This document provides a comprehensive overview of the latest updates and enhancements in HPE Ezmeral Unified Analytics Software (version 1.3.0), including new features, improvements, bug fixes, and known issues.

HPE Ezmeral Unified Analytics Software provides software foundations for enterprises to develop and deploy end-to-end data and advanced analytics solutions from data engineering to data science and machine learning across hybrid cloud infrastructures – delivered as a software-as-a-service model.

New Features

HPE Ezmeral Unified Analytics Software on Openshift

In this release, HPE Ezmeral Unified Analytics Software extends installation support to the OpenShift container orchestration platform. The power of OpenShift makes HPE Ezmeral Unified Analytics Software deployment agile, scalable, and efficient, enabling enterprises to streamline their analytics workflows. For details, see Installing HPE Ezmeral Unified Analytics Software on OpenShift.

Data Source Access Control

With this release, HPE Ezmeral Unified Analytics Software introduces the ability to fine-tune data source access permissions to individual users. Administrators can assign data access permissions to entire data sources, specific schemas/tables, or buckets. New users are denied access to any data source by default and admins can also make data sources public to all users. Data access policies are enforced across all Unified Analytics applications and programmatic data access clients. This feature allows administrators to safeguard sensitive data, ensuring that only allowed personnel can access restricted information and operate in compliance with corporate data and security policies. For details, see Managing Data Access.

HPE MLDE as Part of Unified Analytics

HPE Machine Learning Development Environment (HPE MLDE) is now integrated with HPE Ezmeral Unified Analytics Software. This collaboration brings together the capabilities of HPE machine learning tools within a unified environment to provide a cohesive solution for machine learning (ML) development. You can now directly leverage the features of HPE MLDE within HPE Ezmeral Unified Analytics Software, creating a unified ecosystem. This integration enhances overall productivity, accelerates model development, and maximizes the potential of ML and GenAI initiatives. For details, see HPE Machine Learning Development Environment.

Enhancements

Elevated Security with Advanced Access Management

HPE Ezmeral Unified Analytics Software introduces a dual-layered approach to access management through access token renewal and expiration. Within a user's workspace in a Unified Analytics cluster, automatic token renewal allows them to seamlessly engage in prolonged activities, minimizing disruptions in their workflows. Concurrently, each access token is set to expire quickly; if a token is obtained by some other party through an exploit, the window in which it can be used to act as the authorized user is minimized.

This enhancement strengthens the overall security posture, instilling users with increased confidence in safeguarding their analytics workloads and ensuring a secure computing environment.

Installation Experience

In this release, the upgraded HPE Ezmeral Unified Analytics Software installation experience introduces a “Precheck” feature in the UI. You can now leverage the precheck option to run a preliminary assessment of hardware and host requirements. This proactive step identifies and addresses any potential issues or discrepancies before initiating the setup process for a more efficient and reliable Unified Analytics installation experience.

Other Application Upgrades

For a list of updated applications, including Airflow, Livy, MLflow, Ray, Spark, and Superset, see Support Matrix.

Resolved Issues

This release introduces the ability to access infrastructure services log files through the installer UI.

Known Issues

The following sections describe known issues with workarounds where applicable:

Permission denied error when installing packages while using the Kubeflow notebook images

The Kubeflow notebook images provided by HPE Ezmeral Unified Analytics Software includes KFP SDK V1. When you install the packages while using KFP SDK V2, you will get the permission denied error. This issue occurs as by default the KFP V2 uses the python:3.7 image, which does not include the kfp library and attempts to self-install it.

Workaround: To install the packages while using KFP SDK V2, follow these steps:
  1. To get the custom gcr.io/mapr-252711/kubeflow/python-kfp-writable:3.7-2.7.0 image, choose one of the following options depending on your environment type:
    • (Connected environment only) Directly pull the built image gcr.io/mapr-252711/kubeflow/python-kfp-writable:3.7-2.7.0 from gcr.io.
    • (Air-gapped environment only) Choose one of the following:
      • Build a custom image using the following Dockerfile and then push the image into your local registry.
        FROM python:3.7
ENV PIP_DISABLE_PIP_VERSION_CHECK=1
RUN python3 -m pip install 'kfp==2.7.0' '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<"3.9"'
RUN mkdir -p -m 777 /gcs /s3 /minio /.local /.cache /.config
      • Pull the image from gcr.io and push them into your local registry.
  2. Run the following command to install the required packages. The following codeblock installs the numpy packages when you are using the environment that is behind the proxy.
    from kfp import dsl

@dsl.component(
    # install_kfp_package = False,
    base_image = "gcr.io/mapr-252711/kubeflow/python-kfp-writable:3.7-2.7.0",
    packages_to_install = ['numpy']
)
def say_hello(name: str) -> str:
    hello_text = f'Hello, {name}!'
    print(hello_text)
    return hello_text

@dsl.pipeline
def hello_pipeline(recipient: str = 'World!1') -> str:
    hello_task = say_hello(name=recipient)
    hello_task.set_caching_options(False)
    http_proxy = 'http://<your-proxy-endpoint>'
    no_proxy = '<enter-proxy-values>'
    no_proxy += ',.kubeflow'
    hello_task.set_env_variable('http_proxy', http_proxy)
    hello_task.set_env_variable('https_proxy', http_proxy)
    hello_task.set_env_variable('no_proxy', no_proxy)
    hello_task.set_env_variable('HTTP_PROXY', http_proxy)
    hello_task.set_env_variable('HTTPS_PROXY', http_proxy)
    hello_task.set_env_variable('NO_PROXY', no_proxy)
    return hello_task.output

Permission denied error when submitting the Kubeflow pipeline while using the Kubeflow notebook images

The Kubeflow notebook images provided by HPE Ezmeral Unified Analytics Software includes KFP SDK V1. When you submit the pipeline using KFP SDK V2, you will get the permission denied error. This issue occurs as by default the KFP V2 uses the python:3.7 image, which does not include the kfp library and attempts to self-install it.

Workaround: To submit the pipeline using KFP SDK V2, follow these steps:
  1. To get the custom gcr.io/mapr-252711/kubeflow/python-kfp-writable:3.7-2.7.0 image, choose one of the following options depending on your environment type:
    • (Connected environment only) Directly pull the built image gcr.io/mapr-252711/kubeflow/python-kfp-writable:3.7-2.7.0 from gcr.io.
    • (Air-gapped environment only) Choose one of the following:
      • Build a custom image using the following Dockerfile and then push the image into your local registry.
        FROM python:3.7
ENV PIP_DISABLE_PIP_VERSION_CHECK=1
RUN python3 -m pip install 'kfp==2.7.0' '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<"3.9"'
RUN mkdir -p -m 777 /gcs /s3 /minio /.local /.cache /.config
      • Pull the image from gcr.io and push them into your local registry.
  2. Add the following options to the @dsl.component decorator of your pipleline when creating and submitting your pipeline. 
    @dsl.component(
    install_kfp_package = False,
    base_image = "gcr.io/mapr-252711/kubeflow/python-kfp-writable:3.7-2.7.0",
)
    For example, submit the Hello World pipeline example from https://www.kubeflow.org/docs/components/pipelines/v2/hello-world/.

Cannot sign-in to HPE Ezmeral Unified Analytics Software after reboot
If you cannot sign in to HPE Ezmeral Unified Analytics Software after the nodes reboot, you may have encountered a Postgres issue that impacts the postgres-keycloak pod. The Postgres issue causes the service pods that interact with the postgres-keycloak pod to get stuck in a CrashLoopBackOff state, which then prevents you from signing in to HPE Ezmeral Unified Analytics Software. For additional details and a workaround, see Cannot Sign-In to HPE Ezmeral Unified Analytics Software After Reboot.

Update the SPIFFE CSI driver
After installing and deploying HPE Ezmeral Unified Analytics Software, run the following command to update the SPIFFE CSI driver: 
kubectl -n spire set image ds spire-spiffe-csi-driver spiffe-csi-driver=ghcr.io/spiffe/spiffe-csi-driver:0.2.5

After a reboot, the Unified Analytics hosts may not be ready due to an issue with the SPIFFE CSI driver; updating SPIFFE CSI may resolve it. For details, see Host (Node) Management.

EzPresto does not release memory when a query completes

EzPresto retains allocated memory after query completion for subsequent queries because of an open-source issue (https://github.com/prestodb/presto/issues/15637). For example, if a query uses 10GB of memory, EzPresto does not release the memory when the query completes and then uses it for the next query. If the next query requires additional memory, for instance, 12GB, EzPresto accumulates an extra 2GB and does not release it after query completion. For assitance, contact HPE support.

Configuration changes to long-running pods are not applied in Ray

Configuration changes or upgrades to long-running pods in Ray, such as adjusting resource capacities or expanding persistent volume (PV) storage are not applied in Ray.

Workaround

To ensure successful configuration changes or upgrades, manually delete relevant pods after the reconfiguration or upgrade process. For details, see https://github.com/ray-project/kuberay/issues/527.

Worker nodes do not automatically spawn with JobSubmissionClient in the Ray cluster

When submitting jobs to the Ray cluster using JobSubmissionClient, worker nodes do not spawn automatically.

Workaround

To ensure proper functionality when submitting Ray jobs using JobSubmissionClient, you must manually specify entry point resources as follows:
  • For CPU, set entrypoint_num_cpus to 1
  • For GPU, set entrypoint_num_gpus to 1
For details, see Using JobSubmissionClient to Submit Ray Jobs.

HPE is actively engaging with the community to address this open-source issue (https://github.com/ray-project/ray/issues/42436).

Installation of Unified Analytics on Azure -PPH may get stuck

Installation of Unified Analytics on Azure -PPH may get stuck because of a slower disk. A faster read/write disk (such as SSD) may overcome the issue.

Installer UI does not display the ingress gateway node IP addresses

In rare instances, the installer UI might not display the ingress gateway node IP addresses post installation. These IP addresses are needed for configuring the DNS A record. If you encounter this issue, please refer to the provided guidelines here to retrieve the IP address. For details, see User Interface.

Long-running Spark applications exceed disk quotas for Spark History Server

Repeatedly running long-running Spark applications generates a large volume of logs in the Spark History Server event log directory. This can exceed disk quotas, causing failures in other Spark applications. You must monitor log sizes and manage disk space to mitigate this issue. For details, see Spark.

Specified image pull policy is not applied to a pod

When you create a notebook server and set the imagePullPolicy to IfNotPresent or Never, the specified image pull policy is not set to the pod. In both scenarios, the imagePullPolicy is set to Always. For details, see Notebooks.

Table fails to display empty list when filtering non-existent or inapplicable properties in the Kubeflow UI

In the Endpoints screen of the Kubeflow UI, when you select filter options from the Filter menu or enter property names or values that are non-existent or inapplicable, the table fails to display an empty list as expected. Instead, the table displays all available items.

NVIDIA GPU Cannot Enforce SELinux
Due to a known NVIDIA GPU issue (https://github.com/NVIDIA/gpu-operator/issues/553), SELinux cannot be enforced for GPU deployments.
Workaround
Set GPU hosts to either disabled or permissive mode until this issue is resolved.

Ray Dashboard UI
A known Ray issue prevents the Ray Dashboard UI from displaying the GPU worker group details correctly. To see updates regarding resolution and to learn more, see https://github.com/ray-project/ray/issues/14664.

Spark Magic
To leverage Spark magic (%manage_spark) on Jupyter Notebooks for interactive Livy on GPU, users must manually configure GPU settings (%config_spark). For details, see Enabling GPU Support for Livy Sessions.

Installation Fails due to Insufficient GPU/CPU
The installation process fails if the installation requests more GPU/CPU resources than are available. To prevent installation failures, ensure that the resource requests match the available GPU/CPU capacity.

Failed Workload Cluster Deployment Due to Cluster Name Length
Workload cluster deployment fails when the cluster name has more than 21 characters.
Workaround
Rename the workload cluster to a name that is 21 characters or less.

Installation

Before you install or upgrade, HPE recommends that you back up your data.  If you encounter any issues during or after the installation process, please contact HPE support. We appreciate your feedback and strive to continually enhance your product experience. 

Additional Resources

Thank you for choosing HPE Ezmeral Unified Analytics Software. Enjoy the new features and improvements introduced in this release.