Nvidia Spark-RAPIDS Accelerator for Spark

This topic describes Nvidia spark-rapids accelerator support for Spark.

Starting from HPE Ezmeral Runtime Enterprise 5.5.0, you can use RAPIDS Accelerator for Apache Spark by Nividia to accelerate the processing for Spark by using the GPUs.

For details on Spark support for requesting and scheduling GPUs for executors, see Custom Resource Scheduling and Configuration Overview.

The GPU image (spark-gpu-<spark-version>), for example: spark-gpu-3.2.0, has built-in RAPIDS plugin in HPE Ezmeral Runtime Enterprise.

To allocate GPUs and enable RAPIDS plugin, set sparkConf option. For configuration details, see Spark GPU Example.
NOTE You must set spark.rapids.force.caller.classloader option to false.

For full list of configuration details for RAPIDS, see RAPIDS Configuration.