MCP Endpoints for Apache Spark

The Apache Spark MCP endpoint can operate in two modes: with impersonation enabled and disabled. Impersonation enabled is the default mode. Impersonation enabled is the default mode.

In impersonation enabled mode, all Spark sessions will be impersonated according to the user's credentials. A separate session will be created for each user, with a default idle time of 30 minutes. Since each session reserves cluster resources, their simultaneous number is limited and can be controlled by an option.

Impersonation enabled mode is recommended because it is secure and each user only has access to data they have permission to access. In this mode, before accessing Spark for the first time, you must call the "create_sql_session" tool. This tool should also be used to recreate expired sessions.

You can also use impersonation disabled mode. In this mode, sessions are pre-created (by default, there are three) under the user specified as the session-user in the configuration. This mode carries security risks, as all users will have access to data visible to the session-user. However, no time is spent creating sessions, and all sessions in this mode will be re-created automatically if they crash or expire.

To configure a Spark endpoint, you need a Spark section in the configuration file. This is the list of possible parameters:

MCP Spark endpoint options:
  • mcp.spark.enabled - enable Spark endpoint
  • mcp.spark.session-user - user which will create sessions in non-impersonation mode
  • mcp.spark.pool.size - amount of sessions precreated for non-impersonesion mode (3 by default)
  • mcp.spark.userpool.size - amount of available concurrent sessions/users for impersonation mode (3 by default)
  • mcp.spark.session.ttl - ttl for Spark sessions for non-impersonation mode (10h by default)
  • mcp.spark.session.userttl - ttl for Spark sessions for impersonation mode (30m by default)
  • mcp.spark.impersonation-enabled - enable or disable impersonation mode (true by default)
Spark options - these options are passed to the Spark engine:
  • mcp.spark.driver.memory - driver memory for Spark session (1g by default)
  • mcp.spark.driver.cores - amount of driver cores per Spark session (1 by default)
  • mcp.spark.executor.memory - executor memory per Spark session (1g by default)
  • mcp.spark.executor.cores - amount of executor cores per Spark session (1 by default)
  • mcp.spark.executor.instances - amount of executors per Spark session (1 by default)
Iceberg options:
  • mcp.spark.iceberg.extensions - extension to use for Iceberg integration (org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions by default)
  • mcp.spark.iceberg.catalog - iceberg catalog to use (org.apache.iceberg.spark.SparkCatalog by default)
  • mcp.spark.iceberg.catalog.type - catalog type (hive by default)
  • mcp.spark.iceberg.catalog.uri - catalog uri (by default Hive Metastore uri which is set to thrift://localhost:9083)
  • mcp.spark.iceberg.useBuiltIn - use built in Iceberg library (true by default)
NOTE

To use Spark endpoints you need to install mapr-livy, mapr-spark, mapr-hive, mapr-hivemetastore packages.