Configuring Polaris as a metastore for Spark

1. User → MCP Server — HTTPS request with Authorization: Bearer <JWT> (or Basic auth). Contains the MCP tool call (create_sql_session or execute_query).

2. MCP Server → Livy — Creates a Spark session. Injects the user's JWT into the session config as the Polaris catalog token. Spark starts on YARN.

3. Spark execution — Spark executes the SQL. Before reading any table, it calls Polaris to resolve table metadata and file locations.

4. Spark → Polaris — Iceberg REST API call with the user's JWT in Authorization: Bearer. Polaris validates the token against Keycloak, resolves the Polaris principal, and checks catalog role privileges.

5. Spark → MapR FS — Reads Parquet files at the locations returned by Polaris. Returns rows to Livy, which returns them to MCP Server, which returns them to the user.

• Single credential end-to-end. The user's Keycloak JWT is used by MCP Server, Spark, and Polaris. No service account impersonation.

• Polaris principal = Keycloak username. Polaris extracts the preferred_username claim from the JWT to identify the principal. The names must match exactly.

• The Spark catalog configuration is injected dynamically. The MCP server injects the Polaris catalog URL, warehouse, and user JWT into the Livy session config at creation time. No static Spark catalog config is needed in spark-defaults.conf.

ADMIN_TOKEN=$(curl -sk -X POST \
  https://<keycloak-host>:6443/realms/master/protocol/openid-connect/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "client_id=admin-cli&username=admin&password=<admin-password>&grant_type=password" \
  | jq -r .access_token)

curl -sk -X POST https://<keycloak-host>:6443/admin/realms/master/users \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "username": "mapruser1",
    "enabled": true,
    "credentials": [{"type": "password", "value": "mapruser1", "temporary": false}]
  }'

CLIENT_UUID=$(curl -sk \
  "https://<keycloak-host>:6443/admin/realms/master/clients?clientId=edf-client" \
  -H "Authorization: Bearer $ADMIN_TOKEN" | jq -r '.[0].id')

CLIENT_SECRET=$(curl -sk \
  "https://<keycloak-host>:6443/admin/realms/master/clients/$CLIENT_UUID/client-secret" \
  -H "Authorization: Bearer $ADMIN_TOKEN" | jq -r .value)

echo $CLIENT_SECRET

POLARIS_TOKEN=$(curl -sk -X POST \
  https://<keycloak-host>:6443/realms/master/protocol/openid-connect/token \
  -d "client_id=edf-client&client_secret=$CLIENT_SECRET&grant_type=client_credentials" \
  | jq -r .access_token)

POLARIS=/opt/mapr/polaris/current/client/bin/polaris

$POLARIS --access-token $POLARIS_TOKEN \
  catalogs create \
  --storage-type MFS \
  --default-base-location maprfs:///polaris/mcp-warehouse \
  mcp-catalog

$POLARIS --access-token $POLARIS_TOKEN principals create mapruser1

• Principal role - assigned to a principal; groups catalog roles

• Catalog role - scoped to a specific catalog; holds catalog privileges

# Principal role
$POLARIS --access-token $POLARIS_TOKEN principal-roles create mcp-principal-role

# Catalog role
$POLARIS --access-token $POLARIS_TOKEN catalog-roles create \
  --catalog mcp-catalog mcp-catalog-role

# Grant full catalog access to the catalog role
$POLARIS --access-token $POLARIS_TOKEN privileges catalog grant \
  --catalog mcp-catalog \
  --catalog-role mcp-catalog-role \
  CATALOG_MANAGE_CONTENT

# Attach catalog role to principal role
$POLARIS --access-token $POLARIS_TOKEN catalog-roles grant \
  --catalog mcp-catalog \
  --principal-role mcp-principal-role \
  mcp-catalog-role

# Attach principal role to principal
$POLARIS --access-token $POLARIS_TOKEN principal-roles grant \
  --principal mapruser1 \
  mcp-principal-role

mcp = {
  http = {
    port = 5679
    secure = true
    auth.enabled = true
  }
  file.enabled = true
  spark {
    enabled = true
    impersonation-enabled = true   # Run Spark jobs as the authenticated user
    disable-sql-policy = true      # Allow DDL/DML (CREATE, INSERT, DROP)

    iceberg {
      catalog {
        backend = "polaris"
        name = "mcp-catalog"
        uri = "https://<polaris-host>:8181/api/catalog"
        warehouse = "mcp-catalog"
        use-bearer-token = true     # Forward user JWT to Polaris
        scope = "PRINCIPAL_ROLE:ALL"
        token-refresh-enabled = true
        access-delegation = "vended-credentials"
        client.region = "us-east-1"
      }
    }
  }
}

maprcli node services -name data-access-gateway -action restart -nodes $(hostname -f)

spark.jars.ext-libs.use                     true
spark.sql.extensions                         org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

spark.sql.catalog.mcp-catalog                     org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.mcp-catalog.type                rest
spark.sql.catalog.mcp-catalog.uri                 https://<polaris-host>:8181/api/catalog
spark.sql.catalog.mcp-catalog.warehouse           mcp-catalog
spark.sql.catalog.mcp-catalog.rest.auth.type      oauth2
spark.sql.catalog.mcp-catalog.token               <user JWT>
spark.sql.catalog.mcp-catalog.scope               PRINCIPAL_ROLE:ALL

spark.sql.catalog.hive         org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.hive.type    hive
spark.sql.catalog.hive.uri     thrift://<hive-host>:9083

• Global: ~/.claude.json - applies to all projects
• Project-level: .claude/settings.json (committed) or .claude/settings.local.json (ignored by Git, for secrets)

claude mcp add spark-mcp \
  --env NODE_TLS_REJECT_UNAUTHORIZED=0 \
  -- npx -y mcp-remote https://<mcp-host>:5679/spark \
  --static-oauth-client-info "{\"client_id\":\"edf-client\",\"client_secret\":\"$CLIENT_SECRET\"}" \
  --debug

{
  "mcpServers": {
    "spark-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://<mcp-host>:5679/spark",
        "--static-oauth-client-info",
        "{\"client_id\":\"edf-client\",\"client_secret\":\"<edf-client-secret>\"}",
        "--debug"
      ],
      "env": {
        "NODE_TLS_REJECT_UNAUTHORIZED": "0"
      }
    }
  }
}

rm -rf ~/.mcp-auth/mcp-remote-*/

claude mcp list          # Should show spark-mcp
claude mcp get spark-mcp # Shows connection status and available tools

claude

1. Call create_sql_session - waits ~2 min for Spark to start on YARN

2. Call execute_query with SHOW NAMESPACES IN \ mcp-catalog``

3. Call execute_query with SHOW TABLES IN \ mcp-catalog`.db`

4. Construct and execute the aggregation query:

   SELECT user_id, COUNT(*) AS event_count
   FROM `mcp-catalog`.db.events
   WHERE event_date >= DATE_SUB(CURRENT_DATE(), 7)
   GROUP BY user_id
   ORDER BY event_count DESC
   LIMIT 10

5. Return formatted results and optionally suggest follow-up analyses.

• "Now join this with the users table to get the actual usernames"

• "Export the top 10 results to a new table called mcp-catalog.analytics.top_users"

• "Show me the event trend by day for these top 10 users"

• Describe the schema of the events table in detail"