Configure SSL Encryption for Spark on YARN

Prerequisites

Starting in EEP 8.0.0, if you are Non-Admin user starting a Spark Application, you must generate the KeyStore and KeyStore password and set it in Spark configuration files. For details, see Security - Spark Documentation. Otherwise, KeyStore and default KeyStore password generates automatically.

Starting in EEP 6.0.0, you can remove spark.ssl.keyStorePassword, spark.ssl.trustStorePassword, and spark.ssl.keyPassword from the spark-defaults.conf file for additional security. These passwords are stored in the /opt/mapr/conf/ssl-client.xml file and Spark can access passwords from this file itself.
NOTE
If passwords are present in both /opt/mapr/conf/ssl-client.xml and /opt/mapr/spark/spark-<version>/conf/spark-defaults.conf files, then the password from the spark-defaults.conf file is used.

About this task

Complete the following step to manually configure encryption for the Spark HTTP file and broadcast servers:

Procedure

In the spark-defaults.conf file on each spark node, configure the following properties. Starting in EEP 6.0.0, the configured algorithms mentioned in the following code are no longer available for your web service to pick up. You must remove the spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA line to let parties negotiate the matching ciphers.
  • For Spark 2.0.1 and later: 
    spark.ssl.fs.enabled true
spark.ssl.keyPassword <ssl-keystore-password>
spark.ssl.keyStore /opt/mapr/conf/ssl_keystore
spark.ssl.keyStorePassword <ssl-keystore-password>
spark.ssl.trustStore /opt/mapr/conf/ssl_truststore
spark.ssl.trustStorePassword <ssl-keystore-password>
spark.ssl.protocol TLSv1.2
spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
    NOTE
    Starting in EEP 4.0, for secure clusters, you can skip this step. For new installs done through the 6.0 MapR Installer, the installer enables this configuration. For manual installs and upgrades, running configure.sh -R, as the final step in the configuration process, enables these settings.
  • For Spark 1.6.1:
    spark.ssl.akka.enabled true
spark.ssl.fs.enabled true
spark.ssl.keyPassword <ssl-keystore-password>
spark.ssl.keyStore /opt/mapr/conf/ssl_keystore
spark.ssl.keyStorePassword <ssl-keystore-password>
spark.ssl.trustStore /opt/mapr/conf/ssl_truststore
spark.ssl.trustStorePassword <ssl-keystore-password>
spark.ssl.protocol TLSv1.2
spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA

Spark UI SSL is not needed when running Spark on YARN because encryption is provided by the YARN protocol. For versions prior to EEP 4.1.0, to enable users logged in with a normal user account (not mapr or root) to run spark jobs on the cluster, disable Spark SSL for Spark-on-YARN jobs. To disable Spark SSL, add spark.ssl.ui.enabled false to the spark-defaults.conf file on each spark node. The spark-defaults.conf file is in the following location: /opt/mapr/spark/spark-<version>/conf/. Make sure SSL is enabled for the Spark history server.

When you manually configure encryption for Spark, set the same protocol and algorithms for each node. Otherwise, the connection between those components might fail.