Post-Upgrade Steps for Spark
Complete the following steps after you upgrade Spark with or without the Installer.
Post-Upgrade Steps for Spark Standalone Mode
About this task
Procedure
-
(Optional) Migrate Custom Configurations.
Migrate any custom configuration settings into the new default files in the conf directory (/opt/mapr/spark/spark-<version>/conf).
-
If Spark SQL is configured to work with Hive, copy the
hive-site.xml
file into theconf
directory (/opt/mapr/spark/spark-<version>/conf). -
Run the following commands to configure the secondary instances:
- Run configure.sh -R.
-
Restart all the spark secondary instances as the
mapr
user:For Spark 2.x:/opt/mapr/spark/spark-<version>/sbin/start-slaves.sh spark://<comma-separated list of spark master hostname: port>
For Spark 3.x:/opt/mapr/spark/spark-<version>/sbin/start-workers.sh spark://<comma-separated list of spark master hostname: port>
-
Delete the old Spark directory from
/opt/mapr/spark
. For example, if you upgraded from Spark 2.1.0 to 2.3.1, you need to delete/opt/mapr/spark/spark-2.1.0
.Starting with the EEP 6.1.0 release, for Spark 2.2.1 and later versions, after an upgrade the old directory is automatically removed. Only the new directory and the directory with the timestamp is present.
Post-Upgrade Steps for Spark on YARN
Procedure
-
(Optional) Migrate Custom Configurations.
Migrate any custom configuration settings into the new default files in the conf directory (/opt/mapr/spark/spark-<version>/conf). Also, if you previously configured Spark to use the Spark JAR file from a location on the file system, you need to copy the latest JAR file to the file system and reconfigure the path to the JAR file in the
spark-defaults.conf
file. See Configure Spark JAR Location. -
If Spark SQL is configured to work with Hive, copy the
hive-site.xml
file into the conf directory (/opt/mapr/spark/spark-<version>/conf). - Run configure.sh -R.
-
Delete the old Spark directory from
/opt/mapr/spark
. For example, if you upgraded from Spark 2.1.0 to 2.3.1, you need to delete/opt/mapr/spark/spark-2.1.0
.Starting with the EEP 6.1.0 release, for Spark 2.2.1 and later versions, after an upgrade the old directory is automatically removed. Only the new directory and the directory with the timestamp is present.