Ports Used by Spark

To run a Spark job from a client node, ephemeral ports should be opened in the cluster for the client from which you are running the Spark job.

If you do not want to open all the ephemeral ports, you can use the configuration parameter to specify the range of ports.

To set ports to special values, use the spark.driver.port, spark.blockManager.port, and spark.port.maxRetries properties. The spark.port.maxRetries property is 16 by default.

For example, if you need to open port 200 for spark.blockManager.port from 40000, set spark.blockManager.port = 40000 and spark.port.maxRetries = 200.

For a list of Web UIs ports dynamically used when starting spark contexts, see the open source documentation.

The default port numbers that need to be opened on the firewall behind the client and MapR cluster nodes for Spark jobs to operate in YARN client, YARN cluster, and standalone modes are as follows:
Service Port Number
Spark Standalone Master (RPC) 7077
Spark Standalone Master (Web UI) 8580, 8980*
Spark Standalone Worker 8581, 8981*
Spark Thrift Server 2304
Spark History Server 18080,18480*
Spark External Shuffle Service (if yarn shuffle service is enabled) 7337
CLDB 7222
ZooKeeper 5181
Nodes running ResourceManager 8032
MapR Filesystem Server 5660, 5692
* refers to ports for secure clusters