Ports Used by HPE Ezmeral Data Fabric Software

Lists the ports used by Data Fabric services.

Avoiding Port Conflicts

To avoid trouble with port conflicts on your Data Fabric clusters, try these tips:

  • Remap the ports for the HBaseMaster and HBaseRegionServer services to ports below 32768.
  • Set the ephemeral port range to stop at 50029 by changing the value in the file /proc/sys/net/ipv4/ip_local_port_range. Note that this setting changes the available number of ephemeral ports from the default of 28233 ports to 17233.

Ports Needed for POSIX Clients and File System to Communicate With Each Other

POSIX clients communicate with the CLDB and server components of the Data Fabric filesystem. You need to open the relevant ports for TCP connectivity from POSIX clients to the Data Fabric file-system cluster nodes. Open the CLDB, file-system server, and file-system server instances ports, as detailed in the following section.

Services and Ports Quick Reference

The following list defines the ports used by a Data Fabric cluster, along with the default port numbers. All the ports used by Data Fabric software are TCP ports.

Airflow Webserver
Source IP: Nodes/clients accessing Airflow webserver
Destination IP: Nodes running Airflow webserver
Ports: 8780
Purpose: Used by Airflow webserver clients to access the Airflow webserver
Parameter and File where Port is Configured: Not Applicable
Airflow Scheduler
Source IP: Airflow webserver
Destination IP: Nodes running Airflow scheduler
Ports: 8793
Purpose: Used by Airflow webserver to access the Airflow scheduler
Parameter and File where Port is Configured: Not Applicable
API Server (apiserver)
Source IP: Cluster nodes running apiserver
Destination IP: Cluster nodes running apiserver
Ports:
  • 5701
  • 5702
Purpose:Clustering support
Parameter and File where Port is Configured: Not Applicable
CLDB
Source IP: Nodes running any Data Fabric services, clients interacting with the file system
Destination IP: Cluster nodes running CLDB services
Ports: 7222

A client reads CLDB IP and port number from the /opt/mapr/conf/mapr-clusters.conf file. The client initially tries to communicate with CLDB on port 7222. Once it establishes the connection, it fetches the additional CLDB IPs and ports from the connected CLDB.

By default, CLDB listens on ports 7222 and 7223. For performance reasons, additional ports may be opened, depending on the configuration parameter cldb.num.rpc.threads in the /opt/mapr/conf/cldb.conf file. For example, setting cldb.num.rpc.threads=3, opens up ports 7222, 7223 and 7224.

NOTE
The cldb.num.rpc.threads parameter is hard-coded with a default value of 3. To change this value, add this parameter with the new value to the /opt/mapr/conf/cldb.conf file.
NOTE
If you upgrade from Core 5.2.x or Core 6 to Core 6.1 and above, the value of cldb.num.rpc.threads is not changed. The default remains as 3, which means three ports are open for each CLDB node.

The client tries connecting to the CLDBs till timeout occurs in the case of soft mount, while the client indefinitely retries in the case of hard mount. If a client cannot connect to a CLDB port, the CLDB is marked unreachable. For example, assume a CLDB with IP 10.10.10.10 and 3 ports 7222, 7223 and 7224. If a client fails to connect to the CLDB say on port 7223, the CLDB 10.10.10.10 is marked unreachable, and the client will not try the two other ports for the next few minutes. It tries to connect with the next CLDB entry in the list.

For load balancing at CLDB, a client will always pick a random port among the available CLDB ports.

Purpose: file system API calls
Parameter and File where Port is Configured:
  • /opt/mapr/conf/cldb.conf
  • /opt/mapr/conf/warden.conf
  • /opt/mapr/conf/mapr-clusters.conf
CLDB JMX Monitor Port
Source IP: Nodes running CLDB services
Destination IP: CLDB JMX monitor port
Ports: 7220
Purpose: The port on which Collectd gathers CLDB metrics through JMX.
Parameter and File where Port is Configured: Not Applicable
CLDB web port
Source IP: Nodes/clients connecting to the CLDB GUI
Destination IP: Cluster nodes running CLDB services
Ports: 7221
Purpose: CLDB GUI for a cluster with security disabled. For a secure cluster, the port is 7443 as defined by the maprlogin utility.
Parameter and File where Port is Configured: /opt/mapr/conf/cldb.conf
maprlogin utility
Source IP: Connections using the maprlogin utility
Destination IP: Cluster nodes running CLDB services
Ports: 7443
Purpose: When security is enabled for a cluster, the CLDB listens for connections on port 7443. If security is disabled, the maprlogin utility is unable to reach the CLDB.
Parameter and File where Port is Configured: Not Applicable
Data Access Gateway
Source IP: Clients using the HPE Ezmeral Data Fabric Database JSON REST API with HTTPS
Destination IP:Not Applicable
Ports: 8243
Purpose: The port used to connect to the Data Access Gateway using HTTPS
Parameter and File where Port is Configured: rest.https.port in /opt/mapr/data-access-gateway/conf/properties.cfg
Data Access Gateway
Source IP: Node.js OJAI client
Destination IP: Cluster nodes running the Data Access Gateway service
Ports: 5678
Purpose: The port used to connect the OJAI client to the Data Access Gateway
Parameter and File where Port is Configured: grpc.service.port in /opt/mapr/data-access-gateway/conf/properties.cfg
Data Access Gateway
Source IP: Python OJAI client
Destination IP: Cluster nodes running the Data Access Gateway service
Ports: 5678
Purpose: The port used to connect the OJAI client to the Data Access Gateway
Parameter and File where Port is Configured: grpc.service.port in /opt/mapr/data-access-gateway/conf/properties.cfg
Data Access Gateway
Source IP: Go OJAI client
Destination IP: Cluster nodes running the Data Access Gateway service
Ports: 5678
Purpose: The port used to connect the OJAI client to the Data Access Gateway
Parameter and File where Port is Configured: grpc.service.port in /opt/mapr/data-access-gateway/conf/properties.cfg
Data Access Gateway
Source IP: C# OJAI client
Destination IP: Cluster nodes running the Data Access Gateway service
Ports: 5678
Purpose: The port used to connect the OJAI client to the Data Access Gateway
Parameter and File where Port is Configured: grpc.service.port in /opt/mapr/data-access-gateway/conf/properties.cfg
Data Access Gateway
Source IP: Java OJAI thin client
Destination IP: Cluster nodes running the Data Access Gateway service
Ports: 5678
Purpose: The port used to connect the OJAI client to the Data Access Gateway
Parameter and File where Port is Configured: grpc.service.port in /opt/mapr/data-access-gateway/conf/properties.cfg
Apache Kafka Wire Protocol Service
Source IP: Apache Kafka Client
Destination IP: Cluster nodes running the Data Access Gateway service
Ports: 9092
Purpose: The port used by Apache Kafka Wire Protocol Service to connect the Kafka client to the Data Access Gateway service.
Parameter and File where Port is Configured: port in /opt/mapr/data-access-gateway/conf/kafka-server.conf
Drill JMX Port
Source IP: Nodes running the Drillbit service
Destination IP: Drill JMX Port
Ports: 6090
Purpose: The port on which Collectd gathers Drill metrics via JMX.
Parameter and File where Port is Configured: Not Applicable
Drill Web UI
Source IP: Nodes running the Drillbit service
Destination IP: Nodes running the Drillbit service
Ports: 8047
Purpose: TCP port needed for the Drill Web UI and clients using REST API and nodes running the Drillbit service.
Parameter and File where Port is Configured: drill.exec.http.port in /opt/mapr/drill/drill-<version>/conf/drill-override.conf
Drill (User Port)
Source IP: Nodes running the Drillbit service and clients using JDBC/ODBC
Destination IP: Nodes running the Drillbit service
Ports: 31010
Purpose: TCP user port address. Used between nodes in a Drill cluster. Needed for an external client, such as Tableau, to connect into the cluster nodes. Also needed for the Drill Web UI. You can also use this port to connect directly to a Drillbit.
Parameter and File where Port is Configured: drill.exec.rpc.user.server.port in /opt/mapr/drill/drill-<version>/conf/drill-override.conf
Drill (Control Port)
Source IP: Nodes running the Drillbit service
Destination IP: Nodes running the Drillbit service
Ports: 31011
Purpose: TCP port that controls the port address. Used between nodes in a Drill cluster. Needed for multi-node installation of Drill.
Parameter and File where Port is Configured: drill.exec.rpc.bit.server.port in /opt/mapr/drill/drill-<version>/conf/drill-override.conf
Drill (Data Port)
Source IP: Nodes running the Drillbit service
Destination IP: Nodes running the Drillbit service
Ports: 31012
Purpose: TCP data port address. Used between nodes in a Drill cluster. Needed for multi-node installation of Drill.
Parameter and File where Port is Configured: drill.exec.rpc.bit.server.port + 1 in /opt/mapr/drill/drill-<version>/conf/drill-override.conf
Drill (ZooKeeper Port)
Source IP: Clients using JDBC/ODBC and nodes running ZooKeeper services
Destination IP: Nodes running the Drillbit service
Ports: 5181
Purpose: ZooKeeper port used to connect to Drill through the JDBC driver.
Parameter and File where Port is Configured: See the ZooKeeper entry in this list.
Elasticsearch (Components Communication Port)
Source IP: Non-Elasticsearch components, such a web browser, curl, and Kibana, that connect to Elasticsearch.
Destination IP: Nodes running Elasticsearch for monitoring use cases
Ports: 9200
Purpose: Non-Elasticsearch components use this port when communicating with Elasticsearch.
Parameter and File where Port is Configured: You can configure a different port for monitoring use cases when you run the configure.sh script with the -ES parameter.
Elasticsearch (Daemons Communication Port)
Source IP: Nodes running Elasticsearch
Destination IP: Nodes running Elasticsearch for monitoring use cases
Ports: 9300
Purpose: Elasticsearch uses this port for communications between Elasticsearch daemons.
Parameter and File where Port is Configured: Not Applicable
Gateway
Source IP: Nodes sending operations to replicate
Destination IP: Nodes running the gateway service
Ports: 7660
Purpose: The port used by gateway services to listen for incoming replication operations.
Parameter and File where Port is Configured: Not Applicable
Grafana
Source IP: Web Browsers
Destination IP: Nodes running Grafana for monitoring
Ports: 3000
Purpose: Web browsers use this port when connecting to Grafana.
Parameter and File where Port is Configured: Not Applicable
HBase Master
Source IP: HBase Clients
Destination IP: Nodes running HBase Master services
Ports: 16000
Purpose: HBase API and HBase shell use this port to connect to HBase Master
Parameter and File where Port is Configured: /opt/mapr/hbase/hbase-<version>/conf/hbase-site.xml
HBase Master Web UI
Source IP: HBase Master Web UI clients
Destination IP: Nodes running HBase Master services
Ports: 16010
Purpose: Information Web UI of HBase Master
Parameter and File where Port is Configured: /opt/mapr/hbase/hbase-<version>/conf/hbase-site.xml
HBase Thrift Server
Source IP: HBase Thrift Server clients
Destination IP: Nodes running HBase Thrift Server
Ports: 9090
Purpose: The HBase client uses this port to connect to HBase, using the Thrift protocol
Parameter and File where Port is Configured: /opt/mapr/hbase/hbase-<version>/conf/hbase-site.xml
HBase Thrift Web UI
Source IP: HBase Thrift Web UI clients
Destination IP: Nodes running HBase Thrift
Ports: 9095
Purpose: Information Web UI of HBase Thrift
Parameter and File where Port is Configured: /opt/mapr/hbase/hbase-<version>/conf/hbase-site.xml
HBase REST Server
Source IP: HBase REST Server clients
Destination IP: Nodes running HBase REST Server
Ports: 8080
Purpose: The HBase client uses this port to connect to HBase using the HTTP protocol
Parameter and File where Port is Configured: /opt/mapr/hbase/hbase-<version>/conf/hbase-site.xml
HBase REST Web UI
Source IP: HBase REST Web UI clients
Destination IP: Nodes running HBase REST
Ports: 8086
Purpose: Information Web UI of HBase REST
Parameter and File where Port is Configured: /opt/mapr/hbase/hbase-<version>/conf/hbase-site.xml
HBase Regionserver
Source IP: HBase Clients
Destination IP: Nodes running HBase Regionserver services
Ports: 16020
Purpose: HBase API and HBase shell use this port to connect to HBase RegionServer
Parameter and File where Port is Configured: /opt/mapr/hbase/hbase-<version>/conf/hbase-site.xml
HBase Regionserver UI
Source IP: HBase Regionserver Web UI clients
Destination IP: Nodes running HBase Regionserver
Ports: 16030
Purpose: Information Web UI of HBase Regionserver
Parameter and File where Port is Configured: /opt/mapr/hbase/hbase-<version>/conf/hbase-site.xml
HistoryServer RPC
Source IP: Not Applicable
Destination IP: Nodes running MapReduce JobHistory Server
Ports: 10020
Purpose: Not Applicable
Parameter and File where Port is Configured: Not Applicable
HistoryServer Web UI and REST APIs
Source IP: Clients that access Job History Server UI in a non-secure cluster
Destination IP: Secure nodes running MapReduce JobHistory Server in a non-secure cluster
Ports: 19888
Purpose: Non-secure HistoryServer Web UI and REST APIs
Parameter and File where Port is Configured: See mapred-site.xml
HistoryServer Web UI and REST APIs
Source IP: Clients that access Job History Server UI in a secure cluster
Destination IP: Secure nodes running MapReduce JobHistory Server in a secure cluster
Ports: 19890
Purpose: Secure HistoryServer Web UI and REST APIs
Parameter and File where Port is Configured: See mapred-site.xml
Hive Metastore
Source IP: Nodes/clients performing Hive queries/operations
Destination IP: Nodes running the Hive metastore services
Ports: 9083
Purpose: Used by Hive clients to query/access the Hive metastore
Parameter and File where Port is Configured: /opt/mapr/hive/hive-<version>/conf/hive-site.xml
Hiveserver2
Source IP: Nodes or clients performing hive queries using JDBC/ODBC
Destination IP: Nodes running Hiveserver2
Ports: 10000
Purpose: Port through which clients perform hive queries
Parameter and File where Port is Configured: Not Applicable
Hiveserver2 Web UI
Source IP: Not Applicable
Destination IP: Nodes running Hiverserver2 Web UI
Ports: 10002
Purpose: Provides access to Hive configuration settings, local logs, metrics, and information about active sessions and queries.
Parameter and File where Port is Configured: Not Applicable
Hoststats
See hoststats.port and hs.port in the Warden configuration file.
Httpfs
Source IP: Nodes/clients accessing httpfs services
Destination IP: Nodes running httpfs services
Ports: 14000
Purpose: Used by httpfs file clients to access the httpfs server
Parameter and File where Port is Configured:
  • /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/httpfs-env.sh
Hue Webserver
Source IP: Nodes/clients accessing Hue web services
Destination IP: Nodes running Hue web services
Ports: 8888
Purpose: Used by Hue webserver clients to access the Hue webserver
Parameter and File where Port is Configured: /opt/mapr/hue/hue*/desktop/conf/hue.ini
KSQL
Source IP: All cluster nodes
Destination IP: Nodes running KSQL
Ports: 8084
Purpose: KSQL
Parameter and File where Port is Configured: $KSQL_INSTALL_DIR/etc/ksql/ksqlserver.properties
Kafka Connect
Source IP: All cluster nodes
Destination IP: Nodes running Kafka Connect
Ports: 8083
Purpose: Kafka Connect REST API calls
Parameter and File where Port is Configured: /opt/mapr/kafka/kafka-<version>/config/connect-distributed.properties
Kafka REST
Source IP: All cluster nodes
Destination IP: Nodes running Kafka REST
Ports: 8082
Purpose: Kafka Connect REST API calls
Parameter and File where Port is Configured: /opt/mapr/kafka-rest/kafka-rest-<version>/config/kafka-rest.properties
Kafka Schema Registry
Source IP: All cluster nodes
Destination IP: Nodes running Kafka Schema Registry
Ports: 8087
Purpose: Kafka Schema Registry API calls
Parameter and File where Port is Configured: /opt/mapr/schema-registry/schema-registry-<version>/config/schema-registry.properties
Kibana
Source IP: Web browsers
Destination IP: Nodes running Kibana for monitoring use cases
Ports: 5601
Purpose: Web browsers use this port when connecting to Grafana.
Parameter and File where Port is Configured: Not Applicable
MAST Gateway
Source IP: Nodes running MAST Gateway service
Destination IP: Nodes running MAST Gateway service
Ports: 8660
Purpose: Data Fabric clients use this port to connect to the MAST Gateway
Parameter and File where Port is Configured: /opt/mapr/conf/mastgateway.conf
file system server
Source IP: Nodes running any Data Fabric services, clients interacting with the file system
Destination IP: Nodes running FileServer services
Ports: 5660, 5692, 5724, and 5756
Purpose: The filesystem is a random read-write distributed filesystem that allows applications to concurrently read and write directly to disk. Clients use these ports to access the file-system server.
Parameter and File where Port is Configured: Not Applicable
file system server
Source IP: Nodes running the gateway service
Destination IP: Nodes running the file system
Ports: 6660
Purpose: The port on which gateway nodes send replicated operations to nodes in destination clusters.
Parameter and File where Port is Configured: Not Applicable
file system server instances
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: See Working with Multiple Instances of the File System
Purpose: Multiple file system instances
Parameter and File where Port is Configured: Not Applicable
Object Store
Source IP: Nodes accessing MOSS
Destination IP: Nodes running MOSS
Ports: 9000
Purpose: Port for MOSS
Parameter and File where Port is Configured: The moss.port option in /opt/mapr/conf/moss.conf.
CAUTION
The default port for S3 Gateway is also 9000. If you run S3 Gateway and Object Store, change one of the ports to avoid conflicts.
S3 Gateway
Source IP: Nodes accessing the S3 Gateway server
Destination IP: Nodes running the S3 Gateway server
Ports: 9000
Purpose: Port for the S3 Gateway server
Parameter and File where Port is Configured: The ports option in /opt/mapr/objectstore-client/objectstore-client-<version>/conf/minio.json
NFS
Source IP: Nodes/clients accessing the filesystem via the NFS protocol
Destination IP: Nodes running Data Fabric NFS Services
Ports: 2049
Purpose: NFSv3 or NFSv4 access to the file system
Parameter and File where Port is Configured: Not Applicable
NFS
Source IP: Nodes running NFS services
Destination IP: Nodes running NFS services
Ports: 9997, 9998
Purpose: NFS VIP Management
Parameter and File where Port is Configured: /opt/mapr/conf/nfsserver.conf
NodeManager JMX Port
Source IP: Nodes running NodeManager
Destination IP: NodeManager JMX Port
Ports: 8027
Purpose: The port on which Collectd gathers metrics from NodeManager nodes via JMX.
Parameter and File where Port is Configured:Not Applicable
NodeManager
Source IP: Not Applicable
Destination IP: Nodes running NodeManager
Ports: 8099
Purpose: The node manager manages the health of each node in the cluster.
Parameter and File where Port is Configured:yarn.nodemanager.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
NodeManager Localizer RPC
Source IP: Not Applicable
Destination IP: Nodes running NodeManager
Ports: 8040
Purpose: The port that node manager uses to localize resources for a node. With localization, remote resources are downloaded to the local filesystem for access.
Parameter and File where Port is Configured: yarn.nodemanager.localizer.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
NodeManager Web UI and REST APIs
Source IP: External Web browsers and REST clients accessing NodeManager services in a non-secure cluster
Destination IP: Nodes running NodeManager services in a non-secure cluster
Ports: 8042
Purpose: NodeManager HTTP port
Parameter and File where Port is Configured: yarn.nodemanager.webapp.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
NodeManager Web UI and REST APIs
Source IP: External Web browsers and REST clients accessing NodeManager services in a secure cluster
Destination IP: Nodes running NodeManager services in a secure cluster
Ports: 8044
Purpose: NodeManager HTTPS port
Parameter and File where Port is Configured: yarn.nodemanager.webapp.https.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
OpenTSDB
Source IP: OpenTSDB clients, such as Collectd.
Destination IP: Nodes running OpenTSDB for monitoring use cases.
Ports: 4242
Purpose: Collectd uses this port to write metrics to OpenTSDB.
Parameter and File where Port is Configured: You can configure a different port for monitoring use cases when you run configure.sh script with the -OT parameter.
Port Mapper
Source IP: Nodes running Data Fabric NFS Services
Destination IP: Nodes/clients accessing the filesystem using the NFS protocol
Ports: 111
Purpose: RPC Portmap services used to connect to the file system using NFSv3
Parameter and File where Port is Configured: Not Applicable
Ranger (nonsecure Admin UI for http)
Source IP: Nodes or clients accessing the Ranger Admin (UI/API) service
Destination IP: Nodes running the Ranger Admin service
Ports: 6080
Purpose: Used by Ranger Admin clients to access the Ranger Admin service
Parameter and File where Port is Configured: ranger.service.http.port in /opt/mapr/ranger/ranger-2.3.0/ranger-admin/conf/ranger-admin-site.xml
Ranger (secure Admin UI for https)
Source IP: Nodes or clients accessing the Ranger Admin (UI/API) service
Destination IP: Nodes running the Ranger Admin service
Ports: 6182
Purpose: Used by Ranger Admin clients to access the Ranger Admin service
Parameter and File where Port is Configured: ranger.service.https.port in /opt/mapr/ranger/ranger-2.3.0/ranger-admin/conf/ranger-admin-site.xml
Ranger Usersync
Source IP: Nodes or clients accessing the Ranger Usersync service
Destination IP: Nodes running the Ranger Usersync service
Ports: 5151
Purpose: Used by the Ranger Admin to access the Ranger Usersync service
Parameter and File where Port is Configured: ranger.usersync.port in /opt/mapr/ranger/ranger-2.3.0/ranger-usersync/conf/ranger-ugsync-site.xml
ResourceManager JMX Port
Source IP: Nodes running ResourceManager
Destination IP: ResourceManager JMX port
Ports: 8025
Purpose: The port on which Collectd gathers metrics from the ResourceManager using JMX.
Parameter and File where Port is Configured: Not Applicable
ResourceManager Admin RPC
Source IP: Applications that access the ResourceManager
Destination IP: Nodes running ResourceManager
Ports: 8033
Purpose: The port that applications use to access the ResourceManager RPC
Parameter and File where Port is Configured: yarn.resourcemanager.admin.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
ResourceManager Client RPC
Source IP: Clients that submit YARN applications
Destination IP: Nodes running ResourceManager
Ports: 8032
Purpose: The port that clients use to access the YARN applications
Parameter and File where Port is Configured: yarn.resourcemanager.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
ResourceManager Resource Tracker RPC (for NodeManagers)
Source IP: Applications that access the ResourceManager
Destination IP: Nodes running ResourceManager
Ports: 8031
Purpose: The port that applications use to access the Resource Manager Tracker RPC
Parameter and File where Port is Configured: yarn.resourcemanager.resource-tracker.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
ResourceManager Scheduler RPC (for ApplicationMasters)
Source IP: Applications that access the ResourceManager
Destination IP: Nodes running ResourceManager
Ports: 8030
Purpose: The port on which the applications in the cluster talk to the ResourceManager.
Parameter and File where Port is Configured: yarn.resourcemanager.scheduler.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
ResourceManager Web UI (HTTP)
Source IP: Clients that access ResourceManager UI in a non-secure cluster
Destination IP: Nodes running ResourceManager master in a non-secure cluster
Ports: 8088
Purpose: ResourceManager Web UI
Parameter and File where Port is Configured: yarn.resourcemanager.webapp.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
ResourceManager Web UI (HTTPS)
Source IP: Clients that access ResourceManager UI in a secure cluster
Destination IP: Nodes running ResourceManager master in a secure cluster
Ports: 8090
Purpose: ResourceManager Web UI
Parameter and File where Port is Configured: yarn.resourcemanager.webapp.address in /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml
Shuffle HTTP
Source IP: Not Applicable
Destination IP: Shuffle HTTP
Ports: 13562
Purpose: The port that MapReduce Shuffle uses. Transferring the map outputs to reducer inputs in sorted form is the shuffle operation.
Parameter and File where Port is Configured: Not Applicable
Spark Standalone Master (RPC)
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 7077
Purpose: The port on which to submit jobs in a Spark standalone cluster.
Parameter and File where Port is Configured: SPARK_MASTER_PORT in SPARK_HOME/conf/spark-env.sh
Spark Standalone Master (Web UI)
Source IP: Nodes/clients accessing Spark services in a non-secure cluster
Destination IP: Nodes running Spark services in a non-secure cluster
Ports: 8580
Purpose: The port on which browsers connect to Spark master in a non-secure Spark standalone cluster.
Parameter and File where Port is Configured: SPARK_MASTER_WEBUI_PORT in SPARK_HOME/conf/spark-env.sh
Spark Standalone Master (Web UI)
Source IP: Nodes/clients accessing Spark services in a secure cluster
Destination IP: Nodes running Spark services in a secure cluster
Ports: 8980
Purpose: The port on which browsers connect to a Spark master in a secure Spark standalone cluster.
Parameter and File where Port is Configured: SPARK_MASTER_WEBUI_PORT in SPARK_HOME/conf/spark-env.sh
Spark Standalone Worker
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 8081
Purpose: The port on which browsers connect to Spark workers in a Spark standalone cluster.
Parameter and File where Port is Configured: SPARK_WORKER_WEBUI_PORT in SPARK_HOME/conf/spark-env.sh
Spark Thrift Server (if start and stop server using Spark scripts)
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 10000
Purpose: The port on which JDBC clients connect to Spark Thrift server.
Parameter and File where Port is Configured: hive.server2.thrift.port in SPARK_HOME/conf/hive-site.xml
Spark Thrift Server (if start and stop server through Warden, starting in EEP 4.0)
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 2304
Purpose: The port on which JDBC clients connect to Spark Thrift server.
Parameter and File where Port is Configured: hive.server2.thrift.port in SPARK_HOME/conf/hive-site.xml
Spark History Server
Source IP: Clients that access Spark Job History in a non-secure cluster
Destination IP: Nodes running Spark History Server in a non-secure cluster
Ports: 18080
Purpose: The port on which browsers connect to a non-secure Spark history server.
Parameter and File where Port is Configured: spark.history.ui.port in SPARK_HOME/conf/spark-default.conf"
Spark History Server
Source IP: Clients that access Spark Job History in a secure cluster
Destination IP: Nodes running Spark History Server in a secure cluster
Ports: 18480
Purpose: The port on which browsers connect to a secure Spark history server.
Parameter and File where Port is Configured: spark.ssl.historyServer.port in SPARK_HOME/conf/spark-defaults.conf (starting from Spark-2.2.1)
Spark External Shuffle Service
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 7337
Purpose: The port on which Spark jobs connect to External Shuffle server.
Parameter and File where Port is Configured: spark.shuffle.service.port in SPARK_HOME/conf/spark-default.conf
Tez Shuffle
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 13563
Purpose: Port to communicate with the Tez Shuffler. A Tez specific shuffle handler allows data to be shuffled in a way that takes advantage of the new features in Tez
Parameter and File where Port is Configured: Not Applicable
Timeline Server
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 10200
Purpose: Hadoop IPC port used for internal communication in Hadoop
Parameter and File where Port is Configured: Not Applicable
Timeline Server Web Interface (HTTP)
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 8188
Purpose: Non-secure web access for the Timeline Server. The Timeline Server allows storage and retrieval of an application’s current and historic information in a generic fashion.
Parameter and File where Port is Configured: Not Applicable
Timeline Server Web Interface (HTTPS)
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 8190
Purpose: Secure web access for the Timeline Server
Parameter and File where Port is Configured: Not Applicable
Tomcat Port (Hive-on-Tez UI)
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 9383
Purpose: The non-secure port to access the Tez UI. Hive-on-Tez speeds up execution of Hive queries.
Parameter and File where Port is Configured: Not Applicable
Tomcat SSL Port (Hive-on-Tez UI)
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 9393
Purpose: The secure port to access the Tez UI.
Parameter and File where Port is Configured: Not Applicable
Web UI
Source IP: External web browser accessing either a non-secure or a secure cluster
Destination IP: Nodes running the Control System Web UI in a non-secure or a secure cluster
Ports: 8443
Purpose: Control System Web UI
Parameter and File where Port is Configured: /opt/mapr/apiserver/conf/properties.cfg
Zeppelin
Source IP: Not Applicable
Destination IP: Not Applicable
Ports: 9995
Purpose: The port to connect to the Zeppelin Docker container
Parameter and File where Port is Configured: Configurable by setting ZEPPELIN_SSL_PORT when running the Zeppelin Docker image
ZooKeeper
Source IP: Nodes running ZooKeeper services, clients executing ZooKeeper API calls
Destination IP: Nodes running ZooKeeper services
Ports: 5181
Purpose: ZooKeeper API calls
Parameter and File where Port is Configured:
  • /opt/mapr/zookeeper/zookeeper-<version>/conf/zoo.cfg
  • /opt/mapr/conf/warden.conf, /opt/mapr/conf/cldb.conf
  • /opt/mapr/hbase/hbase-<version>/conf/hbase-site.xml
  • /opt/mapr/hive/hive-<version>/conf/hive-site.xml
ZooKeeper follower-to-leader Communication
Source IP: Nodes running ZooKeeper services
Destination IP: Nodes running ZooKeeper services
Ports: 2888
Purpose: ZooKeeper Server > Server Communication
Parameter and File where Port is Configured: /opt/mapr/zookeeper/zookeeper-<version>/conf/zoo.cfg
ZooKeeper Leader Election
Source IP: Nodes running ZooKeeper services
Destination IP: Nodes running ZooKeeper services
Ports: 3888
Purpose: ZooKeeper Server > Server Communication
Parameter and File where Port is Configured: /opt/mapr/zookeeper/zookeeper-<version>/conf/zoo.cfg