Configure Fluentd Services to Write to Elasticsearch Nodes on the Same Rack
On clusters with high-density racks, ensure you have at least one Elasticsearch server per rack and configure each Fluentd service to write to Elasticsearch nodes that run on the same rack as the Fluentd service. This configuration minimizes the impact of log aggregation on other processes that run on the cluster and in particular, minimizes the amount of backbone bandwidth used by the log aggregation.
About this task
Complete the following steps on each node that runs the Fluentd service.
Procedure
-
Open the
/opt/mapr/fluentd/fluentd-<version>/etc/fluentd/es_config.conf
file. -
Edit the
hosts
property to only include Elasticsearch nodes that are on the same rack as the Fluentd service.Example:hosts qa-node90:9200,qa-node91.qa.lab:9200,qa-node92.qa.lab:9200
-
Restart Fluentd.
maprcli node services -name fluentd -nodes <space separated list of fluentd nodes> -action restart
What to do next
WARNING
Changes to the es_config.conf files are overridden by configure.sh.
Therefore, you will need reconfigure the hosts property in the es_config.conf file
after configure.sh is run on Fluentd nodes.