Administering Your Cluster - Removing and Adding Nodes

There are cases where you need to take a node out of the cluster for maintenance or simply replace it with something new.  You can do this pretty easily.

To take nodes out of the cluster:

First, take the node out of hdfs:

Edit conf/hdfs-site.xml:

<property>
  <name>dfs.hosts.exclude</name>
  <value>/full/path/to/exclude/file</value>
</property>

Then use following command to get HDFS to re-read host exclude file and decommission the nodes:

./bin/hadoop dfsadmin -refreshNodes

It's also a good idea to take the nodes out of MapReduce:

Edit conf/mapred-site.xml:

<property>
  <name>mapred.hosts.exclude</name>
  <value>/full/path/to/exclude/file</value>
</property>

Then run the following so MapReduce will reconfigure the nodes:




./bin/hadoop mradmin -refreshNodes




The path to the exclude file is the path in your HDFS filesystem.  The file is just a text file that contains a list of nodes using hostname, IP address, or IP:port format.

If you want to add these nodes back at some point, clear out the exclude file and rerun the refresh commands above. Once that is done, power up the node and run:

bin/hadoop datanode

This will allow the node to re-connect with the NameNode and join back into the cluster. Also make sure the node is present on the conf/slaves file on the master node.

For a brand-new node, install hadoop on it and make sure the respective config files are updated so that it points and connects to the master node.  Then run the 'datanode' command above to join the cluster.

For optimal performance, you'll also want to rebalance filesystem once a node has been added into the cluster.  The node will be empty while the other DataNodes will already have data on it.  To rebalance this, run:

bin/start-balancer.sh 


..this will run in the background until the cluster is balanced.