Administering Your Cluster - Filesystem Stuff

A few weeks ago, I talked about the various commands you can use in hadoop to view and list stuff in your HDFS filesystem.

There are a few more commands out there that can give you some more details on the overall health of your HDFS.

To check your HDFS filesystem you can run, you guessed it, "fsck". It's called with a file path and will recursively check everything that falls under that path.  For example, if we ran it on the root of our dfs filesystem we'd see something like:

$ bin/hadoop fsck /

Status: HEALTHY

 Total size: 489501082 B

 Total dirs: 50

 Total files: 56

 Total blocks (validated): 53 (avg. block size 9235869 B)

 Minimally replicated blocks: 53 (100.0 %)

 Over-replicated blocks: 0 (0.0 %)

 Under-replicated blocks: 53 (100.0 %)

 Mis-replicated blocks: 0 (0.0 %)

 Default replication factor: 2

 Average block replication: 1.0

 Corrupt blocks: 0

 Missing replicas: 0 (0.0 %)

 Number of data-nodes: 1

 Number of racks: 1

FSCK ended at Wed Aug 24 12:09:09 CDT 2011 in 33 milliseconds

I've left out some of the output, but, fsck will also print out the status of every file.   A dot means that the file is healthy.  If it is other than healthy an explanation of why is displayed along with the summary above.  

The summary lets you know how the status of the filesystem, the total size, total number of files and directories, any corrupt blocks, data-nodes, etc.  Here I've just got one node set up.  

You can also get more detailed information by adding the  -files, -blocks, -locations, and -racks options.   Be careful when using these, though, because each successive option relies on the preceding option to be used...i.e. the -blocks option requires the -files option to be used.

To get a quick view of what all the options do, you can type:

bin/hadoop fsck

..with no path and all of the different options will be displayed to you on the console.

While fsck gives you the status of the files in your cluster, the "dfsadmin" command will provide reports on each DataNode.  The -report option will give you an overall view:

$ bin/hadoop dfsadmin -report

Configured Capacity: 241949212672 (225.33 GB)

Present Capacity: 225436766208 (209.95 GB)

DFS Remaining: 224943128576 (209.49 GB)

DFS Used: 493637632 (470.77 MB)

DFS Used%: 0.22%

Under replicated blocks: 53

Blocks with corrupt replicas: 0

Missing blocks: 0

-------------------------------------------------

Datanodes available: 1 (1 total, 0 dead)

Name: 192.168.1.12:50010

Decommission Status : Normal

Configured Capacity: 241949212672 (225.33 GB)

DFS Used: 493637632 (470.77 MB)

Non DFS Used: 16512446464 (15.38 GB)

DFS Remaining: 224943128576(209.49 GB)

DFS Used%: 0.2%

DFS Remaining%: 92.97%

Last contact: Wed Aug 24 12:20:02 CDT 2011

Again, we see a pretty good summary of our HDFS filesystem and the current state of our nodes.  If there were more than one node here, we'd see a listing for each machine in the "Datanodes available" section of the report.

Hadoop also has utilities built into it to set permissions, quotas, and you can also set up a Trash feature to guard against accidental file deletion.