So for the last week, I've begun getting my feet wet with Hadoop, which is an open-source implementation of Google's MapReduce algorithm. It's primarily used in large clusters for processing large sets of data ("Big Data" is the buzzword most commonly used these days).
One of the goals of Hadoop is to reduce the amount of time it takes to process large amounts of data. A typical, standalone "beefy" machine, for example, would take serveral hours to process a multi-terabyte datas set, whereas a Hadoop cluster running on several commodity nodes would help cut the processing time down significantly. [1]
After reading over some background info on Hadoop, I was initially intimidated at the prospects of what it was going to take to set up a test rig so I could start feeling my way around. I envisioned a Hellish time of compiling a bunch of source code and all of the "fun" that could sometimes go with it.
Thankfully, I ran into this. It turns out you can set up your own micro-scale hadoop system on a single machine, or even a virtual machine if you like. The setup was relatively easy to follow, and all it took to install hadoop was to basically unzip a tar file. Once all the pieces were in place, the sample programs were a breeze to run.
What's next?
My next trick will be setting up a multi-node Hadoop cluster. I'll definitely be blogging about my experiences here. I'd also like to dive more into Hadoop's HDFS filesystem and learn more about the commands.
[1] In addition to the Apache homepage for Hadoop, I found this website to so far be a great resource, and I plan on purchasing the book that goes along with it: http://www.manning.com/lam/ The book seems to really cut to the chase of what you need to know about Hadoop and MapReduce.
No comments:
Post a Comment