Earlier, I provided a summary of some of the challenges involved in writing a large data processing job from the ground up....bottlenecks and fault-tolerance were just a couple of the elements that could end up adding complexity to an already challenging task.
Hadoop and MapReduce can help simplify the scalability challenges with many of its already built-in capabilities.
As the name implies, MapReduce programs are broken down into "Mapping" and "Reducing" phases. During mapping, the data is fed into its respective elements in the mapper while the reducer processes the outputs from the mapper and produces the result. Simply put, the mapper processes the data into something that the reducer can aggregate over. As you can see, we are saved from the headache of having to figure out how to process and aggregate the data from our earlier example.
Other functions that we had to wrestle with in our previous example (e.g. dividing up the data and shuffling tasks off to other servers) are also bulit-in to MapReduce / Hadoop.
The basic, main units of a MapReduce job consist of lists and (key/value) pairs.
When putting together a MapReduce job, you'll need to specify a mapper and reducer. The inputs to your MapReduce job must be set up as a list of key/value pairs. You can process several documents or just a single, large log file.
For the word count program, the mapper will act on a list of filenames and the content of those filenames. What is output is a list of words and the number of times the word appears in that document, for example:
<"foo", 1>
...and depending on how the program is written, we may see that pair show up multiple times for every instance of "foo" in each document.
Once all of the documents are processed, the output of the mappers are then aggregated into one big list of pairs. In this example, all pairs sharing the same "foo" keyword are grouped into a new key/value pair. It may look something like this:
<"foo", list(1,1,1,1,1)>
The above is what will be fed to the reducer. The output of the reducer will be:
<"foo", 6>
..which is the number of times foo is seen in the set of documents that we fed to the MapReduce job. This output is then written to files that can then be viewed and parsed. What you might see is:
foo 6
...and a list of other words and the frequency at which they showed up in the document set.
**Once again, this summary is based on my readings from Hadoop in Action.
No comments:
Post a Comment