When putting together a MapReduce program, it's important to make sure that you are using your reducer tasks efficiently.
Rookie mistakes (apparently) include sending everything to a single reducer, which kinda defeats the purpose of distributed computing.
With multiple reducers, Hadoop uses built-in "HashPartitioning" to help determine which reducer a key/value pair from the mapper will go to. The key is hashed to determine the reducer to send it to.
Sometimes, even with HashPartitioning in place, another mistake can occur when key/value pairs emitted from a map job with similar data in them are sent to different reducers. The result of this is that the information would accidentally be processed multiple time, leading to erroneous results. From the book I'm reading, they used the following flight information data as an example, with San Francisco being the departure city:
(San Francisco, Los Angeles) Passenger1
(San Francisco, Dallas) Passenger2
...if one isn't careful, the same rows can be sent to different reducers, which isn't a good thing. The departure city would end up being processed twice.
To avoid the above mistake, a programmer can customize the partitioning portions of their MapReduce jobs so that common departure points are sent to the same reducer.
The process of sending data output from the mapper to various reducer tasks in the cluster is referred to as 'shuffling'.
No comments:
Post a Comment