Challenges of Scaling Big Data Jobs

In my reading and research so far [1], I've found (not surprisingly) that it's easy to run into all kinds of challenges when putting together a program to process large sets of data..especially if you were to write this from the ground up.

Using word counting as yet another example, one can see where complexity and bottlenecks can quickly snowball as the amount of data to process grows.

If one were to begin thinking about a way to process the number of times a particular word or set of words shows up across several documents, the idea may start out as a program that rather simply loops through a bunch of documents, counts the words, and then displays the results.

That's all fine and good, but, if your number of documents becomes huge, this can easily bog down a single machine and it may take an unreasonably long time to get the results.

This could be sped up by adding more systems into the mix and have the counting work be distributed over those systems,  with each machine processing a specific chunk of the data.   You can then add in another part to your program that sends the results to a dedicated system that aggregates the results and gives them back to the user.

The drawback to the above, though, would come if you had all your data stored on this dedicated machine.    Potential bottlenecks would be that the systems's bandwidth wouldn't be able to keep up with all the machines reading from and storing information to it.   To alleviate this, you can store a set number of documents on each the processing nodes, and those machines will only process the documents stored in them.  When the processing is done, the results can be sent to the aggregator.

We're done, right?  I wish...I found that one could also run into issues where the processing of all these unique words can easily eat up the memory capacity of the machine tasked with aggregating all of these results.

This can be solved by again splitting up the work among several machines.   For example, you can break the processing job up into 26 nodes (one for each letter of the alphabet) and each node will handle only the words that begin with its designated letter.  This means that you'll have to modify your original program once again to divvy out the work to each respective node.

Whew!  You can see where this gets crazy rather quickly, and we haven't even taken into account what to do if one of these nodes were to die on us.

MapReduce and Hadoop are designed to take care of a lot of this plumbing for us (in other words, the heavy lifting portions of partitioning data, fault tolerance, inter-node communication, etc. are built-in).  I'll be writing on how that can help with the above word count example in later posts.

[1]  The example that I just wrote about came from the so-far-awesome book titled:  Hadoop in Action.  I suggest grabbing a copy if you'd like to learn more.