Break it on Down

Some notes I took breaking down MapReduce.  Many thanks to this:


Map Portion 
    Transform (Map) Input Values to Output Values
    
    Input is Key/Value Pairs <K1, V1>
        example - Key = Line number in a text document, Value = Text String of that Line
    Map Function is repeatedly called to pull in all Input data and process it into output pairs.
        example - The Map Function scans the lines and counts the words in the input. 
    Output is another set of Key/Value Pairs <K2, V2>
        example - Key = word found, Value = count (i.e. number of times word is found)
    Map Output is the Input to the Reducer.
    
    ASCII / Graphical Example:
    
    Key=Line 1  Value="to be or not to be" ---> Mapper ---> Key = "to" , Value = 2
                                                                                  Key = "be" , Value = 2
                                                                                  Key = "or" , Value = 1
                                                                                  Key = "not", Value = 1

    Key=Line 2  Value="to do is to be" ---> Mapper ---> Key = "to" , Value = 2
                                                                            Key = "do" , Value = 1
                                                                            Key = "is" , Value = 1
                                                                            Key = "be" , Value = 1
                                                             
    Input Key Value (Line number) is not used / ignored by the mapper (this is pretty typical).
    
    Multiple Mappers can be running in parallel generating outputs for the Reducer.


Reduce Portion
    Reduce (or Merge) Values generated from the Mapper.
        Sometimes this is optional, as the Mapper ends up doing all the work.
    Reduce Function - Steps through and combines all the Values
        example Values - Sums, Counts, Strings
    Output can then be:  Written to a file, Displayed on Screen, Loaded into a DataBase, or sent to another MapReduce Job,    etc.
        * By sending to another MapReduce Job, pipelines can be set up to break large amounts of analytic data into small pieces.
    
    ASCII / Graphical Example:
    
     Key = "to" , Value = 2                           Key = "to" , Value = 4
     Key = "be" , Value = 2                           Key = "be" , Value = 3
     Key = "or" , Value = 1                           Key = "or" , Value = 1
     Key = "not", Value = 1  ---> Reducer --->  Key = "not", Value = 1
     Key = "to" , Value = 2                           Key = "do" , Value = 1
     Key = "do" , Value = 1                          Key = "is" , Value = 1
     Key = "is" , Value = 1
     Key = "be" , Value = 1
    
Big Picture
    Input Data is split into chunks and sent to Mappers for processing.
    Output is passed to Reducers for consolidation.
    Processing is scaled by adding more Mappers and Reducers.
    
Hadoop - A MapReduce implementation.
    If MapReduce is the blue print, Hadoop is the actual building.
    Main Components - Storage(HDFS) & MapReduce
        Storage = HDFS
    Hadoop glues these components together and provides reliability, scalability, and management (tracking & restaring jobs, keeping track of nodes, etc).
        Data is broken into chunks and distributed to nodes in the cluster.
        Each node works on data stored on it.