No Problem!
Hadoop supports other languages via its "Streaming" API to write Hadoop programs. This is helpful if you just want to write short and simple MapReduce programs. The streaming interacts with the Unix streaming paradigm. Inputs come in through STDIN and are output via STDOUT. The input data must be text based!
An example of using streaming via Unix commands looks something like this:
bin/hadoop jar contrib/streaming/hadoop-0.20-streaming.jar -input input/file.txt -output output -mapper 'cut -f 2 -d' -reducer 'uniq'
...so in the above example, the streaming API is supplied via the streaming.jar found in the hadoop contrib folder. The mapper uses the standard cut command to extract data while the reducer uses uniq to remove all the duplicates it finds in our input file.
We can also use any executable script with Hadoop Streaming that can process a line-oriented stream from STDIN and output it to STDOUT. This includes (but not limited to) scripts written in:
python
php
No comments:
Post a Comment