Your First Hadoop Job
The canonical first Hadoop program is WordCount: read text files from HDFS, count occurrences of each word, and write the results back to HDFS. It's simple enough to write in under 100 lines of Java yet touches every core MapReduce concept — a Mapper class, a Reducer class, and a Driver (main method) that configures and submits the Job.
Cricket analogy: WordCount as the first Hadoop program is like a beginner's first net session facing only straight half-volleys — simple enough to build confidence, yet it still requires footwork, timing, and shot selection, the same fundamentals every real innings needs.
Writing the Mapper and Reducer
The Mapper's map() method receives a line offset (LongWritable key) and a line of text (Text value), tokenizes it, and emits a (word, 1) pair for every token via context.write(); the Reducer's reduce() method receives a word and an Iterable of all the 1s emitted for it across every mapper, sums them, and writes the final (word, totalCount) pair — Hadoop's shuffle-and-sort phase between map and reduce guarantees all values for the same key arrive at one reducer already grouped.
Cricket analogy: Each mapper emitting a (word,1) pair for every token is like every fielder independently signaling 'catch taken' the instant it happens; the shuffle-and-sort phase grouping all pairs for one key at one reducer is like the third umpire collecting every camera angle of one specific appeal before ruling.
Compiling, Packaging, and Submitting
You compile the Mapper, Reducer, and Driver classes, package them with their dependencies into a single JAR, upload your input text file into HDFS with hdfs dfs -put, then launch the job with hadoop jar wordcount.jar WordCount /input /output — Hadoop will refuse to run if the output directory already exists, a deliberate safeguard against silently overwriting previous results.
Cricket analogy: Hadoop refusing to run if the output directory already exists is like the groundstaff refusing to re-mark a pitch that's already prepared for tomorrow's Test — you don't silently overwrite a match surface that's already set up, you must clear it deliberately first.
public class WordCount {
public static class TokenizerMapper
extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) sum += val.get();
context.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}hdfs dfs -put input.txt /input
hadoop jar wordcount.jar WordCount /input /output
hdfs dfs -cat /output/part-r-*Reading the Results
MapReduce output isn't a single sorted file — by default the job writes one part-r-00000, part-r-00001, etc. file per reducer, so to see the combined result you typically run hdfs dfs -cat /output/part-r-* or hdfs dfs -getmerge /output ./local-result.txt rather than expecting one neat file.
- WordCount is the canonical first Hadoop program, touching Mapper, Reducer, and Driver concepts.
- The Mapper tokenizes each line and emits a (word, 1) pair per token.
- The Reducer sums all the 1s for a given word into a total count.
- Hadoop's shuffle-and-sort phase groups all values for the same key at one reducer.
- Code is compiled and packaged into a single JAR before being submitted with hadoop jar.
- Hadoop refuses to run if the specified output directory already exists.
- MapReduce output is split into multiple part-r-NNNNN files, one per reducer, not a single file.
Practice what you learned
1. What does a MapReduce Mapper's map() method typically emit for WordCount?
2. What guarantees that all values for the same key arrive at one reducer already grouped?
3. What happens if you run a MapReduce job with an output directory that already exists?
4. How is MapReduce output typically structured on HDFS?
5. What must happen before you can run 'hadoop jar wordcount.jar WordCount /input /output'?
Was this page helpful?
You May Also Like
What Is Hadoop?
An introduction to Apache Hadoop, the open-source framework that popularized distributed storage and processing of massive datasets on commodity hardware.
Hadoop Daemons
What NameNode, DataNode, ResourceManager, NodeManager, and the other Hadoop background processes actually do, and how they interact.
The Hadoop Ecosystem
A tour of the major projects that surround core Hadoop — Hive, Pig, HBase, Sqoop, Flume, ZooKeeper, and Oozie — and how they fit together.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics