The Word Count Job
Word count is the 'hello world' of Hadoop: it reads text files, counts how often each word appears, and outputs the totals. Despite its simplicity, it exercises every core MapReduce concept — a Mapper that emits key-value pairs, an automatic shuffle-and-sort phase, a Reducer that aggregates values per key, and a Driver class that wires everything together and submits the job to YARN. Understanding it thoroughly makes far more complex MapReduce jobs approachable, because nearly every batch job is a variation on this same map-shuffle-reduce skeleton.
Cricket analogy: Is like learning to bowl a straight, line-and-length delivery before attempting a doosra — word count is the fundamental technique every Hadoop developer masters before tackling complex joins or graph algorithms.
Writing the Mapper
The Mapper class extends org.apache.hadoop.mapreduce.Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>, where for text input KEYIN is LongWritable (the byte offset) and VALUEIN is Text (the line itself). The map() method tokenizes each line — typically with StringTokenizer or a split on whitespace — and calls context.write(word, one) for every token, emitting a Text key and an IntWritable value of 1. Note that no aggregation happens in the mapper itself; it simply emits one (word, 1) pair per occurrence, leaving summation entirely to the reducer.
Cricket analogy: Is like a scorer at the boundary rope who, for every single run scored, jots down a slip reading 'Player X: 1' — no totaling happens at the rope, they just record each raw event as it occurs.
public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
@Override
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
StringTokenizer tokenizer = new StringTokenizer(value.toString());
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken().toLowerCase());
context.write(word, one); // emit (word, 1)
}
}
}Writing the Reducer and Driver
The Reducer extends Reducer<Text, IntWritable, Text, IntWritable> and receives, for each unique word, an Iterable of all the IntWritable values the shuffle phase grouped together — the reduce() method simply sums them and writes the total. The Driver's main() method builds a Job object, sets the jar-by-class, mapper/reducer/combiner classes, output key/value types, and input/output paths, then calls job.waitForCompletion(true) to submit it to YARN and block until it finishes. Setting the same class as both the Combiner and Reducer (job.setCombinerClass(WordCountReducer.class)) is a standard optimization: it pre-aggregates counts locally on each mapper node before the shuffle, cutting network traffic substantially when a word like 'the' appears thousands of times in one split.
Cricket analogy: Is like a team's chief scorer collecting every boundary-rope slip for 'Player X' after the innings and adding them into one final total — that's the reducer; a combiner is like each fielder pre-adding their own slips before handing them in.
public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
@Override
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
public class WordCountDriver {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCountDriver.class);
job.setMapperClass(WordCountMapper.class);
job.setCombinerClass(WordCountReducer.class); // local pre-aggregation
job.setReducerClass(WordCountReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}Running the Job
Once compiled into a jar, the job is submitted with hadoop jar wordcount.jar WordCountDriver /input/books /output/wordcounts, where the input path must exist in HDFS and the output path must NOT already exist, or the job fails fast to avoid silently overwriting results. After completion, results land as one or more part-r-00000-style files in the output directory, and the job's counters (visible in the YARN ResourceManager UI or via mapred job -counter) report useful diagnostics like the number of input records, map output bytes, and combiner input/output records, which reveal how much the combiner reduced shuffle traffic.
Cricket analogy: Is like submitting a completed scorecard to the match referee — the input pitch report must exist beforehand, and you can't submit to a slot where a scorecard for that match already exists, since that would overwrite official records.
Always set a Combiner when your reduce function is associative and commutative (like sum, min, max) — it runs on each mapper's output before the shuffle and can cut network I/O by an order of magnitude for high-frequency keys like common English words.
- The Mapper emits one (word, 1) pair per token; no aggregation happens at map time.
- The shuffle-and-sort phase automatically groups all values for the same key before they reach the Reducer.
- The Reducer sums the Iterable<IntWritable> of counts for each unique word key.
- The Driver wires mapper, combiner, reducer, and I/O paths into a Job and submits it via job.waitForCompletion(true).
- Using the Reducer class as the Combiner pre-aggregates counts locally, reducing shuffle network traffic.
- The output directory must not already exist, or the job fails to prevent silently overwriting results.
- Job counters reveal diagnostics like combiner input/output record counts, showing how much shuffle traffic was saved.
Practice what you learned
1. What does the Mapper emit for each word it encounters in a line?
2. What happens between the Mapper and Reducer phases?
3. Why is setting a Combiner (using the same class as the Reducer) beneficial for word count?
4. What happens if the specified output directory already exists when you submit a MapReduce job?
5. In the Reducer's reduce() method signature Reducer<Text, IntWritable, Text, IntWritable>, what does the second IntWritable represent?
Was this page helpful?
You May Also Like
Hadoop vs Spark
A technical comparison of MapReduce's disk-based batch model and Spark's in-memory DAG execution, and when to reach for each.
Hadoop Best Practices
A practical guide to designing, tuning, and operating Hadoop clusters so they stay reliable and performant as data volumes grow.
Hadoop Quick Reference
A condensed cheat sheet of the HDFS and YARN commands, key configuration properties, and defaults you reach for daily.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics