Your First Hadoop Job

Analogy categoryAnalogies

The canonical first Hadoop program is WordCount: read text files from HDFS, count occurrences of each word, and write the results back to HDFS. It's simple enough to write in under 100 lines of Java yet touches every core MapReduce concept — a Mapper class, a Reducer class, and a Driver (main method) that configures and submits the Job.

🏏

Cricket analogy: WordCount as the first Hadoop program is like a beginner's first net session facing only straight half-volleys — simple enough to build confidence, yet it still requires footwork, timing, and shot selection, the same fundamentals every real innings needs.

Writing the Mapper and Reducer

The Mapper's map() method receives a line offset (LongWritable key) and a line of text (Text value), tokenizes it, and emits a (word, 1) pair for every token via context.write(); the Reducer's reduce() method receives a word and an Iterable of all the 1s emitted for it across every mapper, sums them, and writes the final (word, totalCount) pair — Hadoop's shuffle-and-sort phase between map and reduce guarantees all values for the same key arrive at one reducer already grouped.

🏏

Cricket analogy: Each mapper emitting a (word,1) pair for every token is like every fielder independently signaling 'catch taken' the instant it happens; the shuffle-and-sort phase grouping all pairs for one key at one reducer is like the third umpire collecting every camera angle of one specific appeal before ruling.

Compiling, Packaging, and Submitting

You compile the Mapper, Reducer, and Driver classes, package them with their dependencies into a single JAR, upload your input text file into HDFS with hdfs dfs -put, then launch the job with hadoop jar wordcount.jar WordCount /input /output — Hadoop will refuse to run if the output directory already exists, a deliberate safeguard against silently overwriting previous results.

🏏

Cricket analogy: Hadoop refusing to run if the output directory already exists is like the groundstaff refusing to re-mark a pitch that's already prepared for tomorrow's Test — you don't silently overwrite a match surface that's already set up, you must clear it deliberately first.

java

public class WordCount {
  public static class TokenizerMapper
      extends Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(LongWritable key, Text value, Context context)
        throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
      }
    }
  }

  public static class IntSumReducer
      extends Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text key, Iterable<IntWritable> values, Context context)
        throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) sum += val.get();
      context.write(key, new IntWritable(sum));
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

bash

hdfs dfs -put input.txt /input
hadoop jar wordcount.jar WordCount /input /output
hdfs dfs -cat /output/part-r-*

Reading the Results

MapReduce output isn't a single sorted file — by default the job writes one part-r-00000, part-r-00001, etc. file per reducer, so to see the combined result you typically run hdfs dfs -cat /output/part-r-* or hdfs dfs -getmerge /output ./local-result.txt rather than expecting one neat file.

WordCount is the canonical first Hadoop program, touching Mapper, Reducer, and Driver concepts.
The Mapper tokenizes each line and emits a (word, 1) pair per token.
The Reducer sums all the 1s for a given word into a total count.
Hadoop's shuffle-and-sort phase groups all values for the same key at one reducer.
Code is compiled and packaged into a single JAR before being submitted with hadoop jar.
Hadoop refuses to run if the specified output directory already exists.
MapReduce output is split into multiple part-r-NNNNN files, one per reducer, not a single file.

Practice what you learned

1. What does a MapReduce Mapper's map() method typically emit for WordCount?

2. What guarantees that all values for the same key arrive at one reducer already grouped?

3. What happens if you run a MapReduce job with an output directory that already exists?

4. How is MapReduce output typically structured on HDFS?

5. What must happen before you can run 'hadoop jar wordcount.jar WordCount /input /output'?

Was this page helpful?

Hadoop Daemons

Related Study Notes in Programming

Browse all study notes

Programming

Apache Spark Study Notes

Programming · 30 topics

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

Apache Airflow Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#HadoopStudyNotes#YourFirstHadoopJob#Hadoop#Job#Writing#Mapper#StudyNotes#SkillVeris#ExamPrep

Your First Hadoop Job