Writing a MapReduce Job in Java

A complete Hadoop MapReduce job in Java is made of three cooperating pieces: a Driver class containing the main() method that configures and submits the job, a Mapper subclass implementing map(), and a Reducer subclass implementing reduce() (with an optional Combiner, often the same class as the Reducer for associative operations). The Driver constructs a Configuration and a Job object bound to it, wires in the mapper, reducer, combiner, and input/output paths, then calls job.waitForCompletion(true) to submit the job to the cluster (or LocalJobRunner in local mode) and block until it finishes, returning a boolean success flag the driver typically uses to set the process exit code.

Both Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> and Reducer<KEYIN, VALUEIN, KEYOUT, VALUEOUT> are generic classes parameterized by their input and output key/value types, and every type used must implement Hadoop's Writable interface (for values) or WritableComparable (for keys, since keys must be sortable) — common built-ins include Text, IntWritable, LongWritable, DoubleWritable, and NullWritable. Critically, the reducer's KEYIN/VALUEIN types must exactly match the mapper's KEYOUT/VALUEOUT types, since that's the intermediate contract the framework enforces at job submission time; a mismatch, like a mapper emitting IntWritable values while the reducer expects Text, causes a runtime ClassCastException rather than a compile-time error, because the framework can't fully verify generic type parameters across the shuffle boundary.

A typical driver builds the job with Job.getInstance(conf, "job name"), calls setJarByClass() so Hadoop can locate the job's JAR on the classpath, sets setMapperClass(), setCombinerClass() (optional), and setReducerClass(), declares setOutputKeyClass() and setOutputValueClass() for the final output types, optionally overrides setMapOutputKeyClass()/setMapOutputValueClass() when intermediate types differ from final output types, and finally wires FileInputFormat.addInputPath() and FileOutputFormat.setOutputPath() before calling waitForCompletion(). Hadoop refuses to run if the output path already exists, as a safety measure against accidentally overwriting a previous job's results, so driver code (or the shell script invoking it) commonly deletes or timestamps the output directory before resubmitting during iterative development.

Before submitting to a real cluster, it's standard practice to run the job in local mode (mapreduce.framework.name=local, the default when no cluster config is present) against a small sample dataset, or to use a unit testing library like MRUnit or plain JUnit tests against the Mapper/Reducer classes directly by calling map()/reduce() with mocked Context objects, catching logic bugs cheaply before burning cluster time. Once packaged into a JAR (typically with dependencies shaded in via Maven's shade plugin or a similar mechanism), the job is submitted with hadoop jar mycode.jar com.example.WordCountDriver /input/path /output/path, and progress and errors can be inspected afterward with yarn logs -applicationId <id> or through the YARN ResourceManager web UI.