What the ApplicationMaster Does
The ApplicationMaster (AM) is a framework-specific, per-application process that YARN launches inside the first container it grants for a submitted job; unlike the ResourceManager, which knows nothing about MapReduce, Spark, or Tez internals, the AM understands exactly how its own application should be split into tasks. It requests additional containers from the RM to run those tasks, monitors their progress by communicating directly with each container's NodeManager, and reports overall job status back to the client, which means YARN itself has no idea whether an individual job succeeded, only the AM tracks that.
Cricket analogy: The AM is like a specific team's team manager appointed once the franchise slot is confirmed, who then decides exactly how to use net sessions and net bowlers, something the tournament board itself has no visibility into.
Resource Negotiation: The AMRMClient Protocol
An AM negotiates resources through the AMRMClient (or the lower-level AMRMClientAsync) protocol: it calls registerApplicationMaster() once at startup, then repeatedly calls allocate(), passing a list of ResourceRequests describing desired container count, memory, vcores, and locality preferences (node-local, rack-local, or any), and receiving back both newly granted Containers and updates on containers that completed. This allocate() call also doubles as the AM's heartbeat to the RM, if it stops, the RM eventually kills the AM's container and, depending on retry policy, reschedules a fresh AM attempt.
Cricket analogy: It's like a captain radioing the ground office an over-by-over request for extra practice nets with a preference for the main ground over the side nets, and that same radio call doubling as proof the captain is still actively managing the session.
ApplicationMaster Failure and Recovery
If an AM crashes or its heartbeats stop, the RM's ApplicationsManager can restart it, controlled by yarn.resourcemanager.am.max-attempts (default 2) at the cluster level and overridable per job; each restart is a new ApplicationAttempt with a fresh ContainerId. For MapReduce, a restarted AM can optionally recover already-completed tasks from the job history rather than redoing them, but frameworks without that recovery logic must rerun the entire job from scratch, which is why long-running Spark or Tez applications typically checkpoint progress externally.
Cricket analogy: Like a team being allowed exactly two team-manager replacements per season if the original manager resigns mid-tournament, with the new manager sometimes able to review prior match notes rather than starting scouting from zero.
AMRMClientAsync<ContainerRequest> amClient =
AMRMClientAsync.createAMRMClientAsync(1000, allocListener);
amClient.init(conf);
amClient.start();
RegisterApplicationMasterResponse response =
amClient.registerApplicationMaster(appMasterHost, 0, appTrackingUrl);
Resource capability = Resource.newInstance(2048, 2);
Priority priority = Priority.newInstance(0);
ContainerRequest containerAsk =
new ContainerRequest(capability, null, null, priority);
amClient.addContainerRequest(containerAsk);
// later, on shutdown
amClient.unregisterApplicationMaster(
FinalApplicationStatus.SUCCEEDED, "job complete", null);The AM's registerApplicationMaster() call also returns the maximum container size the cluster will ever grant, so an AM must clamp its own ResourceRequests to that ceiling or the RM will reject the request outright.
- The ApplicationMaster is a per-application, framework-specific process launched inside the first container YARN grants.
- It negotiates further containers through the AMRMClient / AMRMClientAsync protocol via registerApplicationMaster() and repeated allocate() calls.
- The allocate() call doubles as the AM's heartbeat to the RM; missing it too long causes the RM to kill the AM's container.
- yarn.resourcemanager.am.max-attempts controls how many times a failed AM can be restarted as a fresh ApplicationAttempt.
- MapReduce AMs can recover completed tasks from job history on restart; many other frameworks must rerun the whole job.
- Long-running Spark or Tez applications typically checkpoint progress externally since AM restart alone doesn't guarantee task-level recovery.
Practice what you learned
1. Which client protocol does an ApplicationMaster use to request additional containers from the ResourceManager?
2. What does the AM's allocate() call serve as, in addition to requesting resources?
3. What controls how many times a failed ApplicationMaster can be restarted?
4. Why do long-running Spark or Tez applications typically checkpoint progress externally?
5. Where does YARN launch the ApplicationMaster for a newly submitted job?
Was this page helpful?
You May Also Like
YARN Architecture
An overview of YARN's split-brain design that separates cluster resource management from per-application scheduling and monitoring.
ResourceManager and NodeManager
A deep dive into YARN's two master/worker daemons -- the ResourceManager's scheduling and the NodeManager's per-node container enforcement.
Running Jobs on YARN
The end-to-end path of submitting, monitoring, and debugging a job on a YARN-managed Hadoop cluster.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics