Erlang Best Practices: Supervision, OTP, and Fault Tolerance

Analogy categoryAnalogies

Erlang Best Practices for Production Systems

Erlang was designed at Ericsson for telecom switches that needed to run for years without downtime, and its best practices flow directly from that goal: the BEAM virtual machine gives every unit of concurrency its own lightweight process with isolated memory and no shared state, so a single crash can never corrupt a neighboring process. OTP (Open Telecom Platform) codifies these practices into behaviours — gen_server, gen_statem, supervisor, application — so instead of inventing your own process lifecycle, you plug business logic into a proven skeleton for state, restart, and error handling.

🏏

Cricket analogy: Like the Chennai Super Kings building a squad with clearly defined roles — a death-overs specialist, a spin option for turning pitches — Erlang gives each concurrent task its own dedicated, isolated process rather than one do-everything routine.

Let It Crash and Supervision Tree Design

The 'let it crash' philosophy means you should not write defensive code to catch every conceivable error inside a worker process; instead, let the process terminate abnormally and rely on its supervisor to restart it into a known-good state. A supervisor is configured with a restart strategy: one_for_one restarts only the child that died, one_for_all terminates and restarts every sibling child (useful when children share tightly coupled state), and rest_for_one restarts the failed child plus every child started after it in the supervision tree's start order, leaving earlier siblings untouched. Getting the strategy right is a design decision about the dependency direction between children, not a cosmetic choice.

🏏

Cricket analogy: It's like Rohit Sharma resting an out-of-form bowler for one over instead of forfeiting the match — the supervisor swaps out only the failed component (one_for_one) while the rest of the team keeps playing.

Designing gen_server Callbacks Correctly

A gen_server serializes all messages to a process through one mailbox, so handle_call/3 must return quickly because the calling process blocks until you reply — doing a slow database query or HTTP request inside handle_call stalls every other caller waiting on that same server. The idiomatic fix is to do the expensive work asynchronously: reply immediately with {noreply, State} and send the real answer later via gen_server:reply/2 from a spawned helper, or route fire-and-forget work through handle_cast/2. handle_info/2 exists for messages that did not arrive through gen_server:call or gen_server:cast, such as 'DOWN' monitor notifications, timeouts scheduled with erlang:send_after/3, or raw messages from a linked process.

🏏

Cricket analogy: It's like a wicketkeeper standing up to the stumps for spin but standing back for pace — handle_call must react within a tight window, so you never let it do slow, unbounded work that stalls the next ball.

erlang

-module(job_worker).
-behaviour(gen_server).

-export([start_link/1, submit/2, status/1]).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2]).

start_link(Name) ->
    gen_server:start_link({local, Name}, ?MODULE, [], []).

submit(Name, Job) ->
    gen_server:cast(Name, {run, Job}).

status(Name) ->
    gen_server:call(Name, status).

init([]) ->
    process_flag(trap_exit, true),
    {ok, #{pending => 0}}.

%% Keep handle_call fast -- never block on I/O here.
handle_call(status, _From, State) ->
    {reply, {ok, State}, State}.

%% Long-running work goes through handle_cast, not handle_call.
handle_cast({run, Job}, State = #{pending := Pending}) ->
    spawn_link(fun() -> do_work(Job) end),
    {noreply, State#{pending := Pending + 1}}.

%% Out-of-band signals (linked-process exits, timers) land in handle_info.
handle_info({'EXIT', _Pid, normal}, State = #{pending := Pending}) ->
    {noreply, State#{pending := Pending - 1}};
handle_info({'EXIT', Pid, Reason}, State) ->
    error_logger:warning_msg("worker ~p died: ~p~n", [Pid, Reason]),
    {noreply, State}.

do_work(Job) ->
    %% Expensive work happens here, off the gen_server's own mailbox.
    {ok, _Result} = perform(Job),
    ok.

perform(Job) -> {ok, Job}.

Process Registries and ETS for Shared State

Erlang's process dictionary (put/2, get/1) stores state that is local to a single process and invisible to introspection tools and hot code reloads, so stashing anything beyond throwaway scratch values there tends to hide bugs and breaks the OTP convention that a process's real state lives in its gen_server State argument. For state that many processes need to read concurrently — a configuration cache, a session table, a rate-limit counter — ETS (Erlang Term Storage) is the standard tool: it is a separate, mutable table living outside any single process's heap, and readers can query it directly with ets:lookup/2 without funnelling every read through one gen_server's mailbox, which would otherwise become a serialization bottleneck under load.

🏏

Cricket analogy: It's like a stadium scoreboard broadcasting the score to every fan directly, rather than each fan asking the scorer individually — ETS lets many processes read shared state concurrently instead of queuing behind one server.

ETS tables support four access modes — public, protected (the default: any process can read, only the owner process can write), and private — and four table types: set, ordered_set, bag, and duplicate_bag. A protected set owned by the gen_server that manages writes, with other processes calling ets:lookup/2 directly, is the common default for a shared read-mostly cache.

Robust, Precise Error Handling

Good Erlang error handling favors pattern matching over exception handling: functions return tagged tuples like {ok, Value} or {error, Reason}, and callers pattern-match on the shape they expect, letting an unexpected shape crash the process rather than silently continuing with bad data. try ... catch should be reserved for boundaries where you genuinely expect failure and can do something specific about it — a call into a NIF, a port program, or a third-party library with undocumented exceptions — and even then you should catch the narrowest class you can (catch error:{badmatch, _} is more honest than catch _:_), because a blanket catch-all hides bugs that should have crashed and paged someone.

🏏

Cricket analogy: It's like an umpire only overturning a decision on clear DRS evidence for that specific appeal, not reviewing every single ball — narrow, targeted error handling beats a blanket catch-all.

Writing try _ catch _:_ -> ok end around a whole function body is one of the most common Erlang anti-patterns: it silently swallows badmatch, function_clause, and case_clause errors that should have crashed the process and triggered a supervisor restart, turning a visible bug into a process that limps along in a corrupted state.

Tooling and OTP Release Hygiene

Dialyzer performs success typing against -spec annotations on exported functions, catching type mismatches such as passing an integer where a function expects a tuple, without requiring the exhaustive static type system Erlang was deliberately built without; running Dialyzer in CI on every merge catches classes of bugs unit tests miss. observer gives a live view into the running system — the supervision tree, per-process message queue length and memory, ETS table contents — invaluable for diagnosing a production node without stopping it. For deployment, rebar3's release tooling (built on relx) assembles your application and its dependencies into a self-contained OTP release with a boot script rather than shipping loose .beam files; and within that release, worker processes should always be started under a supervisor via supervisor:start_child/2 or a static child spec instead of a naked spawn/1, because an unsupervised process that crashes leaves no trace and no automatic recovery.

🏏

Cricket analogy: It's like a team using Hawk-Eye ball-tracking data and match analytics before selection instead of relying on gut feel — Dialyzer and observer give you objective evidence about your code before it reaches production.

Let It Crash: don't defensively wrap every operation — let a process die and let its supervisor restart it into a clean state.
Choose a supervision strategy deliberately: one_for_one for independent children, one_for_all or rest_for_one when children share dependent state.
Keep handle_call/3 fast; offload slow work to handle_cast/2 or a spawned helper and reply later with gen_server:reply/2.
Use handle_info/2 for monitor 'DOWN' messages, timers, and raw process messages that bypass call/cast.
Prefer ETS over the process dictionary or a single gen_server mailbox for state that many processes need to read concurrently.
Pattern-match on {ok, _} / {error, _} instead of blanket try/catch; reserve catch for true I/O or NIF boundaries and catch the narrowest class possible.
Run Dialyzer with -spec annotations in CI, use observer for live diagnostics, build releases with rebar3/relx, and always start workers under a supervisor rather than a naked spawn/1.

Practice what you learned

1. In OTP supervision, which restart strategy restarts the failed child along with every child that was started after it in the supervisor's child list?

2. Why should a gen_server avoid performing a slow HTTP request inside handle_call/3?

3. What is the main risk of storing shared application state in the Erlang process dictionary?

4. What does Dialyzer check when run against a module with -spec annotations?

5. Why is starting a worker process with a naked spawn/1 discouraged in production OTP code?

Was this page helpful?

Erlang vs Elixir

Building a Chat Server in Erlang