Designing Multi-Stage Pipelines
A multi-stage pipeline breaks the journey from source code to running software into a sequence of discrete phases, each with a single responsibility. Instead of one monolithic script that compiles, tests, scans, and deploys in an undifferentiated blob, a well-designed pipeline separates these concerns into stages such as build, unit-test, integration-test, package, and deploy. This separation matters for two reasons: it gives engineers a fast, legible signal about where a failure occurred, and it lets teams apply different policies — approval gates, parallelism, retry behavior — to different phases of the software delivery lifecycle. The goal is not simply to have many stages, but to have stages that reflect real boundaries in risk, cost, and ownership.
Cricket analogy: A well-run franchise doesn't lump scouting, fitness testing, and match selection into one chaotic afternoon — it separates them into distinct stages so if a player fails the fitness test, everyone knows immediately without waiting through the full trial match, and each stage gets its own selectors and rules.
Stage Boundaries and the Fail-Fast Principle
Good pipeline design orders stages from cheapest-and-fastest to most-expensive-and-slowest. Linting and static analysis run in seconds and catch a large class of trivial errors, so they belong first. Unit tests come next because they are fast and isolate logic errors without external dependencies. Integration tests, which spin up databases or call other services, are slower and belong later. Deployment to a staging or production environment is the most expensive and highest-risk operation, so it sits at the very end, gated behind everything else passing. This ordering is called fail-fast: a developer who broke a simple lint rule should learn that in 15 seconds, not after waiting 20 minutes for a full deployment to fail.
Cricket analogy: A good net session starts with a quick equipment check (seconds), then shadow batting (minutes), then facing real bowling (longer), and only at the very end does a player face a full simulated match innings — so a loose grip is caught in the equipment check, not after batting a full session.
Passing Artifacts Between Stages
Stages must communicate through well-defined artifacts rather than by re-running work. If the build stage compiles a binary or produces a container image, that exact artifact — not a rebuilt copy — should be the one tested, scanned, and eventually deployed. Rebuilding in a later stage risks a different result due to a dependency update or a nondeterministic build step, undermining the guarantee that what you tested is what you ship. Most CI/CD platforms support this via artifact upload/download mechanisms or, for containers, by pushing an immutable image tag once and referencing that same tag in every downstream stage.
Cricket analogy: When a bat is inspected and approved by the umpire before the innings, that exact bat — not a similar one pulled from the kit bag later — must be the one used at the crease, because swapping it for "the same model" risks a subtly different weight or grain that was never actually checked.
name: multi-stage-pipeline
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Compile application
run: make build
- name: Upload build artifact
uses: actions/upload-artifact@v4
with:
name: app-binary
path: dist/app
unit-test:
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Download build artifact
uses: actions/download-artifact@v4
with:
name: app-binary
- run: make test-unit
integration-test:
needs: unit-test
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: testpass
steps:
- uses: actions/checkout@v4
- run: make test-integration
deploy-staging:
needs: integration-test
runs-on: ubuntu-latest
environment: staging
steps:
- uses: actions/download-artifact@v4
with:
name: app-binary
- run: ./deploy.sh stagingThink of a multi-stage pipeline like a hospital triage system: quick, cheap checks (vitals, temperature) happen first to rule out the obvious, and only patients who pass those move on to expensive, specialized diagnostics. Nobody gets an MRI before someone confirms they have a pulse.
A common mistake is making stages too granular — ten tiny stages each with its own runner startup overhead can make a pipeline slower overall than three well-scoped stages, because provisioning a fresh runner or container often costs more time than the work it performs.
Gating Stages with Approvals and Environments
Later stages, especially those touching production, often need a manual approval gate or an environment-level protection rule rather than running automatically on every green build. Modeling deploy-to-staging and deploy-to-production as distinct stages — rather than one 'deploy' stage with an if-condition — makes it possible to attach different reviewers, different secrets, and different rollout strategies to each, and gives a clean audit trail of who approved what.
Cricket analogy: A domestic call-up to the national squad shouldn't be an automatic promotion the moment someone scores runs — it needs a distinct selection-committee sign-off stage separate from the domestic-league stage, so different selectors, different scrutiny, and a clear paper trail of who approved the call-up exist.
- Order stages from fastest/cheapest (lint, unit tests) to slowest/riskiest (deploy) to fail fast.
- Pass the exact built artifact between stages instead of rebuilding it, to guarantee what's tested is what ships.
- Use 'needs'/'depends_on' style declarations to make stage dependencies explicit rather than implicit ordering.
- Separate staging and production deploys into distinct stages so each can have its own approval gate and environment.
- Avoid over-fragmenting stages — runner startup overhead can outweigh the benefit of narrow stage scope.
- Integration tests that need external services (databases, queues) belong after unit tests, not before.
Practice what you learned
1. Why should linting and unit tests run before integration tests in a multi-stage pipeline?
2. What is the main risk of rebuilding an artifact in a later pipeline stage instead of reusing the one produced earlier?
3. In the example GitHub Actions workflow, what does 'needs: unit-test' on the integration-test job accomplish?
4. Why might splitting a pipeline into many very small stages backfire?
5. What is a good reason to model deploy-to-staging and deploy-to-production as separate stages rather than one conditional stage?
Was this page helpful?
You May Also Like
Anatomy of a CI/CD Pipeline
A CI/CD pipeline is a series of automated stages — source, build, test, package, deploy — that a code change flows through, each acting as a quality gate before the next.
Parallelizing Pipeline Jobs
Explore techniques for running pipeline jobs concurrently — matrix builds, test splitting, and fan-out/fan-in patterns — to cut wall-clock time without sacrificing reliability.
Caching and Build Artifacts
Learn how CI/CD pipelines speed up builds with dependency caching and pass data between jobs using artifacts, and where each technique fits.
Rollback and Incident Response in CI/CD
Learn how to design pipelines that can quickly and safely revert a bad deployment, and how rollback strategy fits into a broader incident response process.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics