Running Automated Tests in CI/CD
Automated testing inside a pipeline is what turns 'the build compiled' into 'the change is actually safe to ship.' A typical pipeline runs several layers of tests with different scopes and speeds: fast unit tests that verify individual functions in isolation, integration tests that check how components interact with real or realistic dependencies (a database, a message queue), and slower end-to-end tests that drive the application the way a user would. Structuring the pipeline to run cheap, fast tests first and expensive, slow tests later — and failing fast when an early layer breaks — keeps feedback loops short without sacrificing confidence.
Cricket analogy: Like a batsman's readiness check going beyond just holding the bat — first shadow practice alone (unit), then facing throwdowns from a coach with a real ball (integration), and finally a full net session against genuine bowlers in match conditions (end-to-end), run cheap-to-expensive so a flaw is caught before the costly full net session.
The Test Pyramid in a Pipeline
The test pyramid principle — many unit tests, fewer integration tests, and a small number of end-to-end tests — maps directly onto pipeline design. Unit tests should run on every push within seconds to a couple minutes; integration tests, which often need spun-up services, typically run in a dedicated job with service containers; end-to-end tests, which may need a full deployed environment and browser automation, are the slowest and often reserved for merge-to-main or pre-deploy gates rather than every single commit.
Cricket analogy: Like a domestic system with many junior league matches, fewer first-class matches, and only a handful of international Tests — junior matches happen constantly every weekend, first-class games need a proper venue and staff, and Tests, needing a full squad and international travel, are reserved for the biggest occasions only.
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: '20' }
- run: npm ci
- run: npm run test:unit -- --ci --reporters=default --reporters=jest-junit
- uses: actions/upload-artifact@v4
with:
name: unit-test-results
path: junit.xml
integration-tests:
runs-on: ubuntu-latest
needs: unit-tests
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: test
ports: ['5432:5432']
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run test:integration
env:
DATABASE_URL: postgres://postgres:test@localhost:5432/postgres
Parallelism and Test Splitting
As a test suite grows, running it on a single machine becomes the pipeline's slowest step. Most CI systems support splitting a suite across multiple parallel jobs (a matrix, or 'test splitting'/'sharding') using either fixed partitioning or, more effectively, timing-based splitting that balances historical execution time across shards so no single shard becomes the long pole. Frameworks like pytest, Jest, and Cypress have built-in or plugin-based support for reporting per-test timing that CI orchestrators use to balance shards intelligently.
Cricket analogy: Like a huge net-practice roster too much for one set of nets to handle in a day, so a coach splits bowlers across four net lanes at once — either dividing them evenly by headcount, or, better, using each bowler's actual historical overs-per-hour to balance lanes so no single lane runs long after the others finish.
Handling Flaky Tests
A flaky test — one that passes and fails intermittently without a code change — erodes trust in CI faster than almost anything else, because engineers start reflexively re-running red pipelines instead of investigating. Mature pipelines track flakiness explicitly (often via automatic retry-and-flag mechanisms or dedicated flaky-test dashboards) rather than silently retrying failures forever, and treat a test crossing a flakiness threshold as a bug to be fixed or quarantined, not ignored.
Cricket analogy: Like a bowler who no-balls unpredictably regardless of his run-up — umpires start reflexively re-checking every delivery instead of trusting the call, and a well-run academy tracks his no-ball rate on a dashboard, treating a rate crossing a threshold as a technique flaw to fix or bench him from selection, not just wave through.
A useful mental model: unit tests answer 'did I break this function,' integration tests answer 'do these pieces work together,' and end-to-end tests answer 'does the whole system work for a user' — each answers a different question at a different cost, which is why pipelines should not treat all three as interchangeable or run them all on every single commit.
Blindly retrying failed tests until they pass hides real regressions and lets flakiness compound over time — a retry mechanism should always be paired with visibility (logging, dashboards, alerts) into how often and which tests are actually flaking.
- CI pipelines typically run unit, integration, and end-to-end tests as distinct layers with increasing cost and scope.
- The test pyramid (many unit, fewer integration, few end-to-end) should shape which tests run on every commit versus only pre-merge/pre-deploy.
- Service containers let integration tests run against real dependencies like databases inside the CI job itself.
- Test splitting/sharding, ideally timing-based, is the main lever for reducing wall-clock time on large suites.
- Flaky tests should be tracked and fixed or quarantined, not silently retried into invisibility.
- Fast, cheap tests should run first so pipelines fail fast and give developers quick feedback.
Practice what you learned
1. According to the test pyramid principle, how should the number of tests generally be distributed across layers?
2. Why are service containers (e.g. a Postgres container) used in an integration-test CI job?
3. What is the main advantage of timing-based test splitting over fixed/naive partitioning?
4. What is the recommended way to handle a flaky test long-term?
5. Why should fast, cheap tests generally run before slow, expensive tests in a pipeline?
Was this page helpful?
You May Also Like
Automating Builds in Pipelines
How CI systems turn source code into reproducible build artifacts automatically, covering build triggers, dependency resolution, and artifact publishing.
Test Reporting and Quality Gates
How pipelines surface test and coverage results in a structured, actionable way, and how quality gates use those results to block risky changes automatically.
Parallelizing Pipeline Jobs
Explore techniques for running pipeline jobs concurrently — matrix builds, test splitting, and fan-out/fan-in patterns — to cut wall-clock time without sacrificing reliability.
Anatomy of a CI/CD Pipeline
A CI/CD pipeline is a series of automated stages — source, build, test, package, deploy — that a code change flows through, each acting as a quality gate before the next.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics