Common CI/CD Pitfalls
Most CI/CD failures are not exotic; they are the same handful of mistakes recurring across teams and tooling. Recognizing these patterns early — in code review, in pipeline design, in postmortems — saves far more time than debugging them after they've caused an outage or eroded trust in the pipeline. This topic catalogs the pitfalls that show up again and again, along with the concrete fix for each, so they can be caught by habit rather than rediscovered the hard way.
Cricket analogy: Most cricket collapses aren't caused by unplayable deliveries but by the same recurring mistakes -- playing across the line, chasing a wide one -- and coaches who drill against these familiar patterns in the nets save a team from repeating the same dismissal in a final that costs the match.
Flaky Tests and the 'Just Re-run It' Trap
A flaky test is one that passes and fails intermittently with no code change — often due to timing assumptions, shared test state, or reliance on network calls that occasionally time out. The dangerous pattern is normalizing re-runs: once engineers learn that failure X 'just needs a re-run,' they stop reading failure messages carefully, and a re-run habit that started with one flaky test spreads until real failures get re-run and ignored too. The fix is to quarantine known-flaky tests into a separate, non-blocking suite while they're fixed, rather than leaving them in the blocking path where they train engineers to distrust the pipeline.
Cricket analogy: A flaky test is like a fielder who drops an easy catch only occasionally with no clear cause -- if the captain starts saying 'just give him another chance' every time, soon every dropped catch gets waved off, so smart teams instead bench that fielder to a specialist practice group until the issue is fixed, keeping them out of the main XI.
Hardcoded Secrets and Configuration
Committing an API key, database password, or cloud credential directly into a pipeline YAML file or source code is one of the most common and most damaging pitfalls, because git history retains the secret forever even after it's 'removed' in a later commit. Secrets belong in a dedicated secrets manager or the CI platform's encrypted secret store, referenced by name at runtime, never typed literally into a config file. Equally important: pipeline logs themselves can leak secrets if a command echoes an environment variable or a script prints a full curl command including an Authorization header — most CI platforms automatically mask known secret values in logs, but only if the secret was injected through their secret mechanism rather than pasted as a plain string.
Cricket analogy: Committing an API key to a YAML file is like a team publishing their signal codes in a printed matchday program -- even after they print a corrected program next match, the old one is still in every fan's hands forever, so codes belong in a private dressing-room system, not typed onto a public sheet.
# BAD: secret hardcoded directly in the workflow
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- run: curl -H "Authorization: Bearer sk_live_51Hx..." https://api.example.com/deploy
# GOOD: secret referenced from the platform's encrypted store
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- run: curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/deploy
env:
API_TOKEN: ${{ secrets.DEPLOY_API_TOKEN }}A useful mental test for pipeline design: 'If this pipeline's log output were posted publicly tomorrow, would anything sensitive leak?' Running through this thought experiment regularly catches secret-masking gaps before an actual leak does.
Overly broad service-account permissions are a pitfall that rarely causes a visible failure — the pipeline runs fine for months — until the one time a compromised dependency or a malicious pull request abuses those permissions to do far more damage than a narrowly scoped credential would have allowed. Least-privilege scoping should be a default, not an afterthought triggered by an incident.
Pipeline Configuration Drift and Duplication
Copy-pasting pipeline YAML across a dozen repositories is a pitfall that seems harmless at first and becomes expensive later: a security fix, a new required check, or a build-tool version bump has to be manually propagated to every copy, and inevitably some repos are missed. Reusable workflows, composite actions, or shared pipeline templates centralize this logic so a fix applies everywhere at once. Similarly, letting a pipeline's actual behavior drift silently from its documentation — someone adds a manual step 'just this once' that never gets written down — erodes the pipeline's trustworthiness as a source of truth for the release process.
Cricket analogy: Copy-pasting the same fielding drill instructions to every age-group team's separate coaching manual seems harmless until a rule change means updating a dozen manuals individually and some get missed, so associations now maintain one master drill template that every team pulls from, updated once for everyone.
- Flaky tests left in the blocking path teach engineers to distrust and reflexively re-run the pipeline — quarantine them instead.
- Never hardcode secrets in pipeline YAML or source; use the platform's encrypted secret store, referenced by name.
- Pipeline logs can leak secrets if commands echo them directly rather than through the platform's masking mechanism.
- Overly broad service-account permissions are a silent risk that only surfaces when something is compromised.
- Copy-pasted pipeline configuration across repos causes drift; use reusable workflows or shared templates instead.
- Undocumented manual steps in a 'passing' pipeline erode its value as the actual source of truth for the release process.
Practice what you learned
1. Why is normalizing 're-running a flaky test until it passes' a dangerous pattern?
2. Why is committing a secret directly into a pipeline YAML file dangerous even if it's removed in a later commit?
3. What is the correct way to use a secret in a pipeline, based on the code example?
4. Why are overly broad service-account permissions considered a particularly insidious pitfall?
5. What problem do reusable workflows or shared pipeline templates solve?
Was this page helpful?
You May Also Like
Managing Secrets in CI/CD
CI/CD pipelines need credentials for external systems, so secrets must be stored encrypted, injected at runtime, scoped narrowly, and never committed to source control or build logs.
Pipeline Security Basics
CI/CD pipelines are a high-value attack surface with broad access to source, secrets, and production, so securing them requires trusted inputs, isolated execution, and verified artifacts.
Running Automated Tests in CI/CD
How CI pipelines execute unit, integration, and end-to-end tests automatically, and the design choices — parallelism, flakiness handling, test selection — that keep feedback fast.
Least Privilege for CI/CD Service Accounts
CI/CD service accounts should hold only the exact permissions their pipeline needs, scoped narrowly and time-limited, so a compromised job can't cascade into a broader breach.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics