Buildkite Cheat Sheet
Buildkite CI/CD pipelines covering pipeline.yml syntax, agent hooks, plugins, and self-hosted agent management.
pipeline.yml Basics
A typical Buildkite pipeline with parallel steps and a wait.
steps: - label: ":hammer: Build" command: "make build" key: build - wait - label: ":test_tube: Test %n" command: "make test" parallelism: 4 agents: queue: "default" plugins: - docker#v5.11.0: image: "node:20" - block: ":rocket: Deploy to prod?" branches: "main" - label: ":shipit: Deploy" command: "./scripts/deploy.sh" depends_on: build if: build.branch == "main"
buildkite-agent CLI
Commands run inside a pipeline step, often in agent hooks.
# Upload dynamically generated pipeline stepsbuildkite-agent pipeline upload .buildkite/deploy-pipeline.yml# Set a build-wide metadata key (readable by later steps)buildkite-agent meta-data set "release-version" "1.4.2"buildkite-agent meta-data get "release-version"# Upload/download artifacts between stepsbuildkite-agent artifact upload "dist/**/*"buildkite-agent artifact download "dist/**/*" .# Annotate the build UI with markdownbuildkite-agent annotate "Coverage: 87%" --style "info"
Agent Hook Example
A pre-command hook on a self-hosted agent for shared setup logic.
#!/bin/bash# .buildkite/hooks/pre-commandset -euo pipefailecho "--- :docker: Logging into registry"echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USER" --password-stdinexport NODE_ENV=test
Core Concepts
Terminology you'll see throughout Buildkite docs and the UI.
- Pipeline- a named collection of steps, defined in YAML, tied to a repo
- Step (command/wait/block/trigger/group)- the unit of work; block pauses for manual approval, trigger fires another pipeline
- Agent- a process you run on your own infra that polls for and executes jobs
- Queue- a tag used to route jobs to specific agent pools, e.g. `queue=mac` or `queue=gpu`
- Plugin- a reusable YAML+hook bundle (e.g. docker#v5, ecr#v2) referenced by version tag
- Dynamic pipelines- steps generated at runtime via `buildkite-agent pipeline upload`
Matrix Builds
Fan a single step out across combinations of dimensions without hand-writing each variant.
steps: - label: "Test {{matrix.os}} / node{{matrix.node}}" command: "./scripts/test.sh" matrix: setup: os: ["linux", "macos"] node: ["18", "20", "22"] adjustments: - with: { os: "macos", node: "18" } skip: true agents: queue: "{{matrix.os}}"
trigger & group Steps
trigger fires a downstream pipeline; group visually nests related steps and gates on their combined result.
steps: - group: ":package: Build & Scan" steps: - label: build command: make build - label: scan command: trivy image myapp:$BUILDKITE_COMMIT - trigger: "deploy-pipeline" label: ":rocket: Trigger Deploy" branches: "main" build: message: "Deploy ${BUILDKITE_MESSAGE}" commit: "${BUILDKITE_COMMIT}" env: SOURCE_BUILD: "${BUILDKITE_BUILD_NUMBER}"
Automatic Retry & soft_fail
Handle flaky infra without failing the whole pipeline or masking real failures.
steps: - label: ":test_tube: Flaky Integration Tests" command: "make integration-test" retry: automatic: - exit_status: -1 # agent lost / infra failure limit: 2 - exit_status: 2 limit: 1 soft_fail: - exit_status: 3 # known-flaky exit code, don't block the build - label: ":mag: Lint (non-blocking)" command: "make lint" soft_fail: true
Generating Steps Dynamically
Common pattern: a first step computes which services changed and uploads a pipeline scoped to only those.
#!/bin/bashset -euo pipefailchanged=$(git diff --name-only origin/main...HEAD | cut -d/ -f1 | sort -u)for svc in $changed; do [ -d "services/$svc" ] || continue cat <<EOF - label: ":hammer: Build $svc" command: "make -C services/$svc build" key: build-$svcEOFdone | buildkite-agent pipeline upload
Agent Hook Lifecycle
Hooks run at fixed points around every job; know which one to use for what.
- environment- runs first, before checkout; typically exports secrets/env vars for the whole job
- pre-checkout- runs before Buildkite clones the repo, e.g. to configure git credentials
- post-checkout- runs right after checkout, good for submodule init or workspace prep
- pre-command- runs before the step's command, common place for docker login / tool setup
- post-command- runs after the command regardless of exit status, good for cleanup
- pre-exit- final hook before the job reports its result; use for teardown that must always run
Pin every plugin to an exact version tag (`docker#v5.11.0`, not `docker#v5`) in pipeline.yml — Buildkite plugins are just git repos, and an unpinned major-version reference can silently pull a breaking update into your build.
