Datadog Cheat Sheet
Reference for Datadog Agent configuration, tagging conventions, monitor definitions, and query syntax for metrics and logs.
Agent Configuration
datadog.yaml key settings for the Datadog Agent.
api_key: <YOUR_API_KEY>site: datadoghq.comtags: - env:production - team:platformlogs_enabled: trueapm_config: enabled: trueprocess_config: process_collection: enabled: true
Custom Metrics (DogStatsD)
Sending custom metrics from application code via DogStatsD.
from datadog import statsdstatsd.increment('orders.processed', tags=['env:prod'])statsd.gauge('queue.depth', 42, tags=['queue:orders'])statsd.histogram('request.duration', 0.235, tags=['endpoint:/checkout'])statsd.event('Deploy finished', 'Deployed version 1.4.2 to prod')
Monitor Definition
A metric-based monitor alerting on high CPU usage.
{ "name": "High CPU usage", "type": "metric alert", "query": "avg(last_5m):avg:system.cpu.user{env:production} > 85", "message": "CPU usage is high on {{host.name}} @slack-platform-alerts", "options": { "thresholds": { "critical": 85, "warning": 70 }, "notify_no_data": true }}
Key Concepts
Core Datadog terminology.
- tags- key:value labels applied to hosts/metrics enabling flexible filtering and grouping
- DogStatsD- Local UDP protocol/daemon used to submit custom metrics from app code
- monitor- A configured alert condition on metrics, logs, or APM data
- dashboard- Visual collection of widgets/graphs built from queries
- APM trace- End-to-end request trace spanning services, used for latency/error analysis
Datadog as Code (Terraform)
Defining a metric monitor and a timeseries dashboard widget via the Datadog Terraform provider.
resource "datadog_monitor" "high_latency" { name = "High checkout latency" type = "metric alert" message = "@slack-platform-alerts p95 latency above threshold" query = "avg(last_5m):p95:trace.http.request.duration{service:checkout} > 1" monitor_thresholds { critical = 1 warning = 0.75 } tags = ["team:platform", "service:checkout"]}resource "datadog_dashboard" "platform" { title = "Platform Overview" layout_type = "ordered" widget { timeseries_definition { request { q = "avg:system.cpu.user{env:production} by {host}" } } }}
Log Pipeline Grok Processor & Log-Based Metric
A grok processor parsing access logs, plus a log-based metric that counts checkout errors grouped by status code.
{ "type": "grok-parser", "name": "Parse checkout access logs", "source": "message", "grok": { "matchRules": "access_log %{date(\"dd/MMM/yyyy:HH:mm:ss Z\"):timestamp} %{word:method} %{notSpace:path} %{number:status}" }}{ "name": "checkout-error-rate", "type": "log_based", "query": { "index": "main", "compute": { "aggregation": "count" }, "search": { "query": "service:checkout status:error" }, "group_by": [{ "facet": "@http.status_code" }] }}
Unified Service Tagging (Trace/Log Correlation)
Environment variables that bind APM traces, logs, and metrics to the same service/env/version for cross-product correlation.
export DD_ENV=productionexport DD_SERVICE=checkoutexport DD_VERSION=1.4.2export DD_TAGS="team:platform,tier:critical"export DD_TRACE_SAMPLE_RATE=1export DD_LOGS_INJECTION=true# ddtrace auto-injects dd.trace_id/dd.span_id into log records# when DD_LOGS_INJECTION=true, enabling one-click trace-to-log pivots.
Advanced Concepts
Terminology beyond basic monitors and dashboards.
- SLO- Service Level Objective tracking an error-budget burn rate against a target (e.g. 99.9% good events) over a rolling window
- composite monitor- A monitor combining other monitors with boolean logic (e.g. 1 && 2) to alert only when multiple conditions co-occur
- Watchdog- Datadog's automatic anomaly detection that surfaces unusual patterns without a manually authored monitor
- RUM- Real User Monitoring — browser/mobile session and performance data correlated with backend traces
- service map- Auto-generated topology of service dependencies derived from APM trace data
- forwarder- The Datadog Lambda function that ships CloudWatch logs/metrics/traces from AWS into Datadog
- downtime- A scheduled suppression window that silences monitor notifications for a scope (e.g. env:staging) without disabling the monitor
API: Composite Monitor & Downtime
Creating a composite monitor from two existing monitor IDs and scheduling a downtime via curl against the v1 API.
curl -X POST "https://api.datadoghq.com/api/v1/monitor" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -H "Content-Type: application/json" \ -d '{ "name": "Composite: checkout unhealthy", "type": "composite", "query": "1 && 2", "message": "Checkout is both slow and erroring @pagerduty-platform", "options": { "notify_no_data": false } }'curl -X POST "https://api.datadoghq.com/api/v1/downtime" \ -H "DD-API-KEY: ${DD_API_KEY}" -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -H "Content-Type: application/json" \ -d '{ "scope": ["env:staging"], "start": 1700000000, "end": 1700003600 }'
Standardize on a small, consistent tag taxonomy (env, service, team) applied via the Agent's global tags rather than ad-hoc per-metric tags — inconsistent tagging is the most common cause of unusable dashboards at scale.