Prometheus & Monitoring Interview Questions

1. What is Prometheus and what problems does it solve in monitoring?

2. What is the difference between monitoring and observability?

3. How does Prometheus's pull-based metrics model work?

4. What is the difference between push and pull metric collection?

5. What are the four core Prometheus metric types (counter, gauge, histogram, summary)?

6. What is the difference between a counter and a gauge in Prometheus?

7. What is the difference between a histogram and a summary in Prometheus?

8. What is PromQL and how do you write basic queries?

9. What is the rate() function in PromQL and when do you use it?

10. What are labels in Prometheus and why are they powerful?

11. What is cardinality in Prometheus and why is high cardinality a problem?

12. What are exporters in Prometheus and how do node_exporter and others work?

13. How does service discovery work in Prometheus?

14. What is the Prometheus scrape configuration and how do targets work?

15. How does Prometheus store time-series data (the TSDB)?

16. What is Alertmanager and how does alerting work in Prometheus?

17. How do you write alerting rules and recording rules in Prometheus?

18. What is the difference between an alerting rule and a recording rule?

19. How does Prometheus federation work for scaling?

20. What is remote write and remote read in Prometheus?

21. What is Grafana and how does it integrate with Prometheus?

22. What is the difference between Prometheus and Grafana?

23. What are the RED and USE methods for monitoring?

24. What are SLIs, SLOs, and SLAs and how do they relate to monitoring?

25. What is an error budget and how does it drive reliability decisions?

26. How do you monitor Kubernetes with Prometheus?

27. What is the Prometheus Operator and how does it simplify deployment?

28. How do you handle long-term storage for Prometheus metrics (Thanos, Cortex, Mimir)?

29. How do you reduce noise and alert fatigue in a monitoring system?

30. What are common Prometheus monitoring best practices and pitfalls?

31. How does Prometheus staleness handling work, and why can a series disappear from a query result?

32. Why can increase() return a fractional value, and how do rate() extrapolation and counter resets affect accuracy?

33. A target is showing as DOWN in Prometheus — how do you debug it end to end?

34. How do you run Prometheus and Alertmanager in high availability without getting duplicate pages?

35. Prometheus is using far more memory than expected and restarts are slow — how do you diagnose and fix it?

36. A PromQL query is timing out or returning 'query processing would load too many samples' — how do you make it cheap?

37. How do you alert on the absence of data, and why do you need a dead man's switch?

38. What are exemplars in Prometheus, and how does Prometheus interoperate with OpenTelemetry metrics?

39. How do you secure a Prometheus deployment — scrape credentials, the web UI, and the admin API?

40. How do you unit test Prometheus alerting and recording rules before they reach production?

41. What is the difference between relabel_configs and metric_relabel_configs, and where do you drop unwanted series?

42. Why is averaging percentiles wrong, and how do you aggregate latency correctly with histogram_quantile?

43. An alert is firing in Prometheus but nobody was paged. How do you debug the Alertmanager path?

44. When is the Pushgateway the right tool, and what operational traps does it introduce?

45. What causes 'out of order sample' and 'duplicate sample for timestamp' errors, and how do honor_timestamps and out-of-order ingestion relate?

46. Prometheus will not start after a disk filled up. How do you recover the TSDB without losing more data than necessary?

47. An alert keeps firing and resolving every few minutes. How do you stabilise it without hiding a real problem?

48. A single Prometheus can no longer scrape every target. How do you shard the scrape load?

49. How does PromQL vector matching work, and how do you enrich a metric with labels from an info metric using group_left?

50. How do you monitor something that cannot expose /metrics — an external endpoint, a certificate, or a shell script's output?

51. How do you compare current behaviour with last week in PromQL, and what do offset and the @ modifier actually do?

52. Recording rules are producing gaps and alerts are late. How do you diagnose rule evaluation health and ordering?

53. What are the Prometheus metric naming conventions, and why do base units matter?

54. What is the OpenMetrics exposition format, and how does Prometheus decide which format to parse?

55. How do you choose a scrape interval, and how must the rate() window relate to it?

56. What is the query step in a range query, and how does lookback delta change what you see?

57. How do you choose histogram buckets, and what do native histograms change?

58. Which scrape limits protect a Prometheus server, and what happens when one is exceeded?

59. Why does a rolling deployment spike Prometheus memory, and how do you manage series churn?

60. How do you decide what to precompute as a recording rule, and how should you name it?

61. How do topk, bottomk and the _over_time functions differ, and when does topk mislead you?