Prometheus and Grafana Interview Questions: Complete Guide

In the world of DevOps, monitoring is the eyes and ears of your infrastructure. Prometheus and Grafana have become the de facto standard for open-source monitoring and visualization. Whether you're preparing for a senior DevOps role or a platform engineering position, expect deep questions on how these tools work under the hood. This guide covers real interview questions, from basic concepts to advanced production scenarios. You'll learn about Prometheus architecture, metric types, PromQL queries, alerting rules, and how to debug common issues. We'll also explore a real-world production incident where a misconfigured Prometheus caused a major outage. By the end, you'll be equipped to answer confidently and demonstrate hands-on experience. Let's dive into the world of time-series monitoring and turn you into a Prometheus and Grafana expert.

Prometheus Architecture and Core Concepts

Prometheus follows a pull-based model: it scrapes metrics from HTTP endpoints (targets) at regular intervals. The core components include the Prometheus server (which scrapes and stores data), Alertmanager (for handling alerts), and various exporters (like node_exporter for system metrics). Metrics are stored in a time-series database (TSDB) with labels (key-value pairs). Understanding the architecture is crucial for interview questions about scalability, storage, and reliability. For example, you might be asked: 'How does Prometheus handle high availability?' The answer involves running multiple instances with identical scrape configs and using a load balancer in front of Grafana. Another common question: 'What is the difference between push and pull models?' Prometheus pulls, which makes it easier to detect down targets, but requires network access to all targets. Push models (like Graphite) are better for short-lived jobs. In interviews, emphasize the trade-offs and when to use each.

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
  - job_name: 'api'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: my-api
        action: keep

Metric Types: Counter, Gauge, Histogram, Summary

Prometheus supports four core metric types. Counters are cumulative (only increase, reset on restart) – used for request counts. Gauges can go up and down – used for memory usage. Histograms sample observations and count them in configurable buckets – used for request durations. Summaries also sample observations but calculate quantiles on the client side. Interviewers often ask: 'When would you use a histogram vs a summary?' The answer: Histograms allow aggregation across instances (since buckets are additive), while summaries are better when you need precise quantiles but cannot aggregate. Another question: 'How do you reset a counter?' You can't; counters are monotonic. Use the rate() function to get per-second increase. Example: rate(http_requests_total[5m]) gives requests per second over the last 5 minutes. Understanding these types is fundamental to writing correct PromQL queries.

// Counter example
httpRequestsTotal := prometheus.NewCounter(prometheus.CounterOpts{
    Name: "http_requests_total",
    Help: "Total number of HTTP requests.",
})

// Gauge example
memoryUsage := prometheus.NewGauge(prometheus.GaugeOpts{
    Name: "memory_usage_bytes",
    Help: "Current memory usage in bytes.",
})

// Histogram example
requestDuration := prometheus.NewHistogram(prometheus.HistogramOpts{
    Name:    "request_duration_seconds",
    Help:    "Histogram of request durations.",
    Buckets: prometheus.DefBuckets,
})

PromQL: Querying and Aggregation

PromQL is the query language for Prometheus. Interview questions often involve writing queries to compute rates, averages, percentiles, and more. For example: 'Write a query to get the average CPU usage per node over the last hour.' Answer: avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[1h])). Another common question: 'How do you get the top 5 memory-consuming processes?' Use topk(5, process_resident_memory_bytes). Understanding vector matching is critical: binary operators (like /) require matching label sets. You can use on() or ignoring() to control matching. Also know about recording rules: they precompute expensive queries and store them as new time series. Example: record: job:http_requests:rate5m with expr: rate(http_requests_total[5m]). In interviews, demonstrate that you can write efficient queries and understand the difference between instant vectors and range vectors.

# Rate of HTTP requests per second
rate(http_requests_total[5m])

# 95th percentile of request duration
histogram_quantile(0.95, sum(rate(request_duration_seconds_bucket[5m])) by (le))

# Average memory usage by service
avg by (service) (process_resident_memory_bytes)

# Top 3 CPU-consuming containers
topk(3, rate(container_cpu_usage_seconds_total[5m]))

# Alert if error rate > 5%
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05

Alerting Rules and Alertmanager

Prometheus alerting rules define conditions that trigger alerts. They are evaluated periodically. When an alert fires, Prometheus sends it to Alertmanager, which handles deduplication, grouping, routing, and silencing. Interview questions: 'How do you set up an alert for high CPU usage?' Rule: alert: HighCPUUsage expr: rate(node_cpu_seconds_total{mode="idle"}[5m]) < 0.2 for: 5m labels: severity: critical annotations: summary: "CPU usage > 80% on {{ $labels.instance }}". Another question: 'How does Alertmanager group alerts?' Grouping uses labels; you can group by severity or service to reduce notification noise. Also know about inhibition rules (suppress lower-severity alerts when higher-severity fires) and silences (mute alerts for a specific time). In production, alert fatigue is a common problem; discuss how to tune thresholds, use for: duration, and implement proper routing.

groups:
  - name: example
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.instance }}"
          description: "Error rate is {{ $value | humanizePercentage }}"

Grafana: Dashboards and Data Sources

Grafana is the visualization layer that connects to Prometheus and other data sources. Interview questions: 'How do you create a dashboard that shows CPU usage per host?' Add a panel with PromQL query: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100). Use variables (e.g., $instance) to make dashboards dynamic. Another common question: 'How do you handle time zone differences?' Grafana displays in browser time zone by default; you can override in dashboard settings. Also know about annotations (mark events on graphs), alerting in Grafana (though Prometheus alerts are preferred), and provisioning (dashboards as code via JSON or YAML). In interviews, emphasize your ability to create meaningful dashboards that reduce mean time to detection (MTTD).

{
  "title": "Node Exporter Dashboard",
  "panels": [
    {
      "title": "CPU Usage",
      "type": "graph",
      "targets": [
        {
          "expr": "100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
          "legendFormat": "{{ instance }}"
        }
      ]
    }
  ]
}

Service Discovery and Relabeling

In dynamic environments like Kubernetes, Prometheus uses service discovery to find targets. Relabeling allows you to modify labels before ingestion. Interview questions: 'How does Prometheus discover pods in Kubernetes?' It uses the Kubernetes API to list pods and endpoints. Relabeling can filter targets (e.g., only pods with a certain label) or rewrite metric labels. Example: relabel_configs to keep only pods with app=my-api. Another question: 'What is the difference between relabeling and metric relabeling?' Target relabeling happens before scrape, affecting target labels; metric relabeling happens after scrape, affecting metric labels. Understanding relabeling is essential for managing label cardinality and ensuring correct metric naming. In interviews, show that you can configure service discovery for different environments (static, file-based, cloud).

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: my-app
        action: keep
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node
        action: replace

Storage, Retention, and High Availability

Prometheus's TSDB stores data in blocks (2-hour chunks) with configurable retention. Interview questions: 'How do you handle long-term storage?' Use remote write to send data to long-term storage like Thanos or Cortex. 'How do you achieve high availability?' Run multiple Prometheus servers scraping the same targets, and use a load balancer in front of Grafana. However, this duplicates storage. For deduplication, use Thanos or Cortex. Another question: 'What is the default retention period?' 15 days. You can change it via --storage.tsdb.retention.time. Also know about compaction and downsampling (not natively supported; use Thanos for downsampling). In interviews, discuss trade-offs between cost and reliability, and mention that Prometheus is not designed for multi-datacenter setups without additional tools.

--storage.tsdb.retention.time=30d
--storage.tsdb.retention.size=50GB
--storage.tsdb.path=/data/prometheus
--web.enable-admin-api