Grafana Panel No Data: Fix Valid Queries Fast

The query runs perfectly in Prometheus. Paste it into Grafana and the panel says No Data. You check the datasource — green, saved, healthy. You retype the query, refresh, clone the panel. Still nothing, and the wallboard meeting starts in twenty minutes.

Grafana panels add four layers between your query and the screen: time range, datasource routing, template variables, and interval stepping. Each layer can silently empty a valid query, and none of them reports an error when it does. No Data is Grafana shrugging — the query returned nothing after all four layers had their say.

This guide checks the layers in order, cheapest first. You'll widen ranges to fit scrape intervals, verify datasource UIDs with Query Inspector, unstick template variables, and set min intervals that match collection. Most empty panels fill in under ten minutes once you stop blaming the query. Bring one broken panel and its raw query text: you will fix it live, name the exact layer that lied, and lock it for good.

Time Range Too Short for Your Scrape Interval

Time range is the top empty-panel cause because defaults lie. A 5-minute default range on a 60s-scrape job holds five samples, and range functions needing two points per window plus step misalignment turn that into flickering gaps. Engineers see No Data while colleagues with 6-hour ranges see healthy lines from the identical query.

The fix is arithmetic: range at 4x the scrape interval or wider, min step at or above the interval. Sixty-second jobs get 15-minute minimum ranges with 60s steps; 15s jobs are comfortable at 30 minutes with auto steps. Wallboards deserve 6-hour defaults so slow jobs always have room to draw.

Lock it per dashboard. Document the floor in the dashboard description and set panel defaults so the next viewer can't shrink into gaps. Range bugs recur whenever someone clones a panel and narrows the view to investigate — the floor prevents the investigation from manufacturing its own mystery.

The time picker hides more traps: absolute ranges pinned during an incident keep showing the incident long after recovery, and auto-refresh intervals shorter than the scrape paint partial data as fresh. Prefer relative ranges (last 6h) for routine dashboards and reserve absolute ranges for postmortems. The $__range variable lets panels adapt their computations to the visible window instead of hardcoding lookbacks. When a wallboard disagrees with your laptop, compare picker settings first — different ranges explain most ghost disagreements. Time is a filter, and filters silently empty queries.

# Rendered correctly: range covers many scrape intervals
# Panel: Range last 6h, Min interval 60s (job scrapes every 60s)
rate(http_requests_total[5m])

# Gappy: same query, 5m range with 10s min interval on a 60s job
# Fix: widen range, raise min interval — don't touch the query

Datasource UID Pointing at Nothing

Grafana routes panels by datasource UID, an opaque string that survives renames but changes on recreation. Cloned dashboards, migrated backends, and hand-edited JSON all carry stale UIDs that point at deleted or wrong-environment datasources. The panel queries nothing and reports No Data without complaint.

Query Inspector exposes the routing in seconds. It shows the exact UID hit, the rendered request, and the raw response — a 404 or wrong-cluster payload names the culprit immediately. Compare against a working panel's UID and the mismatch is usually obvious.

Provisioning ends the class. Datasource provisioning files pin stable UIDs per environment, and dashboard JSON references those UIDs instead of whatever the UI generated. Migrations then update the provisioned backend behind a stable UID, and no panel ever orphans again.

Provisioning files make UIDs boring in the best way. Declare each datasource once with a fixed uid, and every dashboard references that string instead of whatever the UI generated. Across environments, keep the same logical names with per-environment URLs: prod-prometheus points at prod in prod and staging in staging, so one dashboard JSON deploys everywhere. Export and import round-trips preserve UIDs, but hand-merging JSON in editors often duplicates or drops them — diff UID fields after every merge. Treat UIDs like DNS: stable, boring, load-bearing.

# Find stale UID references across exported dashboards
# Export JSON, then:
# grep -o '"uid": *"[^"]*"' dashboard.json | sort | uniq -c

# datasource provisioning with a stable UID (provisioning/datasources.yml)
apiVersion: 1
datasources:
  - name: Prometheus-Prod
    uid: prod-prometheus
    type: prometheus
    url: http://prometheus:9090

Template Variables Filtering Everything Out

Template variables interpolate into queries before execution, and a stale value filters everything. After Kubernetes deploys, pod-name variables hold last week's pods; after renames, instance variables hold dead hostnames. The query is valid, the value is extinct, and the result is empty.

Prove it by hardcoding. Replace $pod with a live pod name and rerun — data means the variable lied. Do this per variable, most-stale first, and you'll find the culprit in under a minute without reading variable definitions.

Harden the variables afterward. Set refresh on dashboard load and time-range change, default to All instead of one value, and prefer stable label values like service or deployment over pod names. Variables built on volatile labels are incident generators with a refresh button.

Variable types shape failure modes. Query variables pull values from the datasource and go stale when labels churn; custom variables hold hand-written lists that rot when services rename; constant variables hide filters nobody remembers. Chained variables (region filters zone filters pod) multiply staleness: one stale parent empties every child. Prefer query variables with refresh-on-load for volatile labels, and constants only for truly fixed dimensions like environment tiers. Audit the variable list quarterly — dashboards accumulate variables like closets accumulate boxes.

Min Interval Finer Than Your Data

Min interval controls step granularity: how finely Grafana slices the range for evaluation. Steps far below the scrape interval create empty slots — a 10s step on a 60s job means five of six slots hold no sample. The line flickers between points and gaps that look like outages but are pure arithmetic.

Match the step to collection. Set min interval to the scrape interval explicitly, or leave it on auto with a sane floor. Either guarantees every evaluation slot contains a sample, and the flicker vanishes without touching queries or backends.

Beware copy-paste intervals. A 10s min interval tuned for a fast job travels with cloned panels onto slow-job dashboards and manufactures gaps there. Review min interval on every clone like you'd review the query — it's part of the query's correctness, not decoration.

Two interval variables solve different problems. $__interval divides the visible range into Grafana's max data points, adapting to zoom; $__rate_interval adds scrape-interval awareness so rate() windows never drop below four scrapes. Prefer $__rate_interval inside PromQL range selectors and plain $__interval for display steps. Instant queries ignore intervals entirely, which makes single-stat panels flicker on slow jobs — switch them to range queries with explicit steps. The interval system rewards dashboards that declare collection speed once per panel and reuse it everywhere.

# Panel JSON: min interval matched to a 60s scrape job
{
  "fieldConfig": { "defaults": {} },
  "targets": [
    { "expr": "rate(http_requests_total[5m])",
      "interval": "60s" }
  ]
}

Query Inspector: Splitting Grafana vs PromQL Faults

Query Inspector is the single tool that splits every empty-panel case. It shows the fully rendered query with variables interpolated, the datasource UID and request sent, the raw response received, and per-stage timings. Empty response with a sane rendered query means filters or timing; error response means routing or backend.

Build the split habit. Copy the rendered query into the Prometheus expression browser at the same range: empty in both means PromQL or staleness — take the Prometheus checklist. Data in Prometheus but empty in Grafana means variables, interval, or UID — stay in Grafana. One paste eliminates half the system from suspicion.

Teach the team the two-minute version. Range, inspector, hardcode variables, browser paste — in that order. Panels that once took hours of guesswork resolve before coffee cools, because evidence replaces theories at each step.

Each Inspector tab answers one question. Query shows the interpolated PromQL after variables expanded — copy it to Prometheus to split the fault. Request shows headers, UID, and timestamps for routing bugs. Response shows raw payloads where empty data arrays convict filters and error objects convict backends. Stats shows duration per stage, separating slow backends from slow browsers. Screenshot all four tabs into incident tickets; the next debugger starts from evidence instead of folklore. Inspector fluency is the difference between guessing and knowing.

# Replicate the panel query against Prometheus directly (what Inspector shows)
curl -s "http://prometheus:9090/api/v1/query_range?query=rate(http_requests_total[5m])&start=1699999200&end=1700002800&step=60s" | head -c 500

Locking the Fix: Versions, Docs, and CI Checks

Lock in the fix so the next on-call inherits a working dashboard, not a mystery. Save a version with an annotation naming the cause, document range and interval floors in the description, and set variable refresh defaults that survive deploys. A fixed panel without documentation breaks again on the next clone.

Add lightweight CI for provisioned dashboards: fail on unknown datasource UIDs, volatile default variable values, and min intervals below the owning job's scrape interval. These checks are cheap JSON assertions that catch the entire bug class before merge.

Finally, keep one known-good reference dashboard per datasource. When panels empty, diffing against the reference separates systemic faults (reference also empty — backend or time issue) from local ones (reference fine — panel settings). That single comparison halves diagnosis before any deeper digging starts.

Dashboard version history is an underused safety net: every save snapshots the JSON, and restore takes one click when an edit empties panels. Library panels share one definition across dashboards, so a fix propagates everywhere — but a bad edit does too, so test library changes on a scratch dashboard first. Viewer permissions confuse debugging when teams share links: a viewer without datasource permission sees empty panels that admins cannot reproduce. Check the share settings and the viewer's role before deep debugging. Most mysteries are access control wearing a No Data costume.