When NoData means disk full

Seven unrelated Grafana rules fired DatasourceNoData at once.

That looks like a datasource outage. The dashboard says “no data”, the alert metadata names the Prometheus datasource, and the obvious next action is to check whether Grafana can still reach Prometheus.

That was the wrong mental model.

Grafana could reach Prometheus. Prometheus could reach its scrape targets. The exporters were alive. The failure was lower and meaner: Prometheus was accepting scrapes and then failing to write the samples to disk because the TSDB volume was full.

The result was an alert storm that looked like a query problem while the real root cause was a storage commit failure.

The shape of the failure

The first clue was correlation.

One alert can be a broken exporter. Seven unrelated alerts, all switching to DatasourceNoData at about the same time, means the shared layer is suspect. These rules covered different surfaces: health checks, context coverage, post-session pipeline health, subscription utilization, and node readiness. They should not all lose data together unless something central broke.

Prometheus target discovery said the targets were healthy. Scrapes had recent timestamps. That ruled out the easy story: the collectors had not all died.

Then the query path got weird. A broad up query returned no samples, but the metric names still existed in Prometheus’s label index. That distinction matters: Prometheus still knew about the series. It just had no fresh usable samples for the alert windows.

The smoking gun was in the Prometheus service journal:

Scrape commit failed: disk quota exceeded

That line changes the whole diagnosis. The problem was not “Grafana has no datasource” and not “the exporters stopped emitting.” The scrape write path was failing after collection and before durable storage.

The configuration bug

The Prometheus instance had a long retention target and no size cap:

--storage.tsdb.retention.time=10y

That sounds generous. On a small volume, it is a time bomb.

Retention by time is not a storage budget. It says how long data may live if space exists. It does not say how much disk Prometheus is allowed to consume before it has to evict old blocks. A ten-year retention policy on a 20 GB volume can exhaust the volume when the retained data and write overhead exceed capacity.

It was not tiny forever.

The fix gave Prometheus enough space to recover and a size retention target below the expanded volume’s capacity:

--storage.tsdb.retention.time=10y
--storage.tsdb.retention.size=30GB

The immediate remediation was straightforward: resize the backing volume from 20 GB to 40 GB, add the 30 GB TSDB size cap, reload systemd, and restart Prometheus.

After restart, Prometheus became ready again, count(up) climbed back from zero to the full target set, and the missing health-check metrics returned with fresh timestamps. Disk usage settled around 40 percent of the expanded volume.

Size retention is not a hard filesystem quota. Prometheus counts WAL and memory-mapped head chunks toward that target, but deletes only eligible persistent blocks. WAL/head growth and compaction still need headroom, and free space needs its own alert. Time retention can fit a finite disk when capacity matches ingestion; our configured window exceeded the available space. Prometheus storage documentation describes those limits.

Run the failure locally

The annotated reproduction gist starts real Prometheus and Grafana instances, fills a bounded 64 MiB WAL filesystem, verifies Grafana’s DatasourceNoData, then restores space and checks fresh samples and alert recovery. Download the script and run it with Python 3 and Docker; no configuration edits are needed. The verified local run took 47.85 seconds including cleanup, with cached images.

The reproduction shortens lookback and alert evaluation to ten seconds. It uses WAL-only ENOSPC, while the incident reported quota exhaustion (EDQUOT). Filling the whole test TSDB also exposed a memory-mapped query-tracker crash, so the gist isolates the WAL failure and explains the difference. It leaves logs and API snapshots for inspection and cleans up its own Docker resources.

The debugging rule

DatasourceNoData is a symptom, not a root cause.

When many unrelated Grafana rules flip to NoData together, inspect the shared write path before chasing each exporter:

1. Check whether Prometheus itself is reachable and ready.
2. Check target health and recent scrape timestamps.
3. Query a universal metric like up.
4. Check whether metric names still exist even when samples are missing.
5. Read the Prometheus journal for WAL, block, quota, or compaction errors.
6. Inspect disk usage and retention settings together.

The key test is whether data is failing before collection, during query, or between collection and storage. This incident was the third case.

That middle zone is easy to miss. Most alert runbooks split the world into “exporter down” and “datasource down.” Prometheus adds another possibility: the exporter can emit, the scrape can run, and the commit can still fail.

The bigger lesson

Observability systems need storage SLOs too.

Long retention is useful only when paired with an explicit disk budget and an alert on that budget. Otherwise the monitoring stack becomes the next thing that needs monitoring, and it fails in the worst possible way: by making unrelated signals all look absent at once.

The fix was small. The useful part was the diagnosis: a flood of unrelated NoData alerts is often not seven missing metrics. It is one shared pipeline stage failing underneath them.

That is the pattern worth keeping.