Who Watches the Watchman: When Prometheus Quietly Stops Scraping

The monitoring failure everyone prepares for is the loud one. A service falls over, a rule fires, somebody gets paged. That is the easy case. The system worked.

The failure that actually costs you is the quiet one. Prometheus stops scraping. Panels keep rendering whatever they had last, or go blank in a corner nobody looks at. No alert fires, because alerts fire on data and there is no data. Everything is green, the on-call phone is silent, and that silence reads as health.

It turns up in help threads constantly. Someone's Prometheus quietly stops collecting inside an LGTM stack, and the discovery happens later and by accident, when a human goes looking for a graph that is not there.

Your Alerting System Cannot Alert On Its Own Death

This is structurally different from a service outage, and the difference is the whole problem.

When your database goes down, something outside the database notices. When your monitoring goes down, the thing that would notice is the monitoring. A Prometheus that has stopped scraping cannot evaluate a rule that says "Prometheus has stopped scraping," because rule evaluation is part of what stopped.

The same applies downstream. If your rules fire perfectly but the webhook to your paging provider has returned 401 for three weeks, your stack's only output is a warm feeling.

Absence of alerts is not evidence of health. It is absence of evidence, and those are not the same thing. If you are standing up your first Prometheus and Grafana stack, build the answer to this in on day one instead of bolting it on after the first embarrassment.

The Dead Man's Switch

The fix is not clever. It is an alert that fires constantly, on purpose, forever. Illustrative shape:

groups:
  - name: watchdog
    rules:
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: none
        annotations:
          summary: Always firing by design. Its absence means the pipeline is broken.

Route that one alert to a heartbeat service -- Dead Man's Snitch, Healthchecks.io, whatever your team already pays for -- on its own Alertmanager route with a short repeat_interval so the signal arrives on a predictable cadence. The external service expects that beat. When the beat stops, it pages you.

Note the inversion. Every other alert pages on the presence of something bad. This one pages on the absence of something good, which is exactly the shape a dead collector produces.

The heartbeat also has to land outside the blast radius. If the receiver runs on the same cluster, node, or egress path as Prometheus, the failure that kills your monitoring takes the watchdog with it. A watchdog that shares a failure domain with the thing it watches is decoration. A different provider is best. The same cluster is worthless.

Staleness, Not Just Thresholds

Most alert rules ask whether a number is too high. You also need rules that ask whether the number is still arriving at all. Three cheap ones, as patterns to adapt rather than paste:

# A target is failing its scrape
- alert: TargetDown
  expr: up == 0
  for: 5m

# The target count fell off relative to an hour ago
- alert: ScrapeTargetsMissing
  expr: count(up) < 0.9 * count(up offset 1h)
  for: 10m

# A series we depend on has gone stale
- alert: MetricStale
  expr: time() - max(timestamp(node_cpu_seconds_total)) > 600
  for: 5m

Reach for absent() when an entire job can vanish from service discovery, since up == 0 only matches targets Prometheus still knows about. A job that disappears leaves no series behind to compare against.

Then watch the storage layer. A full data volume or a wedged TSDB causes this often, and it is undramatic from the outside. Alert on free space on the Prometheus data volume and on the prometheus_tsdb_* failure counters. The same discipline belongs on the hosts themselves -- our server monitoring setup guide covers the agent-side half of this.

The Takeaway

Four things, and none of them take a week:

• Add a permanently firing Watchdog alert and route it to an external heartbeat service that pages when the stream stops.
• Add up == 0, target-count, and metric-age rules alongside your threshold alerts.
• Confirm the heartbeat receiver shares no failure domain with Prometheus. Not the cluster, not the network path.
• Test the chain end to end. Stop Prometheus deliberately and confirm a human's phone rings.

That last one is the step teams skip, and the only one that proves the other three. A page nobody receives equals no page at all, which is why rotation and escalation matter as much as the rules -- the substance behind 24/7 monitoring coverage.

Monitoring you have never watched fail is not monitoring you have tested. It is monitoring you have assumed.