An alert needs a next action

Imagine a small download service. Its outside check receives HTTP 200, but the file carries an older release marker than the one that should be public. A message says “site degraded” and arrives without the path, observation time, or a named responder. The check noticed a real discrepancy. The notification did not yet help anyone resolve it.

This is an illustrative service, not a report of an actual incident. The service preflight describes what an outside check can observe. Here the question begins after that observation: what makes the interruption worth a human’s attention?

Start with the user-visible symptom

Write the failure in terms of the service contract: “The public download path returned the wrong release marker at 14:05 UTC.” Include the observed URL or safe path, expected result, actual result, check location, and time. A process being alive is useful diagnostic evidence, but it does not cancel a wrong public response.

Google’s SRE chapter on Monitoring Distributed Systems separates symptoms from causes and explicitly treats a successful HTTP status with incorrect content as an error. Prometheus’s alerting guidance likewise recommends alerting on user-facing symptoms while keeping consoles available to find the cause. For this service, a stale public file is a better primary alert than simultaneous pages for a cache process, an origin process, and a deployment job.

Keep the check’s own uncertainty visible. A probe that cannot run is not proof that users cannot download. Label missing observations as monitoring failure, then confirm the public symptom from another independent path where possible. Do not delay an obvious user-visible incident just to make every checker agree.

Choose the interruption on purpose

“Important” does not always mean “wake someone now.” Route a condition by the decision it requires and the time available to make that decision. A solo maintainer may use different tools from a staffed on-call team, but the distinction still matters.

Interrupt now

Users are affected or a near-term loss is likely, and a responder can mitigate it now. Deliver an acknowledged page or equivalent urgent notification.

Track for later

Action is needed before a known deadline, but waiting until working hours is safe. Create an owned ticket with a due time.

Record only

The observation aids diagnosis or trends, but no one needs to act on it yet. Keep it queryable without interrupting a person.

A short persistence window can filter a harmless blip, but it also delays detection. Set it against the impact the service can tolerate, not a copied default. If a responder would always ignore the notification, change its condition or route. Google SRE’s guidance asks whether an alert is urgent, actionable, and actively or imminently user-visible; that is a useful test before making it interruptive.

Name a responder and a first decision

An alert needs more than a destination address. Name who owns the response, who receives it when that person is unavailable, how receipt is acknowledged, and when to escalate. If no one covers the service overnight, do not describe the route as continuous response. State the coverage boundary and decide whether the service’s promise must change.

A compact alert card can carry the decision without embedding credentials or a long runbook:

Observed

Public download marker differs from the expected release; include path, result, and observation time.

Owner

Current service responder; route to the named backup if acknowledgement is missed.

First move

Repeat the outside request and compare the result with the last known-good release record.

Escalate

If impact persists and no safe mitigation is known, involve the release owner and communicate the uncertainty.

Recovered

The same outside path serves the intended content consistently; close only after checking the observed failure mode.

The details can live in a linked runbook. The notification itself should make the first decision possible even when the responder is tired or unfamiliar with the last change.

Make the first move safe

Begin with read-only checks: reproduce the symptom from outside, record the time and exact response, inspect recent changes, and compare the intended release with what is actually served. Then choose a mitigation with a known boundary. Restarting a process because it is available is not a diagnosis; repeating a deployment after a lost response can make the state harder to understand.

If a change is the likely cause, confirm its outcome before retrying it—the timeout note explains why a missing reply is not a failed operation. If reverting is the safer path, check that the old version can read the current state, as in the rollback note. When neither action is yet safe, say so, preserve evidence, and escalate rather than reporting a speculative recovery.

Test the route all the way to a person

A correct rule is not enough if its notification never reaches an active device. The chain includes the check, rule evaluation, routing, delivery, acknowledgement, and fallback. Prometheus recommends monitoring that chain, including an outside test that notices when monitoring infrastructure itself stops working.

Send a clearly labeled test notification through the real route without breaking production. Verify who received it, whether the backup path works, and how long the handoff took. Also test a missing-checker condition separately from a service failure; otherwise a silent monitor can be mistaken for a healthy service. Keep test messages distinguishable from incidents so they do not teach responders to ignore genuine alerts.

Review what interrupted you

After an incident or a drill, ask whether the alert arrived early enough, reached the right person, described the actual symptom, and led to a useful action. Note duplicate notifications and failures discovered by users before the monitor. Change one condition or route at a time so the effect can be observed.

An alert that repeatedly produces no human action may belong in a ticket or a record. An alert that misses real impact needs a better signal. Neither problem is solved by adding every internal metric to the pager. Google SRE describes how excessive, unactionable pages can obscure serious ones; the cost is not just noise, but attention unavailable when it matters.