# Alert-to-runbook: what a Datadog export loses | SourceX

Canonical: https://refer.sourcex.si/guides/alert-to-runbook-revision-missing-context
Last updated: 2026-11-04

A standalone Datadog export keeps the metric, the limit and the times a limit was crossed. It drops the people, the reasoning and the fix: who got the page, what they tried, what the cause turned out to be and which written procedure changed. It also cannot say whether a recovery meant the problem was solved, so one stubborn fault that clears and returns looks like two short, unrelated blips. That missing context sits in the paging tool and the team wiki.

This before-and-after reading belongs to [alert-to-runbook-revision records](/guides/chain-alert-to-runbook-revision), which explains how the three tools join. Here the job is narrower: take one Illustrative week, read it from a single export, then read it again with everything around it, and list exactly what changed.

## Before: one week of queue-depth numbers

**Illustrative and fictional.** A 380-person company runs product search for online retailers. A message queue feeds each retailer's catalog updates to a pool of indexing workers, and autoscaling adds workers when the backlog grows. Here is one week of one queue-depth check, exactly as a metrics export would show it.

| Time | Monitor | Event | Value |
|---|---|---|---|
| Tuesday 14:05 | Indexing queue depth | Alert | 61,000 messages, limit 50,000 |
| Tuesday 14:31 | Indexing queue depth | Recovered | 38,000 messages |
| Wednesday 09:12 | Indexing queue depth | Alert | 57,000 messages |
| Wednesday 09:38 | Indexing queue depth | Recovered | 41,000 messages |

Two alerts, each lasting 26 minutes, each recovering on its own. A reader would conclude that catalog traffic spiked twice and the platform healed itself both times. Capacity planners might even cite the week as proof that autoscaling works.

## After: the same week with the paging log and the wiki added

Add the incident log and the written record and the story changes. Values remain Illustrative.

**Paging log, Tuesday.** The alert opened incident #4410 and woke the on-call engineer, who acknowledged at 14:07 and noted that extra indexing workers had spun up. At 14:31 the recovery closed the incident by itself. Datadog documents this: when the PagerDuty handle sits in a monitor's recovery message, recovery resolves the linked incident.

**Paging log, Wednesday.** The same check opened a fresh incident, #4417. The Events API documentation says a new trigger carrying the dedup key of a resolved incident does not reopen it, so nothing ties #4417 to #4410. A second engineer spotted one retailer's catalog feed failing validation and retrying without end, swelling the backlog.

**Wiki, Thursday.** An incident review named the cause: one malformed feed stuck retrying since Tuesday. Autoscaling had masked it by draining every other tenant's messages faster. The review cited both incident numbers.

**Wiki, Friday.** The procedure page "Search indexing backlog" moved from version 8 to version 9. Its new first step: before adding workers, check the dead-letter queue for one tenant's messages repeating. The version comment cited the review.

**Paging log, three weeks later.** Incident #4562, same check. The on-call note says the version 9 step found a stuck feed within ten minutes, and the backlog cleared without extra workers.

## What was lost, and where it lives

| Lost context | What it changes | Where it lives | In the Datadog export? |
|---|---|---|---|
| Who responded and what they tried | Shows the human diagnosis, not just the signal | PagerDuty incident log: notifications, acknowledgments, notes | No |
| That two alerts were one fault | Turns two self-healing bursts into a two-day fault masked by autoscaling | Postmortem citing both incident numbers | No |
| The cause | A tenant's malformed feed, not load | Postmortem in Confluence | No |
| The procedure change | Adds a check before scaling | Runbook version 9, with date, author and comment | No |
| Proof the change worked | A later incident resolved with the new step | Responder note on incident #4562 | No |

PagerDuty keeps a log of what happens to each incident, including notifications, who acknowledged or resolved it, reassignments and notes. Confluence's page history lists each version's date, author and comment and lets two versions be compared. Neither reaches back into the Datadog export, and the export does not reach forward into them.

## Why this context is the valuable part

For model training and evaluation, the right-hand side of the case is what matters: an operational decision, the reasoning behind it and a tested outcome. The signal alone is plentiful; the diagnosis and the corrected procedure are scarce. That is why [what data is missing from AI training](/questions/what-data-is-missing-from-ai-training) lists incident and procedure histories, and why a company whose postmortems name incident numbers holds more than its monitoring data suggests.

The same before-and-after logic applies to any document trail where versions record decisions; see the [document revision histories guide](/guides/a-partner-field-guide-to-finding-companies-with-document-revision-histories).

## Questions that show whether the after view exists

Ask the SRE or platform lead, and note the years each answer covers:

1. Does each monitor page through PagerDuty, and does recovery auto-resolve incidents?
2. Do postmortems list every related incident number?
3. Do runbook edits carry a version comment saying why they changed?
4. Do responders note which runbook step they used?

None of these needs an incident, a page or a log line. The CEO authorizes a licensing discussion, usually with the CTO, COO or general counsel advising, because incident notes and runbooks can mention customers, vendors and security controls. The company needs 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. To package the answers, use the [alert-to-runbook-revision metadata packet](/resources/alert-to-runbook-revision-metadata-packet); to score them, the [completeness checklist](/resources/alert-to-runbook-revision-completeness-checklist). Export details are on the [PagerDuty incident export page](/systems/pagerduty-export-incidents) and the [Confluence page](/systems/confluence); more tools are on the [systems hub](/systems).

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If an engineering team you advise answers yes to most of the four questions and an officer will take the call, [register as a partner](/register), check [who qualifies](/who-qualifies) and read the [program terms](/terms) before you introduce it.

## Sources

- https://docs.datadoghq.com/integrations/pagerduty/
- https://developer.pagerduty.com/docs/88922dc5e1ad1-overview-v2-webhooks
- https://developer.pagerduty.com/docs/trigger-events
- https://support.atlassian.com/confluence-cloud/docs/create-edit-and-publish-a-page/
- https://refer.sourcex.si/terms
