One indexing backlog, read twice: what a standalone Datadog export loses from alert-to-runbook revision

Short answer

A standalone Datadog export keeps the metric, the limit and the times a limit was crossed. It drops the people, the reasoning and the fix: who got the page, what they tried, what the cause turned out to be and which written procedure changed. It also cannot say whether a recovery meant the problem was solved, so one stubborn fault that clears and returns looks like two short, unrelated blips. That missing context sits in the paging tool and the team wiki.

Visual briefFull-size diagram
A horizontal strip shows two 26-minute Datadog alerts with blank gaps between them; pieces from PagerDuty and Confluence slide up and down into the gaps, adding the responders' notes, a postmortem naming a stuck feed, runbook version 9 and a later incident resolved with the new step.
Illustrative case for a fictional e-commerce search provider. The export shows two short alerts; the joined history shows one fault and a validated fix.

This before-and-after reading belongs to alert-to-runbook-revision records, which explains how the three tools join. Here the job is narrower: take one Illustrative week, read it from a single export, then read it again with everything around it, and list exactly what changed.

Before: one week of queue-depth numbers

Illustrative and fictional. A 380-person company runs product search for online retailers. A message queue feeds each retailer's catalog updates to a pool of indexing workers, and autoscaling adds workers when the backlog grows. Here is one week of one queue-depth check, exactly as a metrics export would show it.

TimeMonitorEventValue
Tuesday 14:05Indexing queue depthAlert61,000 messages, limit 50,000
Tuesday 14:31Indexing queue depthRecovered38,000 messages
Wednesday 09:12Indexing queue depthAlert57,000 messages
Wednesday 09:38Indexing queue depthRecovered41,000 messages

Two alerts, each lasting 26 minutes, each recovering on its own. A reader would conclude that catalog traffic spiked twice and the platform healed itself both times. Capacity planners might even cite the week as proof that autoscaling works.

After: the same week with the paging log and the wiki added

Add the incident log and the written record and the story changes. Values remain Illustrative.

Paging log, Tuesday. The alert opened incident #4410 and woke the on-call engineer, who acknowledged at 14:07 and noted that extra indexing workers had spun up. At 14:31 the recovery closed the incident by itself. Datadog documents this: when the PagerDuty handle sits in a monitor's recovery message, recovery resolves the linked incident.

Paging log, Wednesday. The same check opened a fresh incident, #4417. The Events API documentation says a new trigger carrying the dedup key of a resolved incident does not reopen it, so nothing ties #4417 to #4410. A second engineer spotted one retailer's catalog feed failing validation and retrying without end, swelling the backlog.

Wiki, Thursday. An incident review named the cause: one malformed feed stuck retrying since Tuesday. Autoscaling had masked it by draining every other tenant's messages faster. The review cited both incident numbers.

Wiki, Friday. The procedure page "Search indexing backlog" moved from version 8 to version 9. Its new first step: before adding workers, check the dead-letter queue for one tenant's messages repeating. The version comment cited the review.

Paging log, three weeks later. Incident #4562, same check. The on-call note says the version 9 step found a stuck feed within ten minutes, and the backlog cleared without extra workers.

What was lost, and where it lives

Lost contextWhat it changesWhere it livesIn the Datadog export?
Who responded and what they triedShows the human diagnosis, not just the signalPagerDuty incident log: notifications, acknowledgments, notesNo
That two alerts were one faultTurns two self-healing bursts into a two-day fault masked by autoscalingPostmortem citing both incident numbersNo
The causeA tenant's malformed feed, not loadPostmortem in ConfluenceNo
The procedure changeAdds a check before scalingRunbook version 9, with date, author and commentNo
Proof the change workedA later incident resolved with the new stepResponder note on incident #4562No

PagerDuty keeps a log of what happens to each incident, including notifications, who acknowledged or resolved it, reassignments and notes. Confluence's page history lists each version's date, author and comment and lets two versions be compared. Neither reaches back into the Datadog export, and the export does not reach forward into them.

Why this context is the valuable part

For model training and evaluation, the right-hand side of the case is what matters: an operational decision, the reasoning behind it and a tested outcome. The signal alone is plentiful; the diagnosis and the corrected procedure are scarce. That is why what data is missing from AI training lists incident and procedure histories, and why a company whose postmortems name incident numbers holds more than its monitoring data suggests.

The same before-and-after logic applies to any document trail where versions record decisions; see the document revision histories guide.

Questions that show whether the after view exists

Ask the SRE or platform lead, and note the years each answer covers:

  1. Does each monitor page through PagerDuty, and does recovery auto-resolve incidents?
  2. Do postmortems list every related incident number?
  3. Do runbook edits carry a version comment saying why they changed?
  4. Do responders note which runbook step they used?

None of these needs an incident, a page or a log line. The CEO authorizes a licensing discussion, usually with the CTO, COO or general counsel advising, because incident notes and runbooks can mention customers, vendors and security controls. The company needs 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. To package the answers, use the alert-to-runbook-revision metadata packet; to score them, the completeness checklist. Export details are on the PagerDuty incident export page and the Confluence page; more tools are on the systems hub.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If an engineering team you advise answers yes to most of the four questions and an officer will take the call, register as a partner, check who qualifies and read the program terms before you introduce it.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is a Datadog export useless without the other systems?

No. It is an accurate record of what was measured and when limits were crossed. It simply stops at the signal: it cannot show who responded, what they concluded or what the team changed afterward.

What if the company writes postmortems in Google Docs instead of Confluence?

The chain is the same; only the place changes. Ask whether postmortems name incident numbers and whether runbook edits are dated and attributed, wherever they live.

Should the introducer see an example incident to judge this?

No. Every question on this page can be answered from metadata: which tools, which years, whether incident numbers appear in postmortems and whether runbook versions carry comments.

Sources

  1. docs.datadoghq.com/integrations/pagerduty
  2. developer.pagerduty.com/docs/88922dc5e1ad1-overview-v2-webhooks
  3. developer.pagerduty.com/docs/trigger-events
  4. support.atlassian.com/confluence-cloud/docs/create-edit-and-publis…
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment