Alert-to-runbook-revision records: tracing Datadog, PagerDuty and Confluence history when the alert clears before the fault does

Short answer

A company connects alert-to-runbook-revision records by tying each Datadog monitor alert to the PagerDuty incident it opened, the responder's notes and postmortem, and the Confluence runbook version that changed because of it. The links are monitor IDs, incident numbers, timestamps and page version comments. A referral partner can describe which links exist, for which years, and who owns them, without opening an alert or a runbook. The trail usually breaks when an alert clears before the underlying fault is fixed.

Visual briefFull-size diagram
A time scale with four rows: a monitor alert that clears after 17 minutes, two separate PagerDuty incidents, an underlying fault that runs on for six days, and a runbook page that moves from version 14 to a validated version 15.
Illustrative timeline for a fictional software company. The alert clock and the fault clock disagree; only the runbook edit closes the gap.

The business task: turning an incident into a better runbook

An engineering team gets paged, stabilizes a service, works out what really went wrong and rewrites the instructions so the next responder does better. The record of that loop, from signal to response to corrected procedure, shows how experienced people diagnose a live problem and what they learned from it. That kind of operational judgment is thin in public data, which is why what data is missing from AI training puts incident and procedure histories near the top of the list.

The loop is spread across three tools. Datadog holds monitors and their alert and recovery events. PagerDuty holds incidents and a log of what happened to each one: integration event data, who was notified and how, who acknowledged and resolved it, automatic actions, reassignments and notes. Confluence holds the postmortem and the runbook, and its page history lists each version's date, author and comment, with any two versions open to comparison.

How can a company connect alert-to-runbook-revision records across systems?

It connects them in three hops: monitor to incident by the integration's link, incident to postmortem by the incident number written into the page, and postmortem to runbook by the version comment or a link. The first hop is usually automatic. The other two depend on team habit.

HopFromToJoin that exists by designJoin that depends on people
1Datadog monitor alertPagerDuty incidentIntegration handle in the monitor notificationWhich monitor fired, if several share a service
2PagerDuty incidentConfluence postmortemNone in Confluence by defaultIncident number typed into the page
3PostmortemRunbook page versionNoneLink or comment on the runbook edit
4Runbook versionLater incidentNoneResponder notes that cite the step used

Hop 4 is the one most teams forget. It is the evidence that the revised runbook actually worked, and it only exists if a responder on a later incident wrote down which step they followed.

One joined timeline (Illustrative)

The timeline below follows one fault at a fictional 400-person software company, as drawn in the graphic above. Every value is Illustrative.

WhenSystemEventWhat joins it to the previous row
Night 1, 02:10DatadogLatency monitor alerts on the payments APIMonitor ID
Night 1, 02:11PagerDutyIncident #1 opens and pages the on-call engineerIntegration handle in the alert message
Night 1, 02:27Datadog and PagerDutyLatency drops; the recovery resolves incident #1 after 17 minutesRecovery notification
Night 2, 01:52PagerDutySame monitor fires again; incident #2 opensNothing links it to #1
Day 3ConfluencePostmortem names a connection pool leak and cites #1 and #2Incident numbers typed into the page
Day 6ConfluenceRunbook moves from v14 to v15, version comment cites the postmortemVersion comment
Day 6Platform teamFix deployed; the fault, open since night 1, endsDeploy note in the postmortem
Day 9PagerDutyIncident #3 resolved using the new v15 step, noted by the responderResponder note citing v15

Read top to bottom, the chain says something no single tool says: one fault lasted about six days, the alert clock showed 17 minutes, and the runbook change was validated three days after it was made.

What breaks when an alert clears before the problem is fixed

The first hop works so smoothly that it can hide the fault. Datadog's documentation says that when a PagerDuty handle is included in a monitor's recovery message, the recovery automatically resolves the linked incident. So a symptom that dips below its threshold closes the incident even if the cause is still there.

When the symptom returns, PagerDuty treats it as new. Its Events API documentation says a new trigger event with the same deduplication key as a resolved incident does not reopen it. The second page becomes incident #2, with no field pointing back to #1. In the incident system alone, one long fault looks like two short, unrelated ones.

Only people repair that: a postmortem that lists both incident numbers, and a runbook version comment that cites the postmortem. When a partner asks a sponsor about this chain, the useful questions are plain ones. Do postmortems name every related incident? Do runbook edits say why they were made? Since roughly when? The answers decide how much of the history can be joined, and none of them requires reading a page.

Who should a partner introduce, and what stays unknown?

The SRE lead or platform engineering manager can describe the chain; the CEO has to authorize a licensing discussion, usually with the CTO, COO or general counsel advising, because incident notes and runbooks can mention customers, vendors and security controls. The incident and on-call field guide, the introduction checklist for incident records and the SOPs and runbooks guide cover who to approach. For page version histories more broadly, see the document revision histories guide; for exports, the PagerDuty incident export page and the Confluence page.

The company baseline: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Unknown until SourceX reviews it under the company's own agreement: whether the company's rights cover the records, what must be removed for security reasons, how complete each hop is, and whether a buyer wants the result. Partners never ask for or handle raw incident data or runbook text. SourceX does not claim a connector for every tool named here, and a partner should not promise one. More record locations are in the systems hub.

Go deeper on alert-to-runbook-revision records

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If an engineering team you advise can show, at the metadata level, that its incidents lead to recorded runbook changes and an officer will take the call, register as a partner, check the company against who qualifies, and read the terms before you submit.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Are postmortems required for this kind of record chain to matter?

No, but they help. A written postmortem is often the only place that names the real cause and the runbook change it led to. Without one, the chain may still exist through incident notes and page version comments, and the sponsor can say how often that happened.

Does it matter if the company moved from one monitoring tool to another?

It matters for coverage, not eligibility. A migration usually splits the alert history into two date ranges with different identifiers. Record both ranges in the packet and mark whether any mapping between old and new monitors exists.

Can I describe the runbooks' subjects to SourceX?

Describe them in general terms, such as payment API, database failover or deploy rollback, and only if the sponsor agrees. Do not copy runbook text, host names, customer names or credentials into any message.

Sources

  1. docs.datadoghq.com/integrations/pagerduty
  2. developer.pagerduty.com/docs/88922dc5e1ad1-overview-v2-webhooks
  3. developer.pagerduty.com/docs/trigger-events
  4. support.atlassian.com/confluence-cloud/docs/create-edit-and-publis…
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment