The scorecard is part of the alert-to-runbook-revision records guide. Fill in the metadata packet first; most checks below reuse its answers.
How each check is scored
Each check gets one of three marks, worth 2, 1 or 0 points, for a maximum of 12. The points are a reading aid, not a threshold. A trail scoring 6 with clear notes can be worth more to a reviewer than one scoring 10 on guesses.
| Mark | Points | Meaning | What to write beside it |
|---|---|---|---|
| Pass | 2 | The sponsor confirms it holds for the years in scope | The year it started holding |
| Partial | 1 | It holds for some years, teams or services | Which ones, roughly |
| Fail | 0 | It does not hold, or nobody can say | Who could find out |
Mark "fail" when the honest answer is "no idea". An unknown scored as a pass is the one mistake that makes the whole card unreliable.
The six checks
Ask each question as written. Every one can be answered from configuration, templates or habit, without reading an alert, an incident or a page.
| Check | Question for the sponsor | Pass | Partial | Fail |
|---|---|---|---|---|
| 1. Coverage | Which years does each of the three tools cover, and do they overlap? | All three overlap for 3+ years | Overlap exists but one tool is newer or was migrated | One tool's history is gone or under a year |
| 2. Alert to incident | Do production monitors page through the integration, rather than by email or chat? | Most monitors carry the paging handle | Some services page another way | Paging is manual or untracked |
| 3. Incident to postmortem | Does each postmortem list every related incident number? | A template field, used consistently | Usually one number, recurrences missing | Postmortems name no incidents |
| 4. Postmortem to runbook | Do runbook edits carry a version comment or link to their cause? | Usual practice since a known year | Some teams, some years | Edits are made silently |
| 5. Clock agreement | When a recovery auto-resolves an incident, is the fault tracked until it ends? | Faults are tracked separately until fixed | Only for major incidents | Recovery is treated as the end |
| 6. Validated fix | Does any later incident record that a revised runbook step was used? | Responders cite the runbook version | Occasionally, by habit | Never recorded |
Checks 2 and 3 show whether the chain exists. Checks 5 and 6 show whether it is trustworthy. A company that passes 2 and 3 but fails 5 has a connected history that tells part of the story wrongly, which is exactly what a reviewer needs to know in advance.
Check 1 also needs the earliest year PagerDuty's incident log covers. That log records acknowledgements, resolutions, automatic actions, reassignments and notes, so its retention decides how much of the response can be rebuilt. The PagerDuty incident export page covers that tool's records.
The stress test: an alert that clears before the fault
Check 5 is where most trails break, and it breaks by design rather than carelessness. When the paging handle sits in a monitor's recovery message, Datadog's PagerDuty integration closes the linked incident as soon as the metric recovers. PagerDuty then treats any later trigger with the same dedup key as a new incident rather than reopening the old one. A symptom that comes and goes therefore leaves a row of short, separate incidents in place of one long fault.
Test the card against a flapping alert rather than a single outage. Illustrative: at a fictional 310-person insurance claims software company, a disk-usage monitor on a reporting database replica crosses its threshold five times in one Saturday afternoon. Each crossing pages the weekend on-call engineer, and each recovers on its own after three to six minutes as a cleanup job frees space. That is five incidents with five numbers. On Tuesday a database engineer writes a review covering two of them, finds an unrotated query log, and edits the "replica disk cleanup" runbook from version 7 to version 8 with a comment naming the review. Nobody has paged on that monitor since, so no responder has cited version 8.
| Check | What the flapping case shows | Mark |
|---|---|---|
| 2. Alert to incident | All five crossings paged through the integration | Pass |
| 3. Incident to postmortem | Review names two of the five incidents | Partial |
| 4. Postmortem to runbook | Version 8 comment names the review | Pass |
| 5. Clock agreement | Incidents total about 22 minutes; the log problem ran for weeks | Fail |
| 6. Validated fix | No later incident to cite version 8 | Fail, for now |
Two lessons come out of it. A run of very short incidents on one monitor is itself a signal worth asking about: "When an alert flaps, does anyone group the incidents?" And a fail on check 6 can simply mean the fix worked and nothing has fired since, so write the reason beside the mark. The sponsor's one-sentence answer on how often this happens is worth more than any count you could estimate. The missing context page reads a recurring alert from the export side.
Reading a finished scorecard (Illustrative)
A fictional 450-person travel booking platform, scored with its head of reliability and its developer experience manager.
| Check | Mark | Points | Note |
|---|---|---|---|
| 1. Coverage | Partial | 1 | Monitoring migrated in 2020; earlier years unmapped |
| 2. Alert to incident | Pass | 2 | Paging handle on production monitors since 2020 |
| 3. Incident to postmortem | Pass | 2 | Template field since 2021 |
| 4. Postmortem to runbook | Partial | 1 | Comments common in two of five teams |
| 5. Clock agreement | Partial | 1 | Faults tracked separately for major incidents only |
| 6. Validated fix | Fail | 0 | Responders rarely cite runbook versions |
| Total | Not applicable | 7 of 12 | Strongest from 2021 onward |
Summarized for a reviewer: "Connected from 2021, with alert-clock gaps outside major incidents and little evidence that fixes were confirmed." That is accurate, short and useful. The data quality due diligence page explains how buyers think about gaps like these, and the document revision histories field guide covers check 4 in more depth.
Who scores it, and what the score does not settle
The SRE lead answers checks 1, 2 and 5; the platform engineering manager answers 3, 4 and 6. Neither can authorize a licensing discussion. That falls to the CEO, usually with the CTO and general counsel involved, because incident notes and runbooks can mention customers, vendors and security controls.
The company baseline still applies: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.
A score settles none of the following: whether the company's rights cover the records, what must be removed for security, whether any buyer wants this history, or what the company would be paid. SourceX reviews those under the company's own agreement, and does not claim a connector for every tool named here. Partners never ask for or handle raw incident data or runbook text.
Next step
Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.
If the scorecard shows a trail you can describe honestly, gaps included, and an officer will take the call, register as a partner, check who qualifies, and read the terms before you introduce the company.