Alert-to-runbook-revision completeness scorecard: pass, partial or fail checks that show where an incident evidence trail breaks

Short answer

An alert-to-runbook-revision trail breaks in six places: missing years, an alert that never reached an incident, an incident no postmortem names, a runbook edit with no reason given, clocks that disagree because the alert cleared before the fault did, and a fix nobody confirmed worked. This scorecard turns each of those into a yes-or-no question for the sponsor, scored pass, partial or fail. You never open a record. The result is a plain statement of how complete the trail is, and where it is not.

Visual briefFull-size diagram
Six checks (coverage, alert to incident, incident to postmortem, postmortem to runbook, clock agreement and validated fix) tick pass, partial or fail while a meter beside them rises to 7 of 12.
Illustrative scorecard for a fictional company. A partial or fail is a described gap, not a reason to stop.

The scorecard is part of the alert-to-runbook-revision records guide. Fill in the metadata packet first; most checks below reuse its answers.

How each check is scored

Each check gets one of three marks, worth 2, 1 or 0 points, for a maximum of 12. The points are a reading aid, not a threshold. A trail scoring 6 with clear notes can be worth more to a reviewer than one scoring 10 on guesses.

MarkPointsMeaningWhat to write beside it
Pass2The sponsor confirms it holds for the years in scopeThe year it started holding
Partial1It holds for some years, teams or servicesWhich ones, roughly
Fail0It does not hold, or nobody can sayWho could find out

Mark "fail" when the honest answer is "no idea". An unknown scored as a pass is the one mistake that makes the whole card unreliable.

The six checks

Ask each question as written. Every one can be answered from configuration, templates or habit, without reading an alert, an incident or a page.

CheckQuestion for the sponsorPassPartialFail
1. CoverageWhich years does each of the three tools cover, and do they overlap?All three overlap for 3+ yearsOverlap exists but one tool is newer or was migratedOne tool's history is gone or under a year
2. Alert to incidentDo production monitors page through the integration, rather than by email or chat?Most monitors carry the paging handleSome services page another wayPaging is manual or untracked
3. Incident to postmortemDoes each postmortem list every related incident number?A template field, used consistentlyUsually one number, recurrences missingPostmortems name no incidents
4. Postmortem to runbookDo runbook edits carry a version comment or link to their cause?Usual practice since a known yearSome teams, some yearsEdits are made silently
5. Clock agreementWhen a recovery auto-resolves an incident, is the fault tracked until it ends?Faults are tracked separately until fixedOnly for major incidentsRecovery is treated as the end
6. Validated fixDoes any later incident record that a revised runbook step was used?Responders cite the runbook versionOccasionally, by habitNever recorded

Checks 2 and 3 show whether the chain exists. Checks 5 and 6 show whether it is trustworthy. A company that passes 2 and 3 but fails 5 has a connected history that tells part of the story wrongly, which is exactly what a reviewer needs to know in advance.

Check 1 also needs the earliest year PagerDuty's incident log covers. That log records acknowledgements, resolutions, automatic actions, reassignments and notes, so its retention decides how much of the response can be rebuilt. The PagerDuty incident export page covers that tool's records.

The stress test: an alert that clears before the fault

Check 5 is where most trails break, and it breaks by design rather than carelessness. When the paging handle sits in a monitor's recovery message, Datadog's PagerDuty integration closes the linked incident as soon as the metric recovers. PagerDuty then treats any later trigger with the same dedup key as a new incident rather than reopening the old one. A symptom that comes and goes therefore leaves a row of short, separate incidents in place of one long fault.

Test the card against a flapping alert rather than a single outage. Illustrative: at a fictional 310-person insurance claims software company, a disk-usage monitor on a reporting database replica crosses its threshold five times in one Saturday afternoon. Each crossing pages the weekend on-call engineer, and each recovers on its own after three to six minutes as a cleanup job frees space. That is five incidents with five numbers. On Tuesday a database engineer writes a review covering two of them, finds an unrotated query log, and edits the "replica disk cleanup" runbook from version 7 to version 8 with a comment naming the review. Nobody has paged on that monitor since, so no responder has cited version 8.

CheckWhat the flapping case showsMark
2. Alert to incidentAll five crossings paged through the integrationPass
3. Incident to postmortemReview names two of the five incidentsPartial
4. Postmortem to runbookVersion 8 comment names the reviewPass
5. Clock agreementIncidents total about 22 minutes; the log problem ran for weeksFail
6. Validated fixNo later incident to cite version 8Fail, for now

Two lessons come out of it. A run of very short incidents on one monitor is itself a signal worth asking about: "When an alert flaps, does anyone group the incidents?" And a fail on check 6 can simply mean the fix worked and nothing has fired since, so write the reason beside the mark. The sponsor's one-sentence answer on how often this happens is worth more than any count you could estimate. The missing context page reads a recurring alert from the export side.

Reading a finished scorecard (Illustrative)

A fictional 450-person travel booking platform, scored with its head of reliability and its developer experience manager.

CheckMarkPointsNote
1. CoveragePartial1Monitoring migrated in 2020; earlier years unmapped
2. Alert to incidentPass2Paging handle on production monitors since 2020
3. Incident to postmortemPass2Template field since 2021
4. Postmortem to runbookPartial1Comments common in two of five teams
5. Clock agreementPartial1Faults tracked separately for major incidents only
6. Validated fixFail0Responders rarely cite runbook versions
TotalNot applicable7 of 12Strongest from 2021 onward

Summarized for a reviewer: "Connected from 2021, with alert-clock gaps outside major incidents and little evidence that fixes were confirmed." That is accurate, short and useful. The data quality due diligence page explains how buyers think about gaps like these, and the document revision histories field guide covers check 4 in more depth.

Who scores it, and what the score does not settle

The SRE lead answers checks 1, 2 and 5; the platform engineering manager answers 3, 4 and 6. Neither can authorize a licensing discussion. That falls to the CEO, usually with the CTO and general counsel involved, because incident notes and runbooks can mention customers, vendors and security controls.

The company baseline still applies: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

A score settles none of the following: whether the company's rights cover the records, what must be removed for security, whether any buyer wants this history, or what the company would be paid. SourceX reviews those under the company's own agreement, and does not claim a connector for every tool named here. Partners never ask for or handle raw incident data or runbook text.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the scorecard shows a trail you can describe honestly, gaps included, and an officer will take the call, register as a partner, check who qualifies, and read the terms before you introduce the company.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does a low score mean I should not introduce the company?

Not by itself. A scorecard full of honest partials describes a known shape, which is useful to a reviewer. The score helps you set expectations and explain the records accurately; eligibility and demand are decided only after SourceX reviews an application.

Can I score the checks by looking at the dashboards myself?

No. Ask the sponsor and record what they say. Looking at monitors, incidents or runbook pages is record access, which partners never need and should not request.

How often should the scorecard be redone?

Only when something changes the answers, such as a monitoring migration, a new postmortem template or a retention policy change. Note the date of each scoring so a later reader knows how current it is.

Sources

  1. docs.datadoghq.com/integrations/pagerduty
  2. developer.pagerduty.com/docs/trigger-events
  3. developer.pagerduty.com/docs/88922dc5e1ad1-overview-v2-webhooks
  4. support.atlassian.com/confluence-cloud/docs/create-edit-and-publis…
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment