# Alert storm worked case, step by step | SourceX

Canonical: https://refer.sourcex.si/guides/alert-deduplication-decisions-exception-examples
Last updated: 2026-11-04

The records that explain an alert-handling exception are the ones made where automation got it wrong and a person corrected it: the alert groups as the tooling formed them, the analyst's merge with a written reason, the rule change that followed and the next incident that proved the change worked. In the Illustrative storm below, one storage fault produced 147 alerts and three incidents before an analyst tied them together.

This worked case is part of the [alert deduplication decisions guide](/guides/workflow-alert-deduplication-decisions), which covers what to inventory before a monitoring tool is replaced. Here we follow one hard case minute by minute, name who decided what, and finish with how an observability consultant can spot the same pattern at a client.

## What went wrong at 14:02 (Illustrative)

Invented for this page. A fictional 380-person logistics software company runs a 24/7 operations center. Its services send alerts through separate integrations into an incident platform that groups alerts by text similarity, and its own alert rules sit in version control.

At 14:02 latency rises on one shared storage volume. Nothing alerts on the volume itself, because nobody wrote a rule for it. Three services that depend on the volume start to fail in their own ways:

- 61 alerts for database latency
- 52 alerts for API error rate
- 34 alerts for queue backlog

That is 147 alerts between 14:03 and 14:13. Each service words its summaries differently, and each sends through its own integration.

Two documented behaviors explain why the tooling did not see one outage. PagerDuty's Intelligent Alert Grouping compares the Summary field for textual similarity by default, so symptoms described in different words may not group. Its Events API merges events that share a dedup_key, but the same key sent through two different integrations on one service is not merged, and a trigger with no key always opens a new alert. Neither behavior is a fault. Together they meant the configuration had no way to know that latency, errors and backlog were one event.

## Seven decisions, and the record each one left

The table is the evidence asset for this page. Read across: the moment, who decided, what they decided and which record holds that decision.

| Time | Who decided | Decision | Record it left | System |
|---|---|---|---|---|
| 14:02 | Nobody | No alert exists for the storage volume | Absence of a rule, visible in rule history | Alert rules in version control |
| 14:03 to 14:13 | Tooling | 147 alerts accepted, grouped by service | Alert groups with timestamps and keys | Incident platform |
| 14:06 to 14:13 | Tooling | Three incidents opened, three engineers paged | Three incident records and page logs | Incident platform |
| 14:19 | Operations center analyst | Merge the three incidents into one | Merge entry plus the note "Common cause: storage volume latency. API and queue alerts are downstream." | Incident platform log and notes |
| 15:40 | Storage team | Fix the volume; all streams clear within six minutes | Resolution entry and clear events | Incident platform |
| Next day | Platform team lead | Add a storage-latency alert and an inhibition rule for the downstream API and queue alerts on the same cluster | Change record citing the merged incident | Version control and change system |
| Six weeks later | Tooling, under the new rule | A similar fault opens one incident with one page | One incident with suppressed downstream alerts | Incident platform |

The 14:19 merge left more than a note. PagerDuty documents that when incidents are merged, their alerts move into the target incident and the source incidents are resolved with the reason "Merged" and a reference to the target. The merge is therefore recorded twice: as a human sentence and as a structured link.

The next-day rule uses a standard mechanism. In Prometheus Alertmanager, an inhibition rule mutes target alerts when a matching source alert is firing, and only when the labels named in its equal list carry the same values. Here, "same cluster" is that condition.

## Why the 14:19 note and the next-day change carry the weight

Most of the 147 alerts are noise for anyone studying how the work was done. Three records are not.

1. **The merge with a reason.** It is a human judgment, made under time pressure, stating a cause before the cause was confirmed. Comparing 14:19 with 15:40 shows the analyst was right, and early.
2. **The rule change citing the incident.** It turns a one-off judgment into policy. Without the citation, the inhibition rule looks like routine tuning; with it, the rule has a documented origin.
3. **The six-week check.** One incident and one page show that the policy did what the analyst intended. A decision without a checked outcome is an opinion.

Read in sequence, those records show the pattern an AI system working in operations would need to learn: symptoms that look unrelated, a person who recognized a shared cause, a stated reason, a configuration change and evidence it held. PagerDuty also notes that manual merges made in its web app train its grouping algorithm, which says something about how much weight such corrections carry inside the tools themselves. The broader case for exception histories is set out in [exception handling records for AI agents](/guides/exception-handling-ai-agents-data).

If one link is missing, the case falls apart. Without the note, the merge looks like tidying. Without the change record, nobody can say why the inhibition rule exists. The [system map for alert deduplication records](/guides/alert-deduplication-decisions-records-across-systems) shows the keys that keep those links intact.

## What this case tells a partner, and what it does not

A partner does not need any of these records to recognize the pattern. The case is a reason to ask questions, not a request for data. Keep discovery at metadata level: which systems, how many years, roughly how many incidents a year, what share of merges carry a written reason, who owns the records.

The company needs 50+ full-time employees at peak (contractors excluded), a documented operating history, rights to license the records and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

Alerts describe production systems, and analyst notes name employees. If the company monitors customer environments, customer contracts may limit what it can do with those alerts. The operations center director can explain the records; the person who can authorize a licensing discussion usually sits higher, as the [who-can-license decision tree](/questions/who-can-license-alert-deduplication-decisions-records) sets out. Whether any record class is licensable, what would need removing, and whether there is buyer interest stays unknown until SourceX reviews it. Partners sign the Channel Partner Agreement before submitting introductions.

## How to recognize a similar storm at a client

Observability consultants see the signs during tuning and migration work. Without opening an alert, ask:

- **Has one outage ever paged several teams at once?** If the answer comes with a story, the case exists.
- **When an analyst merges incidents, do they write why?** In the incident notes, in chat or not at all.
- **Do rule changes reference the incident that prompted them?** A commit message or change ticket that names the incident keeps the chain whole.
- **Are alert rules kept in version control, and since when?**
- **Is the incident platform or monitoring stack being replaced, and when does the old one go dark?**

[PagerDuty incident exports](/systems/pagerduty-export-incidents) describes what one platform holds. The same "unmatched signal, human explanation, checked fix" shape appears in other sectors; see [documented logistics exceptions](/guides/how-to-identify-logistics-exceptions-and-their-documented-resolutions). For managed service providers, where customer contracts add a layer, see [managed service providers and data licensing](/industries/refer-managed-service-providers-data-licensing).

## Next step

> **Refer a company. No cap on rewards.** Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a client's operations center can trace a storm like this from the first alert to the rule that fixed it, [register as a partner](/register), check the company against [who qualifies](/who-qualifies) and read the [program terms](/terms) before introducing its decision-maker.

## Sources

- https://developer.pagerduty.com/docs/events-api-v2/trigger-events/index.html
- https://support.pagerduty.com/main/docs/intelligent-alert-grouping
- https://support.pagerduty.com/main/docs/edit-incidents
- https://prometheus.io/docs/alerting/latest/configuration/
- https://refer.sourcex.si/terms