Data-break-to-repair completeness scorecard: seven pass, partial or fail checks on a data team's fix history

Short answer

A data-break-to-repair trail breaks in five places: run results that were never archived, so the failure itself has no record; commits without a ticket key; model code not kept in version control for the whole period; a verification given only in chat or a meeting; and a definition change that passes every test while a business number moves. Seven checks, each scored pass, partial or fail from a conversation, show whether a company's Snowflake, dbt and Jira history is complete enough to discuss.

Visual briefFull-size diagram
A grid of seven check tiles for a data team's break-and-repair history turns green or orange one by one, and a horizontal completeness bar beneath fills to 7 of 14 points.
Illustrative scorecard for a fictional freight broker. Run artifacts and the definition log are the checks that fail.

This scorecard is part of the data-break-to-repair records guide. If you only need to describe the records, the ten-field description sheet is shorter; the scorecard adds a judgment of how far each link holds.

How the scoring works

Each check scores 2 for pass, 1 for partial and 0 for fail; seven checks make 14. Pass means the analytics engineering lead or controller says the condition holds across the whole period they describe. Partial means it holds for some years or some models and they can say which. Fail means it does not hold or nobody knows, and you write "unknown" next to the zero.

Use estimates. "Roughly what share of data commits carry a ticket key?" is the right shape of question; counting commits yourself is not your job. Record answers in your own notes with the year each condition started. No model names, query text, ticket contents or test output belong there.

Seven checks for a break-to-repair trail

CheckAsk at metadata levelPassPartialFail
1. Coverage overlap"Since when have the warehouse, the model code and the incident tracker all existed?"Three or more years with all three in placeUnder three years of overlapOne of the three replaced without history
2. Run artifact retention"Are test outcomes stored for every run, and since when?"Archived per run for the whole periodArchived from a recent yearOnly the latest run survives
3. Ticket key in commits"Roughly what share of data commits carry a Jira key?"Most commits, enforced by a hook or templateCommon by habit, from a known yearRare or never
4. Model version history"Has the model code been in version control for its whole life?"Yes, from the first modelYes, from a later migrationNo; older SQL lived in the warehouse only
5. Tests on reporting models"Roughly what share of finance-reporting models have data tests?"Most, including relationship tests on keysSome, mostly not-null and uniqueFew or none
6. Verification outcome"Where does finance sign off a corrected report?"Recorded in the ticket or a tracked systemRecorded from a recent year; email beforeIn meetings or chat only
7. Definition change log"When a metric's logic changes, is that written down anywhere?"Logged with reason and approverLogged for major metrics onlyNot logged

Check 2 decides whether a past failure exists at all. dbt writes a run results file for each invocation, listing executed nodes by unique_id with a status, and test results carry a failures count. Only executed nodes appear, so each file is a snapshot of one run. A team that never archived these files has repairs with no recorded break.

Check 3 decides whether the repair joins its reason without reading text. Jira links a commit to a work item when the commit message contains the key, so a high share of keyed commits means tickets and code join by design.

A note on the platform's own statement log: in Snowflake it covers the last 365 days. It anchors when things ran, but it is not scored separately because it cannot show why, and anything older than a year survives only if the company copied it.

The green-run test: when every check passes but the number moves

Check 7 deserves its own question because nothing else catches it. A dbt data test is a query for failing rows and passes when it returns none; the four generic tests check uniqueness, nulls, accepted values and relationships. None of them checks what a metric means. If an engineer changes how freight accessorial charges are allocated, every test can pass, the run is green, and margin for closed months shifts.

Ask the controller one question: "When a reported number changed and nothing failed, how did finance find out, and where was that recorded?" Score pass if the answer names a log or a ticket type for definition changes, partial if it names reopened tickets or a finance change request for some metrics, and fail if the answer is "we noticed in the monthly review." A fail is honest and common; it goes in the description as a known gap.

Illustrative: a freight broker's data team scores 7 of 14

Illustrative. A fictional 230-person freight broker with a four-person data team answered the checks in one call with its analytics engineering lead and a follow-up with the controller.

CheckMarkPointsWhat they said
1. Coverage overlapPass2Warehouse since 2020, dbt since 2020, Jira data project since 2021
2. Run artifact retentionFail0Only the latest run is kept
3. Ticket key in commitsPartial1Common since 2023, not enforced
4. Model version historyPass2dbt project in Git from the start
5. Tests on reporting modelsPartial1Not-null and unique on most; few relationship tests
6. Verification outcomePartial1Ticket comments since 2024; email before
7. Definition change logFail0Not logged
Total7 of 147Fixes traceable; failures and meaning changes not

Summarized for the sponsor: the company can show what was fixed and, from 2023, why; it cannot show the original failure, and definition changes are undocumented. That is a fair description. Investors reviewing several portfolio companies can add these seven questions to a portfolio review handoff checklist, and teams mid-migration can pair it with the ERP replacement handoff checklist. The value of recorded judgment is the same pattern seen in reconciliation exception records.

Who answers, and what the scorecard leaves open

The analytics engineering lead answers checks 1 to 5. The controller or finance lead answers 6 and 7. A licensing discussion usually needs the CFO, COO, CEO or general counsel, because warehouse tables may include customer records under contract terms the data team does not control.

Confirm the company baseline before introducing: 50+ full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. Sign the Channel Partner Agreement first.

The score does not tell you whether the rights cover the records, whether any buyer wants them, or what they would be worth. Those stay unknown until SourceX reviews the company under its own agreement. SourceX does not claim a connector to any tool named here. Other record locations are listed under systems.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If a data team you advise has scored all seven checks, fails included, and an officer will take the call, register as a partner, check who qualifies, and read the terms before you submit.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Do I need to understand dbt to run this scorecard?

No. Each check is a plain question for the analytics engineering lead or the controller. You record their answer and the years it covers; you do not judge the code.

Is a team with no definition log a weak introduction?

Not necessarily. Most teams do not keep one. Scoring it as a fail tells the reviewer where judgment is undocumented, which is useful context rather than a reason to stop.

Should the score go to the company?

Share it with the sponsor who answered, if they want it. It describes their own systems, and they may correct an answer, which improves the description.

Sources

  1. docs.getdbt.com/reference/artifacts/run-results-json
  2. docs.getdbt.com/docs/build/data-tests
  3. support.atlassian.com/jira-software-cloud/docs/reference-issues-in…
  4. docs.snowflake.com/en/sql-reference/account-usage/query_history
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment