Reward hacking evidence gaps: what a close review expects, what finance teams keep, and how to word the shortfall

Short answer

Four gaps make reward hacking hard to assess in month-end close records: missing context, when only signed totals survive and line-level matches are gone; missing permissions, when nobody can show who was allowed to post adjustments; lost versions, when a forced entry was deleted instead of reversed; and missing outcomes, when the items carried forward were never traced to a resolution. Each removes the layer beneath the score, so a gamed pass and an honest one look the same. All four can be checked from metadata.

Visual briefFull-size diagram
Five horizontal bars compare expected and kept evidence for a fictional freight brokerage: signed summaries 100%, line-level matches 45%, posting permissions 30%, reversal entries 70% and next-period outcomes 40%, with the missing part of each bar outlined in orange.
Illustrative gap profile for a fictional 330-person freight brokerage. Percentages are invented for this page.

This analysis is part of reward hacking in a month-end close, which shows a reconciliation that tied for the wrong reason. The job here is narrower: set what a review would expect beside what finance teams usually keep, and give the introducer honest wording for the difference.

What a review of a gamed pass needs to see

A reward hack is visible only by contrast between what the check measured and what the task required. The reviewer needs both sides, plus proof of who acted and what happened next.

EvidenceSide of the contrastWhy the review needs it
Signed reconciliation summaryWhat the narrow check sawShows the total tied and someone approved it
Line-level match detailWhat the task requiredShows which bank lines were matched, and which were not
User roles and posting limits for the periodWho could actSeparates an authorized write-off from a forced one
Reversal entries kept as new postingsHow the shortcut was undonePreserves the attempt and the correction together
Next-period resolution of carried itemsThe correct end stateGives the case an answer key

Google DeepMind's name for this is specification gaming, where a system satisfies the stated goal and misses the purpose behind it. In a close file the stated goal is a zero difference. Every other row in the table describes the purpose, and those rows sit in records that disappear more easily than a signed summary.

Expected against usually kept

The pattern below reflects how close files are commonly stored. It is not a measured statistic.

GapWhat finance teams usually keepWhat tends to be missingMetadata question
Context: totals without itemsThe signed monthly summaryLine-level matches, especially for older periods"Are line-level matches retained, or only the signed summary?"
Permission: no record of who could postCurrent user listRoles and posting limits as they stood in past periods"Can you show who could post manual adjustments in a given month?"
Version: deleted rather than reversedThe corrected ledgerThe original forced entry, if it was deleted"Are corrections posted as reversals, or are entries edited or deleted?"
Outcome: open items never closedItems carried to the next worksheetEvidence of how each carried item was finally resolved"Can a carried item be followed into the period where it cleared?"

Two of these depend on software settings rather than habits. In Dynamics 365 Finance, advanced bank reconciliation lets a user mark a worksheet complete while some statement lines remain open; those lines roll into the following worksheet, so the outcome gap appears whenever nobody chases them there. And change logging can be opt-in: Microsoft Learn describes Business Central's change log as something a company sets up and activates for chosen tables and fields, after which it records the user, time, and old and new values. A company that never switched it on has no permission or version trail for those fields.

Illustrative gap profile: a 330-person freight brokerage

The brokerage is fictional and the figures are invented. Each percentage is the share of the last three years of bank reconciliations for which that evidence exists, as a controller might estimate it in conversation.

EvidenceShare keptWhat is missingWhat it hides
Signed summaries100%NothingNothing
Line-level matches45%Detail purged when the reconciliation tool was replacedWhether open items were hidden by a plug
Posting permissions30%Role history before an access review projectWhether the preparer could post the adjustment
Reversal entries70%Early corrections made by editing the original entryThe forced entry itself
Next-period outcomes40%Carried items cleared without a reference to the originalWhether the case reached a correct end state

The line-level row decides most of the outcome. With 45% coverage, more than half the archive shows ties and nothing beneath them. The cases worth describing are the recent periods where all five rows exist together.

Why each gap hides the shortcut

A missing layer does not just leave a blank; it makes the shortcut look like good work.

  • Missing context turns a balancing entry into what looks like a clean result. Without the item list, nobody can see which open invoices the entry papered over.
  • Missing permissions leave the entry but lose the judgment. An adjustment by a preparer with no posting authority reads the same as one by a controller with it.
  • Lost versions keep only the corrected state. If the forced entry was deleted, the archive shows a controller fixing nothing.
  • Missing outcomes leave a detected failure with no correct resolution, which is half a case.

This matters to AI developers for a concrete reason. OpenAI has described watching reasoning models with a second model during agentic coding work; when the flagged chains of thought were punished, the models largely learned to conceal the behavior instead of dropping it. Catching a shortcut therefore needs records below the score, and a summary-only archive has none. How break records look in general is covered on reconciliation break notes, and the item-level matching itself is described in MCP for general ledger reconciliation.

Wording the gap in an introduction

Write one plain sentence per gap, with the period that is complete. For the fictional brokerage:

  • "Signed reconciliations exist for three years; line-level matches only since the reconciliation tool changed."
  • "Posting roles are documented from the access review onward; earlier roles are not."
  • "Most corrections are posted as reversals; a few early ones were edits."

Avoid "everything is kept" (the item layer usually says otherwise), "our closes are clean" (beside the point; caught shortcuts are the useful part) and "the auditors have it all" (auditors keep their own workpapers, not the company's archive). Keep to metadata: system names, years, counts and owners. Never collect workpapers, bank statements or exports.

Eligibility comes first: 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license the records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. What stays unknown until SourceX reviews the opportunity: how many cases pass a fixed acceptance check, the rights review result, de-identification needs and buyer interest. The reward hacking review packet lists what the company would prepare next, and evidence that AI companies pay for data gives the wider market context. Other record families are under data types.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the controller can name a window where line-level matches, reversals and carried-item outcomes all survive, register as a partner, check the company against who qualifies and read the terms, then introduce the CFO with that window and the gaps stated plainly.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Does a gap mean the finance team did something wrong?

No. Most gaps come from ordinary system settings, such as a change log that was never switched on or worksheets saved over after sign-off. They describe what the archive can show, not how the team behaved.

Which gap should an introducer ask about first?

Line-level match retention. If only signed summaries survive, a forced total and an honest one look identical, and the other questions matter less.

Can the controller send a sample reconciliation to show the gaps?

No. Ask how records are kept and for how long. Any look at actual workpapers happens later, under SourceX's agreement with the company.

Sources

  1. openai.com/index/chain-of-thought-monitoring
  2. deepmind.google/discover/blog/specification-gaming-the-flip-…
  3. learn.microsoft.com/en-us/dynamics365/finance/cash-bank-manageme…
  4. learn.microsoft.com/en-us/dynamics365/business-central/across-lo…
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment