Partial-credit evaluation evidence gaps: why stalled onboardings, shared logins and moving checklists stop step scoring

Short answer

Four gaps make partial-credit evaluation hard to assess: missing context (stalled attempts were never kept, so only finished cases survive), missing permissions (approvals made under shared logins or by the wrong role cannot count), versions that moved (the checklist changed and nobody dated it), and unrecorded final outcomes (the vendor was approved, but nobody can tell if the first payment worked). Each gap removes scoreable steps. None rules a company out; together they set how many cases can be graded.

Visual briefFull-size diagram
Six segmented bars, one per kind of evidence a step-scored onboarding needs, each with ten segments expected; filled segments show what companies typically keep and outlined orange segments show what is missing, with the largest gaps in stalled attempts, approver authority, checklist version and first payment link.
Illustrative pattern, not measured data: which evidence for a step-scored vendor onboarding workflow archives tend to keep and which they tend to lose.

This gap analysis is part of the guide to partial-credit evaluation in an ordinary workflow, which scores a vendor onboarding that collected the tax form, insurance certificate and bank details but never got the controller's approval. That case is the reference here. The question is what a review needs to score attempts like it across a whole archive, and what usually stands in the way.

Why step scoring is sensitive to gaps

A pass-or-fail grade needs one fact per case: did it finish? Partial credit needs a fact per step. Published agent benchmarks such as TheAgentCompany grade workplace tasks step by step, with extra weight for finishing. Apply that kind of rule to a company's records and every missing step record turns into a step nobody can grade. A gap that would cost a pass-or-fail archive nothing can cost a step-scored archive most of its value.

What a review expects for each onboarding

For the reference workflow, a review expects six kinds of evidence on every attempt, finished or not.

  1. Step entries. A separate record for each stage: tax form, insurance, bank details, final authorization.
  2. Completion evidence. The stage's output exists: a signed Form W-9, a dated proof of cover, a bank account submission.
  3. Stalled and rejected attempts. Requests that stopped partway, kept with their last status.
  4. Approver and authority. Who approved each step, and whether policy allowed them to.
  5. Checklist version. Which list of steps applied on the date of the attempt.
  6. Downstream result. Whether the activated vendor was paid without a problem.

Some systems make items 1 and 4 easy. Microsoft's documentation for Dynamics 365 Finance says approval-flagged vendor edits wait as proposed changes until every approval completes, and bank accounts run through a separate approval workflow. History like that is close to what a review needs. Email-based setup rarely is.

Expected versus kept: the gap table

The pattern below describes how workflow archives commonly store data. It is not a measurement of any company.

EvidenceExpectedTypically keptUsually missingEffect on scoring
Step entriesOne record per stepSteps run in the workflow toolSteps handled by emailThose steps cannot be graded
Completion evidenceDocument or submission per stepDocuments on finished vendorsDocuments on abandoned requestsCredit cannot be checked
Stalled attemptsKept with last statusRarely beyond a purge windowMost unfinished requestsRemoves the cases partial credit exists for
Approver and authorityNamed approver plus roleAn approver nameWhether the role was allowed"Approved" may not count
Checklist versionDated procedure historyCurrent procedureWhen steps were added or removedOld cases marked down unfairly
Downstream resultLink to first paymentPayment exists in APLink back to the onboardingComplete-looking cases may not be complete

Four gaps, worked through the onboarding

Missing context. If the onboarding request was tracked by email and only the finished vendor record survives, nobody can tell which steps happened in which order, or which attempts stalled. The stalled ones are often the most informative, because they show where work breaks down, and they are the first to disappear. An archive of finished vendors can still support pass-or-fail testing but not partial credit.

Missing permissions. The final authorization step counts only if the right person approved. If approvals were made under a shared login, or a clerk approved when the policy required the controller, the step cannot be scored as done even though the workflow shows "approved." Vendor bank detail changes are especially sensitive here; see vendor bank account change verification.

Versions that moved. A company that added an insurance requirement last year has two checkpoint lists. Scoring older onboardings against the newer list marks them down for a step that did not yet exist. A dated procedure history prevents that, and even an approximate date ("the insurance step started in spring 2025") is far better than none.

Unrecorded final outcomes. The vendor was approved, but was it ever paid without a problem? If the first payment bounced because the bank details were wrong, the onboarding looked complete and was not. Linking onboarding to first payment closes that gap.

Saying it plainly in an introduction note

Name each gap, its scope and who told you. Illustrative wording, all fictional:

GapNote that helpsNote that misleads
Missing context"Rejected requests purged after 90 days; finished onboardings kept since 2021""All onboardings are in the system"
Missing permissions"Controller says two AP clerks shared one approver login until 2023""Approvals follow policy"
Versions that moved"Insurance step added around spring 2025; no change log""The procedure has not really changed"
Final outcomes"First payments carry the vendor ID, but bounced payments are not flagged""Finance tracks payment problems"

Admitting a gap costs nothing at this stage. Discovering one in review that the note glossed over costs time and trust.

What only a review can measure

How many attempts carry every step record, how many stalled requests survive, whether workflow history came through an ERP migration, how tax identifiers and bank details would be removed, and whether any buyer wants step-scored workflows are settled in SourceX's review under an agreement with the company. Partners stay at metadata level and never handle vendor files.

Finance, procurement and IT usually share these records, and the CEO authorizes a licensing discussion. The company needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English.

For the related case of grading quality rather than steps, see rubric-based evaluation; for why developers need test material at all, what model evaluation is. Rate the company first with the partial-credit evaluation qualification scorecard; after an introduction it prepares the partial-credit evaluation review packet.

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the gaps leave most steps recorded with an approver, register as a partner, confirm the basics on who qualifies, and read the terms before you introduce the CFO or COO with your gap notes attached.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Why do gaps matter more for partial credit than for pass or fail?

A pass-or-fail score needs only the final state. Partial credit needs every step on its own, so a gap in any one step removes the ability to score that step, and a missing stalled attempt removes the very cases where partial credit means something.

Is a workflow that keeps only finished onboardings worthless?

No. It can still support pass-or-fail testing, and finished cases still show the order and timing of steps. What it cannot show is how far unfinished attempts got.

Should the company re-enter missing approvals from memory?

No. Approvals written in after the fact are not records of what happened. State the gap and the date from which approvals were captured in the tool.

Sources

  1. arxiv.org/abs/2412.14161
  2. learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/v…
  3. learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/v…
  4. irs.gov/forms-pubs/about-form-w-9
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment