Partial-credit evaluation: scoring a vendor onboarding that collected every document but missed final approval

Short answer

Partial-credit evaluation scores a multi-step task by how far an attempt got, not only by whether it finished. In an ordinary business workflow, it separates a vendor onboarding that collected the tax form, insurance certificate and bank details but never got the controller's approval from one that did nothing at all. For a referral, what matters is whether the company records each step separately, with its own date and approver, and who can authorize a licensing discussion.

Visual briefFull-size diagram
Four onboarding checkpoints, three passed and controller approval missing, scored three ways: pass or fail gives 0%, the share of checkpoints gives 75%, and a weighted partial score that adds a bonus only for finishing gives 37.5%.
Illustrative onboarding at a fictional company. The weighted formula follows the one published for TheAgentCompany benchmark.

What does partial-credit evaluation mean in an ordinary business workflow?

Partial-credit evaluation scores a task by its steps. Instead of a single pass or fail, the task is split into checkpoints, and an attempt earns credit for each checkpoint it completes, usually with extra weight for finishing the whole thing.

The idea matters to AI developers because most real work is long and fails late. A software agent that gathers every document for a new supplier and then forgets to route the file for approval has done most of the work and still left the company unable to pay that supplier. A pass-or-fail score treats it the same as an agent that did nothing. A step-by-step score shows exactly where it stopped. Our explainer on what model evaluation is covers why developers need test material like this, and the guide to AI agent benchmarks shows how published tests are built.

Research benchmarks already publish the arithmetic. TheAgentCompany, a benchmark of workplace tasks, scores each task with checkpoints. Its full completion score is 1 only if every checkpoint passes. Its partial completion score is half the share of checkpoint points earned plus half the full completion score, so an attempt that finishes earns an extra 50% that an unfinished one cannot. Business processes with recorded, separately approved steps are the raw material for scoring of this kind.

Scoring one vendor onboarding three ways (Illustrative)

This case is fictional. A 210-person facilities services company adds a new HVAC subcontractor. Its onboarding procedure has four checkpoints, each worth one point, and the company runs Dynamics 365 Finance with vendor approvals switched on.

CheckpointWhat counts as doneRecord it leavesAttempt status
1. Tax formSigned Form W-9 with legal name and taxpayer identification number on fileDocument attached to the vendor requestDone
2. InsuranceCertificate of insurance on file, valid on the start dateDocument with expiry dateDone
3. Bank detailsAccount submitted to the bank account approval workflowWorkflow entry with statusDone
4. Final authorizationController approves the proposed vendor record, which makes the vendor activeWorkflow approval with approver and timeNot done

The IRS describes Form W-9 as the way a payee gives a payer its correct taxpayer identification number for information returns, which is why it is the first checkpoint. Checkpoint 4 is where the process actually takes effect. Microsoft's documentation for the vendor approval workflow says edits to approval-flagged fields are held as proposed changes, and the proposed values are written to the vendor record only after every approval step is complete. Bank details have their own separate workflow, in which approvers can reject, delegate or cancel. So after this attempt, three documents sit in the request and the vendor still cannot be paid.

Here is the same attempt scored three ways:

Scoring ruleArithmeticScore
Pass or failNot every checkpoint passed0%
Share of checkpoints3 of 4 checkpoints75%
Weighted partial (TheAgentCompany formula)0.5 × 3/4 + 0.5 × 037.5%
Weighted partial, if the controller had approved0.5 × 4/4 + 0.5 × 1100%

The share-of-checkpoints row overstates progress, since the one missing step is the one that matters. The pass-or-fail row hides the progress entirely. The weighted rule sits between them and keeps a finished onboarding far ahead of an unfinished one. Which rule a buyer uses is the buyer's choice. What a company's records must supply is the evidence to apply any of them: each checkpoint recorded separately, with a date and an approver.

Evidence fields each checkpoint needs

A checkpoint can only earn credit if a reviewer can tell, from the record alone, that it happened, when, and on whose authority. For the onboarding above, that means four fields per checkpoint plus one for the task as a whole.

FieldWhy the score depends on itOnboarding example (Illustrative)
Checkpoint identityCredit is assigned per step, so steps must be distinguishableTax form, insurance, bank details, final authorization
Completion evidenceA step counts only if its output existsW-9 attached, certificate attached, bank workflow entry
TimestampOrder and timing matter when a later step depends on an earlier oneCertificate valid on the start date
Actor and approverShows the step was done by someone allowed to do itController named on the approval, or missing
Final state of the taskDecides the finishing bonusVendor proposed, not active

The fifth field is the one most often lost. When a request is abandoned, some systems delete it or leave it in a draft state that reports never show. An archive that keeps only finished onboardings can be scored pass or fail, but not for partial credit, because every attempt in it passed.

One example cannot show how many attempts stalled, how many were later completed under a new request, or whether stalled requests were purged. Those are archive-level questions that SourceX answers in its review with the company.

Does the company qualify, and what stays unknown?

The company needs 50 or more full-time employees at peak (contractors excluded), documented operating history, rights to license its records, and an authorized contact. Companies in the U.S., Canada and the EU qualify, as do companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when at least 70% of the data is in English. The company qualification checklist covers the basics.

Unknown until SourceX reviews it: how many onboardings carry complete step records, how tax identifiers and bank details are removed or de-identified, whether software vendor terms limit use, and whether any buyer wants this task type. Partners never request exports, screenshots or sample files. Related concepts are in the data types hub.

Go deeper on partial-credit evaluation

Next step

Refer a company. No cap on rewards. Default reward on a company's first qualifying transaction: the greater of $50,000 plus 5% and 15% of Actual Supplier Payout, with no cap; further qualifying transactions first agreed within 12 months of the first Closing pay 15%. Eligibility and payment conditions apply, including the USD 100,000 Minimum Transaction, and a signed Schedule A or addendum can differ. A lead, meeting or signature alone does not trigger payment.

If the controller confirms that each onboarding step is recorded with its own date and approver and that stalled requests stay in the system, register as a partner, check the company against who qualifies, and read the terms before you introduce the CFO or COO.

  1. Step 1Share your linkSend your personal link to a company you know.
  2. Step 2Company appliesThe company applies itself at /apply.
  3. Step 3Company signs and is paidThe company signs, delivers its data and is paid by SourceX under its agreement.
  4. Step 4You get your rewardthe greater of $50,000 plus 5% and 15% of the actual supplier payout, with no cap. A reward becomes payable after SourceX has paid the referred company under its signed agreement and the payout conditions in the program terms are met, including finance approval and your tax and payment details.

Common questions

Is partial credit the same as a rubric score?

They overlap. A rubric grades the quality of an output against criteria; partial credit grades progress through the required steps of a task. Many evaluations use both: checkpoints for whether each step happened, a rubric for how well.

Why would anyone give credit to a task that did not finish?

Because a score of zero hides the difference between an attempt that did nothing and one that stopped one step short. Partial credit keeps that difference visible, while a finishing bonus keeps the unfinished attempt clearly below a finished one.

Should I ask the client for an onboarding file to see whether its records are complete?

No. Partners never handle vendor or employee records. Ask how many onboardings happen, which systems hold each step, how long workflow history is kept and who owns it.

Sources

  1. arxiv.org/abs/2412.14161
  2. learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/v…
  3. learn.microsoft.com/en-us/dynamics365/finance/accounts-payable/v…
  4. irs.gov/forms-pubs/about-form-w-9
  5. refer.sourcex.si/terms

Free resources

Know a company with valuable proprietary data?

Become a referral partner, get your link and introduce the company’s CEO (in an owner-run company, usually the owner). Referral partners can join from most countries. We can't work with partners in countries or regions under U.S. sanctions or on U.S. restricted-party lists. The companies you introduce should be based in the U.S., Canada and the EU, or in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia with at least 70% of their data in English.

Refer a company →

I own a business

Explore licensing your company's data directly to SourceX for AI training and evaluation. Start a short assessment; no uploads needed.

Start an assessment