Test suites as AI training data: why test cases make code changes verifiable

Why do test suites make code verifiable for AI training?

A test case is an objective check: a change either passes it or it does not. Test suites linked to the code changes and bug fixes they verify let an AI developer ask whether a model-generated change actually works, instead of guessing from how the code reads. That pass-or-fail signal is scarce in public data and common inside software companies.

For a fractional chief technology officer (CTO), this is an everyday asset. You know which clients have a real test culture, which kept manual test cases in a management tool for years, and which are about to retire that tool.

This page covers what test data a software company holds, how to tell a strong suite from a weak one, and how to raise it before introducing the company to SourceX.

What counts as test data?

A suite whose tests reference tickets and commits is worth more than a pile of scripts. The linking is what lets a buyer connect a fix to the test that proves it. History also matters: see the page on why git history matters more than a snapshot for how the repository side fits.

Where does the data live?

Common locations include:

• The source repository, in test directories.
• A continuous integration system with build and test logs.
• A test management tool holding manual cases, plans and run results.
• An issue tracker with defects and links to tests and commits.
• Spreadsheets used by QA teams before a tool was adopted.
• Release notes and sign-off records.

Retention is the issue: CI systems often purge old logs, and test management tools may be retired. Ask how far back results go and who owns exports. The company's own vendor documentation states what export options exist; this page makes no claim about any particular tool.

The verifiability screen

Use these questions in a first conversation. They ask about structure, not contents.

• Linkage: do tests, defects and code changes reference each other by ticket or commit?
• History: are results kept for several years, or only the last few builds?
• Coverage: is there a meaningful number of tests in the areas that matter, not just a smoke suite?
• Outcome labels: are defects marked fixed, rejected or deferred, with reasons?
• Ownership: did employees write the tests under assignment terms, and are they free of client-owned code?
• Access: can someone export test cases and run history?

A company with 50+ full-time employees at peak (contractors excluded), several years of documented operations and an authorized sponsor, which passes most of these, is a candidate. Start with the company fit checker, a preliminary non-binding screen.

When should a fractional CTO raise it?

Record types in neighboring parts of the stack share this shape. Compare architecture decision records, which capture design reasoning where tests capture verification.

What rights and privacy points apply?

Tests can carry more than logic:

• Test fixtures sometimes contain real customer data copied from production; those need removal or redaction.
• Manual cases may quote customer names or contract details.
• Tests for client-owned software, written by an agency, may belong to the client.
• Third-party test libraries and open-source components carry their own terms.
• Credentials and tokens occasionally appear in test configuration.

Redaction and de-identification requirements are agreed with the company before any work begins. The PII redaction matrix lists typical treatments by record type. If the company mainly writes tests for clients' systems without consent, treat it as a red flag.

How the introduction works

1. Register and pass your referral link to the founder or VP of engineering, or file basic fit details through the referral form.
2. SourceX screens size, history, breadth of systems and rights.
3. The company lists its repositories, CI, test tool and tracker with years of coverage; the data inventory builder can help organize that.
4. Price and terms are agreed before any buyer sees the opportunity; nothing binds until the company signs.
5. Buyers typically respond within about two weeks of the company being deal-ready.
6. After signature, fixtures and credentials are cleaned under the agreed redaction rules and the data is delivered; the company is paid once.
7. Your reward is released after SourceX receives its fee.

You never open a test file or run an export. For a related support-side record, see KCS article histories.

What to say to a software founder

How rewards work

Partners earn 25% of the eligible platform fees SourceX actually collects from the referred company's licensing deals, capped at $100,000 per referred company. The reward is paid only after the buyer pays and SourceX receives its fee; an introduction, meeting or signed agreement alone does not trigger payment, and no reward is guaranteed. The reward is a share of SourceX's fee and is never deducted from what the company receives. Check your client agreements and any disclosure rules before registering; read about the role at referral opportunities for fractional CTOs.

When it is not a fit

Pass when tests exist only as a few scripts without history, when the archive is wiped after each release, when the code and tests belong to clients, or when the company is below the size baseline. A client with deep CI logs but no linked tracker may still qualify through other systems; it simply scores lower on this screen.

Next step

Ask one software client how many years of test results could be exported tomorrow. If the answer is several, register as a partner and introduce them, or have the sponsor apply at sourcex.si/apply.