The Work-Test Cycle

ยท 10 min read
talos testing process release engineering

In short. Talos, a local coding assistant written in Java, is built with two loops. An edit loop runs while code changes and ends with the required local gates, and a release gate runs only for a candidate. The procedure asks for one short record per coherent batch, and the records hold the exact commands, source hashes and results of each run. The examples below come from the author's unpublished records, and the procedure they follow is new in its current form.

A green suite, and 132 failures

A Windows CI run of Talos failed 132 of 13,254 unit tests. The end-to-end, coverage and final check steps never ran. The local suite was green, and none of the 132 was a product defect.

The GitHub runner's temporary folder is an 8.3 short-name alias, C:\Users\RUNNER~1\..., and the checkout sits on drive D:. 113 tests had bound a workspace at the unresolved temporary path while the product resolves real paths. 16 used fake model-artifact identities that returned the configured path where the product records the real one. One test expected a configured model path where the model library stores the real one. One relativized a C: path against a D: working directory. One timing test exceeded a three-second wall-clock limit on the runner.

The repair took a day and weakened no product check. Its one production change was a package-private counter in one class, read only by a test. A local lane was built to reproduce the runner, with a short-name temporary folder and the project on another drive. Red runs showed which tests failed for the intended reasons, and showed that under a short-name root two alias tests passed only because a different operand was rejected first. A new test pinned a refusal that no test had referenced before. A deliberate mutant, the refusal disabled for one run, proved the new test fails without it. The timing test became a count assertion with its own mutant. The final local full check ran 14,198 tests, 13,255 unit and 943 end-to-end, with 0 failures and the same 19 skips as the previous accepted run, and a manual CI run on the fixed commit passed both jobs.

A green local suite was evidence for one filesystem layout. Several rules of the process exist to keep that kind of distinction explicit.

Two loops

The edit loop and the release gate The edit loop states scope and invariants, reproduces the failure at the real boundary, fixes it with focused tests, runs adjacent and end-to-end tests, repeats for more changes, runs the full check locally before the batch is accepted, and ends with a batch record and review. The release gate runs only for an authorized candidate. The hard quality gate is a local task. Release staging runs the exact-commit check, builds the SBOM and attests the artifacts in CI. Download verification, the installed-product smoke test from the staged artifact, terminal evidence, the two-model live audit and publishing are not run by the build. A failed candidate is fixed, committed and staged again. EDIT LOOP scope and invariants stated before editing reproduce at the boundary the real one where possible fix and focused tests cause checked on each red adjacent and end-to-end before scope widens full check, six gates local, before acceptance batch record and review then the batch is accepted RELEASE GATE, ONLY FOR A CANDIDATE candidate authorized hard quality gate local maintainer packet release staging check, SBOM, attest verify the download installed-product smoke from the staged artifact real terminal evidence two-model live audit when model behavior is claimed publish on failure, fix, commit, stage again runs in CI a local Gradle task not run by the build
Figure 1. The edit loop and the release gate, with each step marked as run in CI, run by a local Gradle task, or not run by the build.

The maintained procedure, work-cycle-docs/work-test-cycle.md, separates "an inner development loop and a Release QA Gate". A full release packet "is reserved for candidate acceptance, not repeated after every edit."

The inner loop has seven steps.

  1. Branch, version and working-tree state are confirmed.
  2. The objective, preserved invariants, affected boundaries, acceptance checks and stop conditions are stated briefly.
  3. Source and callers are inspected, and the failure is reproduced through the real affected boundary where possible.
  4. A coherent correction and focused regression coverage are written.
  5. Focused checks and git diff --check run. The intended cause of each failed assertion is checked before its expectation is changed.
  6. Adjacent integration and relevant deterministic end-to-end checks run before scope widens, especially for authority, persistence and shared policy.
  7. Required local gates and independent review proportional to risk complete before the batch is accepted.

The release gate starts only when a candidate is authorized. It requires a full automated gate on the exact candidate commit, staged artifacts with checksums, an SBOM and attestations, a smoke test of the product installed from the candidate artifact, manual evidence from a real terminal session, and a live audit on two models when model behavior is part of the release claim. The release runbook stages the artifacts first and runs the product checks on what was staged. Failed staging evidence is not repaired in place. The product or process is fixed, the fix is committed, and staging runs again from the new commit.

What one check holds

Full Gradle check is required before a batch that touches runtime behavior, release wiring, public documentation contracts or shared policy is accepted. It holds six gates.

In these records the passing full checks executed between 11,578 and 13,498 unit tests and between 932 and 943 end-to-end tests, and took from about 1 hour 5 minutes to 2 hours 38 minutes on one local machine. The longest run was slowed by another process.

Distinctions kept apart

The procedure and the local working guidance name several things that look alike and are kept apart.

A focused pass is not acceptance. In one batch a new final-status contract passed its focused and adjacent tests. The required full check then failed 4 of 12,500 unit cases. A cause check found three stale exact-output expectations and one structural assertion pointing at an old location, all consistent with the accepted contract, so those expectations were updated. The rerun completed 12,501 unit cases with no failures and the nineteen existing skips, then failed three end-to-end checks, because a qualification reader rejected the new status event. That run was recorded as "an incomplete failed check, not full acceptance." Only the reader and its fixture were fixed, a regression class covering sixteen malformed or misplaced status events was added, and a fresh full check passed.

An exit code is not a result. The build has reporting lanes, candidateTest and candidateE2eTest, that run with ignoreFailures = true so results are preserved when tests fail. Those lanes can finish with exit code zero while failures remain, and the procedure states that "reporting-oriented candidate test tasks can finish with exit code zero while test failures remain." A separate task, qualityReportGate, turns their summaries into a hard gate. It requires zero failures and errors, instruction coverage of at least 82 percent, zero skipped end-to-end tests, every JSON scenario passed, a Qodana result bound to the current source with no critical issue, and an existing jar.

A passing subset is not a release verdict. A completed offline batch is kept distinct from installed-product, live-model and exact-candidate acceptance. For document-comparison lanes, the live audit runbook adds that "A passing subset is not evidence that either profile or the candidate has passed the full audit."

Review agreement is not execution evidence. In the local working guidance, a review, including one by an automated agent, does not count as execution evidence. In the CI repair and the drive-letter batch described below, code changed after a review finding was run again before the final full check.

Execution success is not answer correctness. The current local guidance adds that runtime execution and verification facts do not show that every requested result was answered correctly.

Red first, then a mutant

A green test proves little until it has been seen to fail for the right reason. Several of the later batch records follow one shape. A red run comes before the fix, with the failures checked against their intended causes, then the fix, then a deliberate mutant that disables the new behavior and must turn the test red again. No mutation-testing plugin is involved. The mutants are edits applied to the source for a single run, and the source is restored and checked by hash afterwards.

Review can also catch the fix, and an invalid run is not counted. Another batch fixed Windows paths that lost their drive letter in the request contract, so a target named with a drive could point at the wrong place. A temporary characterization test recorded the behavior first and was then removed, with its output kept. The plan allowed a second step that mapped absolute targets inside the workspace to relative ones. Review rejected it, because it broke the preparation of forbidden targets, and it was removed before acceptance. One red run was marked invalid because the restored file did not compile, and it was not counted as evidence. The full check passed 14,225 tests, 13,282 unit and 943 end-to-end, and the Windows CI run on the commit took 111.4 minutes, beyond the earlier 90-minute limit.

One record per batch

Each coherent outcome gets one short record with five fields, which are the objective, the invariants and boundaries, the evidence, what was assumed or not tested, and the next action. A condensed version of the CI repair record, reconstructed from its plan and analysis files, shows the shape.

Objective     Windows CI green again, with no product check weakened.
Invariants    No alias, link, protected-path or artifact identity check relaxed.
Boundaries    Fixtures in twelve test classes, one test helper, one test counter.
Evidence      red001     failures for the intended causes
              mutation002  refusal disabled in the source, new test fails
              full001    14,198 tests, 0 failures, same 19 skips
              CI run     both jobs passed on the fixed commit
Not tested    Starting Talos at a drive root, likely failing, not run end to end.
Next          Release staging still has a 90-minute limit. Open.

Evidence identity is concrete. A run record holds the full command and arguments, the Java home, the commit and a SHA-256 for every source file. A result record is parsed from the JUnit XML rather than from console output and carries a hash of the log. Runs go to fresh labelled output directories with an isolated home directory per test task, and the run scripts refuse an existing directory, so an earlier result is not overwritten. Failed runs and closed reports are preserved, and current status is updated separately.

The release preparation folder holds about 300 folders, a few of them for work after the code described here. Not every folder is an accepted batch. Some are reviews, designs, investigations or failed attempts kept on purpose.

What is automated, and what is not

The procedure is more complete than its automation.

Where the cycle fell short

The opening example contains a lapse. CI ran only on pushes to the main and beta branches and on pull requests into them, and the work stayed on a feature branch with no such run. No CI run was made for over three months, so 132 unit-test failures accumulated unseen, and the end-to-end suite had not run on a runner in that time. Manual dispatch for any pushed branch has since been added, and the Windows job's timeout was raised from 90 to 180 minutes, with a contract test that fails on the old value. The release staging job still has a 90-minute limit, recorded as open.

Other limits follow from the repository and the records. Linux coverage in CI is a subset. The batch-record format is a practice, and no tool enforces it. About one folder in sixteen has no analysis file, though most of those hold a plan, a closeout or raw outputs instead. Keeping every run also has a cost. The folder holds about 5.7 million files and 104 GB. The current candidate path, including qualityReportGate, has not been run end to end. The last documented release QA packet is the local 0.10.8 packet, made under the earlier process before that gate existed. Release readiness is recorded as unestablished.

What transfers

The cycle is specific to one Java codebase. The following parts do not depend on it.