GCTX: the first full model training and what followed

On September 22, 2026, GCTX completed its first full language-model training run: 93 million parameters, initialized from scratch, with an 8,192-token context.

The run trained on 10,296 DEV examples in about three and a half hours, then evaluated all 1,627 REPORT examples. This was the model-training milestone that the June data-readiness update had been preparing for.

The training completed. The model failed the usefulness test.

Every REPORT output used the same type and scope: refactor(vite), regardless of the source repository. The messages passed the format check, but the model had not learned to describe the actual changes reliably.

That result started a series of smaller experiments.

The input pipeline had concrete problems: lossy tokenization, inconsistent instructions, and long diffs that needed explicit handling. The revised pipeline now reconstructs all 10,299 DEV records exactly and preserves longer changes through multiple input windows.

The review process also needed more precise language. Historical labels had passed automated acceptance checks; that did not establish independent human factual review. Later assistant screening against original diffs found errors that those checks had missed. A separate audit found one training/REPORT duplicate, now quarantined for subsequent selection. Later diagnostics stayed within DEV.

A new training run reduced validation loss by 51.25%, yet scored 0/64 on the fixed panel measuring whether messages described the principal change correctly. It stopped after the first epoch under its predefined quality rule.

The next question was whether the model could reliably learn a small, checked subset of its own training examples.

On October 4, a controlled comparison tested three learning rates from identical model and optimizer states, using the same 56 examples and equal training budgets:

Both lower rates learned all 56 training messages. The separate DEV panel remained unsuccessful. Usability judgments came from assistant screening against source diffs, not independent human evaluation.

GCTX can now fit that small set reliably. Useful generalization remains open.

The next planned experiment compares 56 versus 512 examples checked against their source diffs, with a common starting checkpoint and equal update budgets. It will measure whether broader training coverage improves results across 256 DEV validation cases. Those cases have already been used during development; they are not an untouched test set.

The design is ready. Preparing the larger subset and running the comparison are still ahead.

Full current status lives on the GCTX project page.