Completed
Foundation + Eval Harness
Different open models can be compared on the same tasks and a reproducible report can be generated.
Published report: Minimal Open-Model Eval Harness
Open Model Lab
Completed dates stay fixed; future stages follow actual model/data prerequisites. Development and local compute proceed independently, without a replacement finish-date promise.
Different open models can be compared on the same tasks and a reproducible report can be generated.
Published report: Minimal Open-Model Eval Harness
Repeated runs reproduce their scores, and per-category 95% confidence intervals are ±5 points or narrower.
The work does not stop at fine-tuning; it shows measurable behavior changes, including regressions, beyond eval suite v2's confidence intervals.
The report clearly shows where post-training helps and where it creates risk.
Agent behavior is measured step by step with failure modes, not only success/failure.
A reliable evaluation system categorizes why the agent fails.
The model's behavior is measured on both risky and harmless requests.
An external reader can reproduce one stage's main result from the repository alone.
The model's ability to understand visual interfaces is measured task-by-task.
The effect of data quality on behavior and compute efficiency is quantified.
The main bottlenecks slowing model research infrastructure can be measured and improved.
The system can show flawed processes despite correct answers, or identify breakpoints leading to wrong answers.
Behavior changes can be tracked not only from outputs but also from model-internal measurements.
An open-source project demonstrates an end-to-end open-model research-engineering loop.