Open Model Lab
July 2026: Foundation + Eval Harness
Build the basic research infrastructure that can measure model behavior reliably before changing it.
Gate status
- Month
- 2026-07
- Status
- completed
Success criterion
Different open models can be compared on the same tasks and a reproducible report can be generated.
Execution summary
Completed on 2026-07-29 with a frozen 25-task diagnostic suite, three comparable local model runs, 75 validated scored outputs, deterministic comparison reporting, and a manual audit of all 28 recorded failures.
Completed focus
- Run three local open models on the same versioned 25-task suite.
- Cover coding, reasoning, factuality, instruction following, and safety-lite behavior.
- Grade outputs through executable unit tests, normalized exact match, explicit rules, and weighted textual rubrics.
- Record task-level scores, latency, token usage, output quality, and failure modes.
Completed outputs
- Versioned 25-task July evaluation suite across five categories.
- Config-driven Ollama runs for lfm2.5-8b, gpt-oss, and qwen3.6.
- Four deterministic grader paths with dataset and run validation.
- Model comparison, failure audit, milestone report, and frozen baseline manifest.
- Explicit reuse terms: Apache-2.0 for software and CC BY 4.0 for original datasets and documentation, with model outputs, weights, and third-party materials excluded.
End-of-month decision
Proceed to the August SFT and data-quality gate using the frozen July baseline as the unchanged comparison point. Any grader, extraction, generation-limit, or model-artifact change must create a new evaluation identity rather than rewrite the July results.