Open Model Lab
Minimal Open-Model Eval Harness
Can three open models be evaluated on the same small task suite with reproducible score, cost, latency, and failure-mode reporting?
Status
- Status
- published
- Month/theme
- July 2026: Foundation + Eval Harness
Status: Published. The July gate completed the first end-to-end Open Model Research Harness workflow: 25 versioned tasks, three comparable local model runs, 75 validated scored outputs, deterministic reporting, a manual failure audit, and a frozen baseline for August SFT work.
Research question
Can three open models be evaluated on the same small task suite with reproducible score, cost, latency, and failure-mode reporting?
Recorded setup
- A versioned 25-task suite with five tasks in each of five categories.
- Three local Ollama model artifacts run with one shared configuration.
- Temperature 0.0, maximum 1024 output tokens, seed 42, and 300-second timeout.
Recorded measurements
- Task-level score, pass state, grader metadata, and failure mode.
- Latency, input/output token counts, and local-runtime throughput.
- Manual audit labels kept separate from the recorded grader outcomes.
Report coverage
- Harness architecture and validation path
- Versioned evaluation suite and shared run configuration
- Three-model diagnostic comparison
- Manual audit of all recorded failures
- Known limitations and August handoff
Published artifacts
- Final July evaluation dataset and model/run configurations.
- Three validated 25-result local runs and deterministic comparison.
- Failure audit, milestone report, and frozen baseline manifest.
July diagnostic results
| Model | Passed | Failed | Pass rate | Mean score |
|---|---|---|---|---|
| lfm2.5-8b | 12 | 13 | 48.0% | 0.452 |
| gpt-oss | 23 | 2 | 92.0% | 0.920 |
| qwen3.6 | 12 | 13 | 48.0% | 0.540 |
These outcomes apply only to the frozen 25-task July suite. They are not a general model ranking.
Failure audit
All 28 recorded failed rows were reviewed without changing the original grader outcomes.
| Audit label | Count |
|---|---|
| Likely format or instruction failure | 11 |
| Likely grader false negative | 7 |
| Truncation or runtime related | 10 |
| Likely substantive model failure | 0 |
Key findings
- gpt-oss recorded the highest pass rate within this specific suite at 92.0%; this is not a general model ranking.
- The audit classified 11 failed rows as likely format or instruction failures, 7 as likely grader false negatives, and 10 as truncation or runtime related.
- Visible thinking output, literal grader behavior, code extraction, and the fixed 1024-token limit materially affected recorded outcomes.
- The July scores remain frozen even where the audit found likely measurement artifacts; future fixes require a new evaluation identity.
Claim boundary
This report validates a small diagnostic harness. Its 25 draft tasks do not establish general model quality, a leaderboard, or statistical significance.