Open Model Research Harness completes its July evaluation gate

July 29, 2026 Note 5 min read

Open Model Research Harness has completed its July 2026 Foundation + Eval Harness gate.

The result is a small, reproducible evaluation pipeline that can run local open models on the same versioned task suite, select deterministic graders, record task-level scores and failure modes, validate the resulting artifacts, and produce a comparable report. It is not a general benchmark or a model leaderboard. It is the baseline I need before changing model behavior in the next stage.

ItemResultStatusMeaning

Evaluation suite25 tasks, 5 categoriesDoneAll three models used the same versioned task set.

Final model runs3 local Ollama modelsDoneEach model produced 25 scored results under one shared configuration.

Scored outputs75 total, 0 grader errorsDoneThe complete run set passed artifact validation.

Failure audit28 failed rows reviewedDoneRecorded failures were inspected instead of treated as self-explanatory scores.

July baselineDataset, configs, graders, reportsDoneThe baseline is frozen before later evaluation changes.

Reuse termsApache-2.0 + CC BY 4.0DoneCode and original research materials now have explicit licensing boundaries.

SFT + data qualityAugust gatePlannedFuture behavior changes will be measured against the July baseline.

What the harness now does

The final July suite contains five tasks in each of five categories: coding, reasoning, factuality, instruction following, and safety-lite. Four grader paths evaluate them: executable unit tests, normalized exact match, explicit rule checks, and weighted textual rubrics.

The runner records the model artifact, prompt, output, latency, token counts, grader decision, score, confidence, and failure mode for every task. Validators then check that the dataset and result files are structurally complete before compatible runs can enter the deterministic comparison.

The three final runs used lfm2.5-8b, gpt-oss, and qwen3.6 through Ollama with the same 25 tasks, temperature 0.0, maximum output length of 1024 tokens, seed 42, and a 300-second timeout.

Recorded results

ModelPassedFailedPass rateMean score

lfm2.5-8b121348.0%0.452

gpt-oss23292.0%0.920

qwen3.6121348.0%0.540

Within this specific suite, gpt-oss recorded the highest pass rate. That sentence is deliberately narrow. Twenty-five draft tasks on one local machine do not establish general model quality, and the raw scores became less straightforward as soon as I inspected the failed outputs.

The useful result was in the failures

The first comparison contained 28 failed result rows. I reviewed every one against the task definition, complete model output, grader configuration, output token count, and runtime completion state.

The audit classified 11 as likely format or instruction failures, 7 as likely grader false negatives, and 10 as truncation or runtime related. None were labeled as likely substantive model failures under this audit. That does not mean the models made no mistakes. It means this small run exposed measurement and protocol problems before it could support a stronger model-quality claim.

The failure patterns were concrete:

lfm2.5-8b exposed visible thinking before final answers. That extra text broke strict answer formats, and the code extractor missed three usable implementations that followed the thinking block.
One gpt-oss response returned the correct JSON inside a Markdown fence, which correctly failed a bare-JSON instruction. Its other recorded failure was likely a literal-rubric false negative sensitive to apostrophe form.
Ten qwen3.6 failures reached the fixed 1024-token limit. Several responses left generated content in Ollama thinking metadata while the final answer was empty or incomplete.

This changed what I consider the first milestone. Producing a score table is not enough. An evaluation harness also has to preserve raw evidence, distinguish model behavior from runtime behavior, make grader limitations visible, and avoid silently repairing inconvenient results.

Freezing an imperfect baseline

I froze the July dataset, model and run configurations, grader implementations, validators, comparison logic, reports, and local run identities as the july_2026 baseline. The recorded scores remain unchanged even where the audit found likely false negatives or token-limit effects.

That imperfection is intentional. If I improve code extraction, adjust generation limits, change a rubric, or replace a mutable model artifact later, the new evaluation must get a new identity. Rewriting the July results would make the August comparison cleaner and the research record weaker.

Making the reuse terms explicit

A reproducible baseline also needs clear terms for reuse. The repository now licenses its software, scripts, tests, configuration files, and schemas under the Apache License 2.0. Original datasets and documentation, including the material under datasets/, docs/, reports/, and releases/, are available under CC BY 4.0.

Those licenses do not relicense model outputs, model weights, third-party materials, trademarks, or content with separate terms. Those remain governed by their respective sources. The repository’s NOTICE records that boundary in one place.

The next gate is supervised fine-tuning and data quality. The question is no longer whether I can run three models and collect numbers. It is whether a controlled SFT experiment produces measurable improvements and regressions against a baseline whose limitations are already known.

The canonical project description and roadmap live on the Open Model Research Harness project page. Open Model Lab has the completed July gate, published report, run summaries, and dataset record. The implementation, frozen baseline, comparison source, and failure audit are in the public repository.