Open Model Research Harness
A 12-month public research-engineering project for reproducible open-model evals, post-training, agents, safety, monitorability, and systems efficiency.
Actively developing. Not usable yet.
Current status
July evaluation gate completed
The first end-to-end evaluation baseline is frozen: 25 versioned tasks, three comparable local model runs, 75 validated scored outputs, deterministic reporting, and a manual audit of all 28 recorded failures.
- Latest public artifact
- July 2026 evaluation baseline ↗ Jul 29, 2026
- Next checkpoint
- Begin the August SFT and data-quality gate, measuring improvements and regressions against the unchanged July baseline.
- ✓July 2026: Foundation + Eval Harness — 25 tasks, three local model runs, 75 validated scored outputs, deterministic comparison, failure audit, and frozen baseline
- August 2026: SFT Pipeline + Data Quality — planned
- September 2026: Preference Optimization / DPO — planned
- October 2026: Agent Harness / Tool Use — planned
- November 2026: Agent Evals + Long-Horizon Tasks — planned
- December 2026: Safety / Red Teaming / Refusal Quality — planned
- January 2027: Reasoning Behavior + Process Evaluation — planned
- February 2027: Monitorability / Interpretability Start — planned
- March 2027: Multimodal / UI Understanding / Computer Use — planned
- April 2027: Training / Inference Systems Efficiency — planned
- May 2027: Data Efficiency + Scaling Ladder — planned
- June 2027: Final Integration + Public Portfolio — planned
What it is
Open Model Research Harness is a 12-month public research-engineering project for studying open LLMs through reproducible evaluations, post-training experiments, agentic task harnesses, safety evaluations, monitorability probes, and systems-efficiency measurements.
The goal is not to claim frontier-model capability. The goal is to build and document the eval-first research workflow needed to understand how model behavior changes across SFT, preference optimization, tool use, safety constraints, and inference/training efficiency work.
The project is inspired by frontier-lab research workflows, but intentionally scoped to open models, small experiments, reproducible evaluation, and public reporting.
It is a modular research harness and public portfolio for:
- evals
- SFT
- preference optimization / DPO
- agent harnesses
- long-horizon agent evals
- safety and refusal quality
- reasoning process evaluation
- monitorability / interpretability probes
- multimodal UI understanding
- systems efficiency
- data efficiency and scaling ladders
Why it exists
Open-model work is easy to overclaim when the evaluation loop is weak. A fine-tune, preference run, or agent demo can look useful in isolation while hiding regressions, safety failures, trace-level mistakes, or systems bottlenecks.
This project makes the measuring system the first deliverable. The completed July 2026 gate is not a model release; it is a small reproducible eval harness that compares multiple open models on the same tasks and reports score, local-runtime latency, token usage, output quality, and failure modes.
How it differs
The project is not structured as a leaderboard or benchmark marketing page. It is a public engineering record: each report has a research question, a setup, measurements, backing artifacts, and a claim boundary. The July report is published because its dataset, configurations, runs, comparison, failure audit, and baseline identity exist; later reports and dashboards remain planned until their evidence exists.
The work also connects several layers that are often presented separately: eval design, post-training, agent traces, safety behavior, monitorability probes, and systems profiling. The point is to show how behavior changes across the loop, not to claim that small open models match larger private systems.
What it is not
- It is not a claim that I run a frontier AI lab.
- It is not a leaderboard.
- It is not a benchmark marketing page.
- It is not a claim that small open models match frontier models.
- It is not a large-scale pretraining project.
Research loop
Measure
Measure model behavior before changing it.
Modify
Modify behavior with SFT, DPO, tools, or constraints.
Compare
Compare base/SFT/DPO/agent variants on the same eval suite.
Diagnose
Diagnose failures with taxonomy and traces.
Report
Report results with claim boundaries.
Roadmap
Foundation + Eval Harness
Build the basic research infrastructure that can measure model behavior reliably before changing it.
SFT Pipeline + Data Quality
Measure the behavioral difference between a base model and an instruction-tuned model.
Preference Optimization / DPO
Use preference data to improve SFT behavior in a more controlled way, then measure behavioral side effects.
Agent Harness / Tool Use
Move the model from passive question-answering into a tool-using agent setup.
Agent Evals + Long-Horizon Tasks
Measure agent success and failure taxonomy on multi-step tasks.
Safety / Red Teaming / Refusal Quality
Measure safety behavior by quality and balance, not just refusal rate.
Reasoning Behavior + Process Evaluation
Evaluate not only final answers, but where reasoning processes break down.
Monitorability / Interpretability Start
Start analyzing model behavior with internal signals in addition to external scores.
Multimodal / UI Understanding / Computer Use
Enter multimodal model work through screenshot and UI-understanding tasks.
Training / Inference Systems Efficiency
Add systems and profiling knowledge to make research experiments more efficient.
Data Efficiency + Scaling Ladder
Measure whether better data mixtures produce better behavior with the same compute.
Final Integration + Public Portfolio
Turn the 12-month work into a showable, reproducible, publishable portfolio.
Module map
Eval Harness
Reproducible task execution, grading, metrics, and failure labels.
SFT Pipeline
Instruction data cleaning, supervised fine-tuning, and behavior comparison.
Preference Optimization / DPO
Chosen/rejected pairs, DPO training, and side-effect measurement.
Agent Harness
Tool registry, file operations, test execution, and replayable traces.
Long-Horizon Agent Evals
Multi-step coding tasks and trace-level failure taxonomy.
Safety / Red Teaming
Refusal quality, over-refusal, under-refusal, jailbreak robustness, and safe completion.
Reasoning Process Evaluation
Outcome scoring compared with observable process and self-correction signals.
Monitorability
Internal-signal probes for failure prediction and representation drift.
Multimodal UI Understanding
Screenshot QA, OCR plus reasoning, UI grounding, and visual failure modes.
Systems Efficiency
Latency, throughput, batching, KV cache, quantization, and profiling.
Data Efficiency / Scaling
Data mixtures, filtering, small scaling ladders, and score/GPU-hour.
Final Integration
Reproducible scripts, reports, dashboard, README, and portfolio packaging.
Status
As of 2026-07-29, Open Model Research Harness has completed its July Foundation + Eval Harness gate. The frozen july_2026 baseline contains a versioned 25-task diagnostic suite across five categories, four deterministic grader paths, three comparable local Ollama runs, 75 validated scored outputs, deterministic comparison reporting, and a manual audit of all 28 recorded failures.
Within this specific suite, gpt-oss recorded a 92.0% pass rate; lfm2.5-8b and qwen3.6 each recorded 48.0%. These are frozen diagnostic outcomes, not a general model ranking. The audit found that visible thinking text, literal grader checks, code extraction, and a fixed 1024-token limit materially affected the recorded failures. The original scores remain unchanged so future work has an honest baseline.
The next planned gate is August 2026 SFT Pipeline + Data Quality. It will compare behavior changes and regressions against the frozen July baseline rather than silently revising the first evaluation.
Where it lives
The public lab section is Open Model Lab. The completed July gate, published report, evaluation suite, and run summaries live there. Source code, configurations, reports, and the frozen baseline are in the public repository.