Open Model Research Harness
A 13-stage public research-engineering project for reproducible open-model evals, post-training, agents, safety, monitorability, and systems efficiency.
Actively developing. Not usable yet.
Current status
Local-first; independent development and compute
The evaluation foundation is complete. SFT preparation includes a validated data contract, 200 training examples, 40 validation examples, and a 24-example smoke subset.
- Latest public artifact
- SFT data preparation report ↗ Sep 24, 2026
- Next checkpoint
- Prepare evaluator/local-runner readiness and v2 input identities; validate resumable smoke training, then measure the exact base before production SFT.
- ✓Foundation + Eval Harness — Completed July 29, 2026
- Eval Suite v2 — Gate: versioned suite and validated local runner
- SFT Pipeline + Data Quality — Gate: exact-base v2 scores and smoke update; pilot data prepared
- Preference Optimization / DPO — Gate: chosen SFT checkpoint and preference pairs
- Agent Harness / Tool Use — Gate: usable tool/trace interfaces and sandbox
- Agent Evals + Long-Horizon Tasks — Gate: working agent loop and actual held-out attempts
- Safety / Red Teaming / Refusal Quality — Gate: selected variants and validated rubrics
- Release 1 — Phase 1 Portfolio — Gate: reproducible Phase 1 evidence
- Multimodal / UI Understanding — Gate: selected VLM artifacts and held-out UI tasks
- Data Efficiency + Scaling Ladder — Gate: one shared ladder with versioned inputs
- Training / Inference Systems Efficiency — Gate: comparable real runtime/hardware measurements
- Reasoning Behavior + Process Evaluation — Gate: reasoning checkpoints and actual outcomes
- Monitorability / Interpretability — Gate: selected activations and held-out failure splits
- Final Integration + Public Portfolio — Gate: accepted final artifacts; unrun branches stay pending
What it is
Open Model Research Harness is a 13-stage public research-engineering project for studying open LLMs through reproducible evaluations, post-training experiments, agentic task harnesses, safety evaluations, monitorability probes, and systems-efficiency measurements.
The goal is not to claim frontier-model capability. The goal is to build and document the eval-first research workflow needed to understand how model behavior changes across SFT, preference optimization, tool use, safety constraints, and inference/training efficiency work.
The project is inspired by frontier-lab research workflows, but intentionally scoped to open models, small experiments, reproducible evaluation, and public reporting.
It is a modular research harness and public portfolio for:
- evals
- SFT
- preference optimization / DPO
- agent harnesses
- long-horizon agent evals
- safety and refusal quality
- reasoning process evaluation
- monitorability / interpretability probes
- multimodal UI understanding
- systems efficiency
- data efficiency and scaling ladders
Why it exists
Open-model work is easy to overclaim when the evaluation loop is weak. A fine-tune, preference run, or agent demo can look useful in isolation while hiding regressions, safety failures, trace-level mistakes, or systems bottlenecks.
This project makes the measuring system the first deliverable. The completed July 2026 gate is not a model release; it is a small reproducible eval harness that compares multiple open models on the same tasks and reports score, local-runtime latency, token usage, output quality, and failure modes.
How it differs
The project is not structured as a leaderboard or benchmark marketing page. It is a public engineering record: each report has a research question, a setup, measurements, backing artifacts, and a claim boundary. The July report is published because its dataset, configurations, runs, comparison, failure audit, and baseline identity exist; later reports and dashboards remain planned until their evidence exists.
The work also connects several layers that are often presented separately: eval design, post-training, agent traces, safety behavior, monitorability probes, and systems profiling. The point is to show how behavior changes across the loop, not to claim that small open models match larger private systems.
What it is not
- It is not a claim that I run a frontier AI lab.
- It is not a leaderboard.
- It is not a benchmark marketing page.
- It is not a claim that small open models match frontier models.
- It is not a large-scale pretraining project.
Research loop
Measure
Measure model behavior before changing it.
Modify
Modify behavior with SFT, DPO, tools, or constraints.
Compare
Compare base/SFT/DPO/agent variants on the same eval suite.
Diagnose
Diagnose failures with taxonomy and traces.
Report
Report results with claim boundaries.
Roadmap
Foundation + Eval Harness
Build the basic research infrastructure that can measure model behavior reliably before changing it.
Eval Suite v2
Make the evaluation suite sensitive enough to detect post-training effects before any model is modified.
SFT Pipeline + Data Quality
Measure the behavioral difference between a base model and an instruction-tuned model.
Preference Optimization / DPO
Use preference data to improve SFT behavior in a more controlled way, then measure behavioral side effects.
Agent Harness / Tool Use
Move the model from passive question-answering into a tool-using agent setup.
Agent Evals + Long-Horizon Tasks
Measure agent success and failure taxonomy on multi-step tasks.
Safety / Red Teaming / Refusal Quality
Measure safety behavior by quality and balance, not just refusal rate.
Release 1 — Phase 1 Portfolio
Package Phase 1 as a reproducible, readable portfolio before Phase 2 begins.
Multimodal / UI Understanding / Computer Use
Enter multimodal model work through screenshot and UI-understanding tasks.
Data Efficiency + Scaling Ladder
Measure whether better data mixtures produce better behavior with the same compute.
Training / Inference Systems Efficiency
Add systems and profiling knowledge to make research experiments more efficient.
Reasoning Behavior + Process Evaluation
Evaluate not only final answers, but where reasoning processes break down.
Monitorability / Interpretability Start
Start analyzing model behavior with internal signals in addition to external scores.
Final Integration + Public Portfolio
Turn the 13-stage work into a showable, reproducible, publishable portfolio.
Module map
Eval Harness
Reproducible task execution, grading, metrics, and failure labels.
Eval Suite v2
Versioned benchmark subsets and suitable İzci artifacts, with paired confidence intervals; development proceeds before production runs are ready.
SFT Pipeline
Pilot data prepared; production LoRA training follows exact-base v2 runs and validated local smoke training.
Preference Optimization / DPO
Chosen/rejected pairs, DPO training, and side-effect measurement.
Agent Harness
A coding-agent harness built on İzci's open-source harness: file editing, code execution, test running, and replayable traces.
Long-Horizon Agent Evals
Multi-step coding tasks and trace-level failure taxonomy.
Safety / Red Teaming
Refusal quality, over-refusal, under-refusal, jailbreak robustness, and safe completion.
Release 1
Phase 1 reports, dashboards, and reproducible scripts.
Multimodal UI Understanding
Screenshot QA, OCR plus reasoning, UI grounding, and visual failure modes, including a small project-trained VLM.
Data Efficiency / Scaling
Data mixtures, filtering, small scaling ladders trained from scratch, and score/GPU-hour.
Systems Efficiency
Latency, throughput, batching, KV cache, quantization, and profiling, including a project-built inference engine.
Reasoning Process Evaluation
Outcome scoring compared with observable process and self-correction signals, including models trained with RL on verifiable rewards.
Monitorability
Internal-signal probes for failure prediction and representation drift.
Final Integration
Reproducible scripts, reports, dashboard, README, and portfolio packaging.
Status
As of November 8, 2026, the evaluation foundation is complete and SFT data preparation is ready. The current execution plan separates development from resumable local compute: independent code, dataset and test work need not wait for an entire İzci release. Actual runs still require their own input artifacts and evidence. The frozen july_2026 baseline contains a versioned 25-task diagnostic suite across five categories, four deterministic grader paths, three comparable local Ollama runs, 75 validated scored outputs, deterministic comparison reporting, and a manual audit of all 28 recorded failures.
Within this specific suite, gpt-oss recorded a 92.0% pass rate; lfm2.5-8b and qwen3.6 each recorded 48.0%. These are frozen diagnostic outcomes, not a general model ranking. The audit found that visible thinking text, literal grader checks, code extraction, and a fixed 1024-token limit materially affected the recorded failures. The original scores remain unchanged so future work has an honest baseline.
The SFT stage has a pinned Qwen3-8B MLX artifact, a validated data contract, and a versioned pilot containing 200 training and 40 validation examples. Exact-base v2 measurements and a validated local smoke update/save/reload precede production SFT claims. The 13-stage plan retains two phases and Release 1, but earlier target months are historical; no replacement completion date is committed before representative local runs. Local-first execution separates development from resumable compute; hardware-specific measurements still require matching devices.
Where it lives
The public lab section is Open Model Lab. The completed July gate, published report, evaluation suite, and run summaries live there. Source code, configurations, reports, and the frozen baseline are in the public repository.