Building Fresh

Open Model Research Harness

A 13-stage public research-engineering project for reproducible open-model evals, post-training, agents, safety, monitorability, and systems efficiency.

Actively developing. Not usable yet.

Started
Jul 1, 2026
Updated
Nov 8, 2026
→

Gathering feedback — open an issue with your thoughts.

Open in repo ↗

Current status

Local-first; independent development and compute

The evaluation foundation is complete. SFT preparation includes a validated data contract, 200 training examples, 40 validation examples, and a 24-example smoke subset.

Latest public artifact
SFT data preparation report ↗ Sep 24, 2026
Next checkpoint
Prepare evaluator/local-runner readiness and v2 input identities; validate resumable smoke training, then measure the exact base before production SFT.
Roadmap1 / 14 — 7% · effort-weighted
  • ✓Foundation + Eval Harness — Completed July 29, 2026
  • Eval Suite v2 — Gate: versioned suite and validated local runner
  • SFT Pipeline + Data Quality — Gate: exact-base v2 scores and smoke update; pilot data prepared
  • Preference Optimization / DPO — Gate: chosen SFT checkpoint and preference pairs
  • Agent Harness / Tool Use — Gate: usable tool/trace interfaces and sandbox
  • Agent Evals + Long-Horizon Tasks — Gate: working agent loop and actual held-out attempts
  • Safety / Red Teaming / Refusal Quality — Gate: selected variants and validated rubrics
  • Release 1 — Phase 1 Portfolio — Gate: reproducible Phase 1 evidence
  • Multimodal / UI Understanding — Gate: selected VLM artifacts and held-out UI tasks
  • Data Efficiency + Scaling Ladder — Gate: one shared ladder with versioned inputs
  • Training / Inference Systems Efficiency — Gate: comparable real runtime/hardware measurements
  • Reasoning Behavior + Process Evaluation — Gate: reasoning checkpoints and actual outcomes
  • Monitorability / Interpretability — Gate: selected activations and held-out failure splits
  • Final Integration + Public Portfolio — Gate: accepted final artifacts; unrun branches stay pending
StatusLocal-first / evaluator and runner readiness next, SFT data prepared
TimeframeStarted July 2026 / future stages are artifact-gated
ScopeOpen models, small experiments, reproducible evaluation, public reporting
Claim boundaryNo frontier-model capability claim

What it is

Open Model Research Harness is a 13-stage public research-engineering project for studying open LLMs through reproducible evaluations, post-training experiments, agentic task harnesses, safety evaluations, monitorability probes, and systems-efficiency measurements.

The goal is not to claim frontier-model capability. The goal is to build and document the eval-first research workflow needed to understand how model behavior changes across SFT, preference optimization, tool use, safety constraints, and inference/training efficiency work.

The project is inspired by frontier-lab research workflows, but intentionally scoped to open models, small experiments, reproducible evaluation, and public reporting.

It is a modular research harness and public portfolio for:

  • evals
  • SFT
  • preference optimization / DPO
  • agent harnesses
  • long-horizon agent evals
  • safety and refusal quality
  • reasoning process evaluation
  • monitorability / interpretability probes
  • multimodal UI understanding
  • systems efficiency
  • data efficiency and scaling ladders

Why it exists

Open-model work is easy to overclaim when the evaluation loop is weak. A fine-tune, preference run, or agent demo can look useful in isolation while hiding regressions, safety failures, trace-level mistakes, or systems bottlenecks.

This project makes the measuring system the first deliverable. The completed July 2026 gate is not a model release; it is a small reproducible eval harness that compares multiple open models on the same tasks and reports score, local-runtime latency, token usage, output quality, and failure modes.

How it differs

The project is not structured as a leaderboard or benchmark marketing page. It is a public engineering record: each report has a research question, a setup, measurements, backing artifacts, and a claim boundary. The July report is published because its dataset, configurations, runs, comparison, failure audit, and baseline identity exist; later reports and dashboards remain planned until their evidence exists.

The work also connects several layers that are often presented separately: eval design, post-training, agent traces, safety behavior, monitorability probes, and systems profiling. The point is to show how behavior changes across the loop, not to claim that small open models match larger private systems.

What it is not

  • It is not a claim that I run a frontier AI lab.
  • It is not a leaderboard.
  • It is not a benchmark marketing page.
  • It is not a claim that small open models match frontier models.
  • It is not a large-scale pretraining project.

Research loop

01

Measure

Measure model behavior before changing it.

02

Modify

Modify behavior with SFT, DPO, tools, or constraints.

03

Compare

Compare base/SFT/DPO/agent variants on the same eval suite.

04

Diagnose

Diagnose failures with taxonomy and traces.

05

Report

Report results with claim boundaries.

Roadmap

July 2026Completed

Foundation + Eval Harness

Build the basic research infrastructure that can measure model behavior reliably before changing it.

completed Report
Artifact-gatedArtifact gate

Eval Suite v2

Make the evaluation suite sensitive enough to detect post-training effects before any model is modified.

planned
After v2 base runsArtifact gate

SFT Pipeline + Data Quality

Measure the behavioral difference between a base model and an instruction-tuned model.

planned
After SFT checkpointArtifact gate

Preference Optimization / DPO

Use preference data to improve SFT behavior in a more controlled way, then measure behavioral side effects.

planned
After usable tool interfacesArtifact gate

Agent Harness / Tool Use

Move the model from passive question-answering into a tool-using agent setup.

planned
After shared ladder inputsArtifact gate

Data Efficiency + Scaling Ladder

Measure whether better data mixtures produce better behavior with the same compute.

planned

Module map

completed

Eval Harness

Reproducible task execution, grading, metrics, and failure labels.

planned

Eval Suite v2

Versioned benchmark subsets and suitable İzci artifacts, with paired confidence intervals; development proceeds before production runs are ready.

planned

SFT Pipeline

Pilot data prepared; production LoRA training follows exact-base v2 runs and validated local smoke training.

planned

Preference Optimization / DPO

Chosen/rejected pairs, DPO training, and side-effect measurement.

planned

Agent Harness

A coding-agent harness built on İzci's open-source harness: file editing, code execution, test running, and replayable traces.

planned

Long-Horizon Agent Evals

Multi-step coding tasks and trace-level failure taxonomy.

planned

Safety / Red Teaming

Refusal quality, over-refusal, under-refusal, jailbreak robustness, and safe completion.

planned

Release 1

Phase 1 reports, dashboards, and reproducible scripts.

planned

Multimodal UI Understanding

Screenshot QA, OCR plus reasoning, UI grounding, and visual failure modes, including a small project-trained VLM.

planned

Data Efficiency / Scaling

Data mixtures, filtering, small scaling ladders trained from scratch, and score/GPU-hour.

planned

Systems Efficiency

Latency, throughput, batching, KV cache, quantization, and profiling, including a project-built inference engine.

planned

Reasoning Process Evaluation

Outcome scoring compared with observable process and self-correction signals, including models trained with RL on verifiable rewards.

planned

Monitorability

Internal-signal probes for failure prediction and representation drift.

planned

Final Integration

Reproducible scripts, reports, dashboard, README, and portfolio packaging.

Status

As of November 8, 2026, the evaluation foundation is complete and SFT data preparation is ready. The current execution plan separates development from resumable local compute: independent code, dataset and test work need not wait for an entire İzci release. Actual runs still require their own input artifacts and evidence. The frozen july_2026 baseline contains a versioned 25-task diagnostic suite across five categories, four deterministic grader paths, three comparable local Ollama runs, 75 validated scored outputs, deterministic comparison reporting, and a manual audit of all 28 recorded failures.

Within this specific suite, gpt-oss recorded a 92.0% pass rate; lfm2.5-8b and qwen3.6 each recorded 48.0%. These are frozen diagnostic outcomes, not a general model ranking. The audit found that visible thinking text, literal grader checks, code extraction, and a fixed 1024-token limit materially affected the recorded failures. The original scores remain unchanged so future work has an honest baseline.

The SFT stage has a pinned Qwen3-8B MLX artifact, a validated data contract, and a versioned pilot containing 200 training and 40 validation examples. Exact-base v2 measurements and a validated local smoke update/save/reload precede production SFT claims. The 13-stage plan retains two phases and Release 1, but earlier target months are historical; no replacement completion date is committed before representative local runs. Local-first execution separates development from resumable compute; hardware-specific measurements still require matching devices.

Where it lives

The public lab section is Open Model Lab. The completed July gate, published report, evaluation suite, and run summaries live there. Source code, configurations, reports, and the frozen baseline are in the public repository.

open-model-evalspost-trainingagent-harnessessafety-evalsmonitorabilitysystems-efficiency