Open Model Lab

Open Model Lab 13-Stage Plan

A 13-stage public plan in two phases for building an eval-first research-engineering harness around open models, post-training, agents, safety, monitorability, and systems efficiency.

Current execution plan

Development and resumable compute are separate workstreams. Interfaces, datasets, graders and tests can advance while long local jobs run. Actual SFT, DPO, agent comparisons and reports wait for their own required artifacts, not an entire İzci release or a calendar month.

Local-first execution keeps development moving while resumable experiments run. Hardware-specific measurements still require matching devices. Earlier target months are historical; no replacement completion date is committed before representative local runs.

Execution policy updated November 8, 2026. The public repository plan records the workstreams, stage gates and recovery protocol; this does not imply queue automation has been deployed.

Two independent workstreams

Development

Build the next ready interface, evaluator, loss check, data fixture or report component. Small fixtures test the code; they are not production evidence.

Compute

Run one resumable local GPU job at a time, with identified inputs, checkpoints and task progress. Pause at safe boundaries for short enabling tests; accept results only after quality checks.

Plan principles

Eval-first

Model behavior should be measured before it is modified.

Same-suite comparison

Base, SFT, DPO, and agent variants should be compared on the same task suite when possible.

Claim boundaries

Every report states what it can and cannot claim.

No unsupported progress

Planned dashboards and reports stay planned until runs, configs, datasets, and reports exist.

Detectable effects

A difference is reported as a change only when it exceeds the suite's confidence interval.

Controlled comparisons

Compared variants share the base model, runtime, and precision.

Reuse before rebuilding

Where İzci already provides evaluation tooling, question sets, or a harness, the Lab builds on them and spends its effort on questions İzci doesn't ask.

Research stages

Phase 1 — Open models: evaluation, post-training, agents, and safety

July 2026
Completed

Foundation + Eval Harness

Build the basic research infrastructure that can measure model behavior reliably before changing it.

Success criterion: Different open models can be compared on the same tasks and a reproducible report can be generated.

Stage completion decision: Proceed to the SFT and data-quality stage using the frozen July baseline as the unchanged comparison point. Any grader, extraction, generation-limit, or model-artifact change must create a new evaluation identity rather than rewrite the July results.

completed
Artifact-gated
Artifact gate

Eval Suite v2

Make the evaluation suite sensitive enough to detect post-training effects before any model is modified.

Compute prerequisite: Versioned complete suite, selected model/runtime identities, repeatable local runner and smoke update. Prepare available subsets without waiting for an entire İzci release.

Success criterion: Repeated runs reproduce their scores, and per-category 95% confidence intervals are ±5 points or narrower.

Stage completion decision: Can the suite detect effects of the size SFT and DPO are expected to produce?

planned
After v2 base runs
Artifact gate

SFT Pipeline + Data Quality

Measure the behavioral difference between a base model and an instruction-tuned model.

Compute prerequisite: The exact selected base artifact scored on eval suite v2, accepted smoke save/reload, and versioned training data. Code and masks can be developed earlier with fixtures.

Success criterion: The work does not stop at fine-tuning; it shows measurable behavior changes, including regressions, beyond eval suite v2's confidence intervals.

Stage completion decision: Is the SFT checkpoint reliable enough to serve as the reference for preference optimization?

planned
After SFT checkpoint
Artifact gate

Preference Optimization / DPO

Use preference data to improve SFT behavior in a more controlled way, then measure behavioral side effects.

Compute prerequisite: The chosen stage 3 checkpoint and valid preference pairs; objective and schema tests may proceed before the full training job.

Success criterion: The report clearly shows where post-training helps and where it creates risk.

Stage completion decision: Is DPO improving real behavior or only optimizing preference style?

planned
After usable tool interfaces
Artifact gate

Agent Harness / Tool Use

Move the model from passive question-answering into a tool-using agent setup.

Compute prerequisite: A usable versioned orchestrator/trace interface and sandboxed tools. No whole-product release is a prerequisite for code or fixture tests.

Success criterion: Agent behavior is measured step by step with failure modes, not only success/failure.

Stage completion decision: Can agent traces explain why the model succeeds or fails?

planned
After working agent loop
Artifact gate

Agent Evals + Long-Horizon Tasks

Measure agent success and failure taxonomy on multi-step tasks.

Compute prerequisite: Stage 5's actual loop, identified model variants and held-out tasks; prepare taxonomy and task fixtures independently.

Success criterion: A reliable evaluation system categorizes why the agent fails.

Stage completion decision: Do the evals reveal actionable failure patterns rather than just pass/fail rates?

planned
After selected variants
Artifact gate

Safety / Red Teaming / Refusal Quality

Measure safety behavior by quality and balance, not just refusal rate.

Compute prerequisite: Identified checkpoints/agent variants and validated contrast/rubric sets; rubric development need not wait for all model runs.

Success criterion: The model's behavior is measured on both risky and harmless requests.

Stage completion decision: Can the safety eval distinguish good refusal from lazy or excessive refusal?

planned
After Phase 1 evidence
Artifact gate

Release 1 — Phase 1 Portfolio

Package Phase 1 as a reproducible, readable portfolio before Phase 2 begins.

Compute prerequisite: Reproducible actual results for stages 1–7; packaging code can be prepared earlier, but dashboards contain real runs only.

Success criterion: An external reader can reproduce one stage's main result from the repository alone.

Release decision: Which Phase 1 artifacts carry into Phase 2 unchanged, and which need a new version?

planned

Phase 2 — Internals, data, and systems, with project-trained models

Phase 2 adds small project-trained models and a project-built inference engine. Independent code, fixtures and metrics may be prepared before Phase 1 packaging finishes; each actual experiment waits only for the specific model/data artifacts it needs.

After suitable VLM artifacts
Artifact gate

Multimodal / UI Understanding / Computer Use

Enter multimodal model work through screenshot and UI-understanding tasks.

Compute prerequisite: Identified open/project VLMs and held-out UI tasks; screenshot/metric development can proceed before the checkpoints exist.

Success criterion: The model's ability to understand visual interfaces is measured task-by-task.

Stage completion decision: Can the eval distinguish OCR success from actual UI reasoning?

planned
After shared ladder inputs
Artifact gate

Data Efficiency + Scaling Ladder

Measure whether better data mixtures produce better behavior with the same compute.

Compute prerequisite: Versioned natural-text subsets and a tested trainer for one shared ladder. Data/curve-fitting code need not wait for İzci's full release.

Success criterion: The effect of data quality on behavior and compute efficiency is quantified.

Stage completion decision: Does data quality improve score/GPU-hour enough to justify the filtering pipeline?

planned
After benchmarkable runtimes
Artifact gate

Training / Inference Systems Efficiency

Add systems and profiling knowledge to make research experiments more efficient.

Compute prerequisite: Actual comparable model/runtime/hardware runs; develop engine correctness and benchmark interfaces independently.

Success criterion: The main bottlenecks slowing model research infrastructure can be measured and improved.

Stage completion decision: Which bottlenecks most affect experiment velocity and cost?

planned
After reasoning checkpoints
Artifact gate

Reasoning Behavior + Process Evaluation

Evaluate not only final answers, but where reasoning processes break down.

Compute prerequisite: Suitable checkpoints and recorded outcomes; reward/rubric and process-analysis fixtures can be developed first.

Success criterion: The system can show flawed processes despite correct answers, or identify breakpoints leading to wrong answers.

Stage completion decision: Do process signals add useful diagnostic value beyond final-answer grading?

planned
After selected activations
Artifact gate

Monitorability / Interpretability Start

Start analyzing model behavior with internal signals in addition to external scores.

Compute prerequisite: Selected real model activations and held-out failure splits; tiny-model probe/extraction tests are preparation, not results.

Success criterion: Behavior changes can be tracked not only from outputs but also from model-internal measurements.

Stage completion decision: Are internal signals useful enough to guide later experiments?

planned
After accepted final artifacts
Artifact gate

Final Integration + Public Portfolio

Turn the 13-stage work into a showable, reproducible, publishable portfolio.

Compute prerequisite: Actual accepted model/eval/report artifacts. Keep unrun full-scale and hardware branches explicitly pending rather than inventing completion.

Success criterion: An open-source project demonstrates an end-to-end open-model research-engineering loop.

Stage completion decision: Is the project understandable, reproducible, and credible to an external research-engineering reader?

planned