Open Model Lab

Decisions

A public decision log for scope, naming, eval-first sequencing, and claim boundaries.

Decision log

2026-11-08-local-first

Local-first execution with independent workstreams

Decision: Use local-first execution with separate development and resumable-compute backlogs. Replace fixed target months with actual artifact prerequisites.

Rationale: Long local jobs can continue while independent interfaces, datasets, graders and tests are developed. Only actual model/data prerequisites should block an experiment, not an entire release or a calendar month.

Consequence: Earlier May 2027–August 2028 targets are historical. Preserve completed dates, frozen identities, scope and evidence gates. Hardware-specific measurements still require matching devices; no new finish date or scheduler deployment is implied.

recorded
2026-11-08

Research schedule updated

Decision: Eval suite v2 stays in May 2027; SFT moves to June 2027, DPO to July 2027, and agent and safety work to August 2027, when Release 1 closes Phase 1. Phase 2 runs from December 2027 to final integration in August 2028.

Rationale: May 2027 is kept as a margin in case İzci needs more time; the later stages are rescheduled without changing any stage's scope or success criterion.

Consequence: The November 7 targets (Release 1 in March 2028, final integration in December 2030) are replaced. Stage names, scopes, and completion decisions are unchanged.

recorded
2026-11-07

Research schedule updated

Decision: Reorganize the plan into 13 stages in two phases, and pause new stages from November 2026 to April 2027. Phase 1 (evaluation, post-training, agents, safety) runs from May 2027 to Release 1 in March 2028; Phase 2 (multimodal, data, systems, reasoning process, monitorability) runs from February 2029 to final integration in December 2030.

Rationale: İzci, a separate research-agent project, is built until April 2027, and its published evaluation tooling and harness are inputs to Stages 2 and 5. Phase 2 questions need models, data pipelines, and an inference engine built within the project, and those are scheduled before the stages that use them.

Consequence: The September 2027 final-integration target is replaced by Release 1 (March 2028) and final integration (December 2030). Stage names are unchanged; future dates are still reviewed as each stage closes.

recorded
2026-10-17

Research schedule updated

Decision: Continue SFT and data-quality work in November 2026, followed by DPO in December and agent work in January 2027. Final integration is targeted for September 2027.

Rationale: Use a current schedule for the 12 research stages, with completed preparation carried forward.

Consequence: Stage names are independent of target dates. Completed runs and reports retain their actual dates; future dates are reviewed as each stage closes.

recorded
D001

Use "Open Model Research Harness" as the public project name

Decision: Use Open Model Research Harness for the project and Open Model Lab for the public section.

Rationale: The project borrows from rigorous research workflow patterns, but the public name should not imply that this is itself a frontier AI lab.

Consequence: Public routes, tags, titles, and navigation should use open-model language, not frontier-lab branding.

recorded
D002

Eval-first before model modification

Decision: The first stage focuses on eval infrastructure before SFT, DPO, agents, or safety work.

Rationale: Without reproducible evals, every later model change is anecdotal.

Consequence: July is successful only when multiple models can be compared on the same tasks with score, cost, latency, and failure-mode reporting.

recorded
D003

No unsupported leaderboard

Decision: The site will not publish a leaderboard without real, reproducible runs.

Rationale: The project is a research-engineering portfolio, not benchmark marketing.

Consequence: Dashboards and reports remain planned/empty until configs, runs, and reports exist.

recorded
D004

Claim boundaries on every report

Decision: Every report must state what it does and does not claim.

Rationale: Small experiments are valuable, but overclaiming makes them less credible.

Consequence: All planned report pages include a claim-boundary section.

recorded
D005

Static-first public reporting

Decision: The public site should be static/content-driven unless actual run data requires otherwise.

Rationale: The site should be easy to maintain and hard to break.

Consequence: Initial pages are generated from structured content, not a database.

recorded
D006

Freeze the imperfect July baseline

Decision: Preserve the original July scores and attach the failure audit as a separate interpretation layer.

Rationale: Correcting likely grader or runtime artifacts in place would erase the baseline needed for honest later comparisons.

Consequence: Changes to tasks, graders, extraction, generation settings, or model artifacts require a new versioned evaluation identity.

recorded
D007

Eval suite v2 before SFT

Decision: Add Stage 2, Eval Suite v2, before SFT: public benchmark subsets and İzci's context-grounded question sets for the five July categories, about 400 tasks per category, paired bootstrap confidence intervals, and generation limits that avoid truncation.

Rationale: The July suite has five tasks per category, so one task moves a category score by 20 points, and the failure audit attributed none of its 28 failures to substantive model behavior. SFT-sized effects would be indistinguishable from noise.

Consequence: july_eval_v1 stays frozen (D006). Post-training, agent, and safety comparisons use eval suite v2, and differences are reported as changes only when they exceed its confidence intervals.

recorded
D008

Same runtime and precision for compared variants

Decision: Post-training uses one base model with trainable open weights. Base, SFT, DPO, and agent variants are evaluated in the same runtime and precision, with exact weight hashes recorded.

Rationale: The July models are Ollama tags with unverified quantization; mixing runtimes or precisions would confound post-training effects.

Consequence: The July Ollama runs remain a historical baseline, not a comparison point for project checkpoints.

recorded
D009

Project-trained models in Phase 2

Decision: Phase 2 evaluates small models trained from scratch within the project alongside open models.

Rationale: Data-efficiency, systems, and internal-signal questions need control over training data, architecture, and runtime.

Consequence: Model pages separate open models from project-trained checkpoints, and claim boundaries state their small scale.

recorded
D010

Build on İzci's tooling

Decision: Eval suite v2 builds on İzci's evaluation tooling and uses its published English question sets for context-grounded factuality; the agent harness builds on İzci's harness; prompt injection and data exfiltration are left to İzci's security sets.

Rationale: İzci builds these at larger scale, with locked test splits and measured judge agreement. Rebuilding them would duplicate work, and shared sets make the two projects' results comparable.

Consequence: Lab stages concentrate on what İzci doesn't cover: post-training effects on open models, coding agents, and refusal quality. Lab reports record which İzci release they use.

recorded