Development
Build the next ready interface, evaluator, loss check, data fixture or report component. Small fixtures test the code; they are not production evidence.
Open Model Lab
A 13-stage public plan in two phases for building an eval-first research-engineering harness around open models, post-training, agents, safety, monitorability, and systems efficiency.
Development and resumable compute are separate workstreams. Interfaces, datasets, graders and tests can advance while long local jobs run. Actual SFT, DPO, agent comparisons and reports wait for their own required artifacts, not an entire İzci release or a calendar month.
Local-first execution keeps development moving while resumable experiments run. Hardware-specific measurements still require matching devices. Earlier target months are historical; no replacement completion date is committed before representative local runs.
Execution policy updated November 8, 2026. The public repository plan records the workstreams, stage gates and recovery protocol; this does not imply queue automation has been deployed.
Build the next ready interface, evaluator, loss check, data fixture or report component. Small fixtures test the code; they are not production evidence.
Run one resumable local GPU job at a time, with identified inputs, checkpoints and task progress. Pause at safe boundaries for short enabling tests; accept results only after quality checks.
Model behavior should be measured before it is modified.
Base, SFT, DPO, and agent variants should be compared on the same task suite when possible.
Every report states what it can and cannot claim.
Planned dashboards and reports stay planned until runs, configs, datasets, and reports exist.
A difference is reported as a change only when it exceeds the suite's confidence interval.
Compared variants share the base model, runtime, and precision.
Where İzci already provides evaluation tooling, question sets, or a harness, the Lab builds on them and spends its effort on questions İzci doesn't ask.
Build the basic research infrastructure that can measure model behavior reliably before changing it.
Success criterion: Different open models can be compared on the same tasks and a reproducible report can be generated.
Stage completion decision: Proceed to the SFT and data-quality stage using the frozen July baseline as the unchanged comparison point. Any grader, extraction, generation-limit, or model-artifact change must create a new evaluation identity rather than rewrite the July results.
Make the evaluation suite sensitive enough to detect post-training effects before any model is modified.
Compute prerequisite: Versioned complete suite, selected model/runtime identities, repeatable local runner and smoke update. Prepare available subsets without waiting for an entire İzci release.
Success criterion: Repeated runs reproduce their scores, and per-category 95% confidence intervals are ±5 points or narrower.
Stage completion decision: Can the suite detect effects of the size SFT and DPO are expected to produce?
Measure the behavioral difference between a base model and an instruction-tuned model.
Compute prerequisite: The exact selected base artifact scored on eval suite v2, accepted smoke save/reload, and versioned training data. Code and masks can be developed earlier with fixtures.
Success criterion: The work does not stop at fine-tuning; it shows measurable behavior changes, including regressions, beyond eval suite v2's confidence intervals.
Stage completion decision: Is the SFT checkpoint reliable enough to serve as the reference for preference optimization?
Use preference data to improve SFT behavior in a more controlled way, then measure behavioral side effects.
Compute prerequisite: The chosen stage 3 checkpoint and valid preference pairs; objective and schema tests may proceed before the full training job.
Success criterion: The report clearly shows where post-training helps and where it creates risk.
Stage completion decision: Is DPO improving real behavior or only optimizing preference style?
Move the model from passive question-answering into a tool-using agent setup.
Compute prerequisite: A usable versioned orchestrator/trace interface and sandboxed tools. No whole-product release is a prerequisite for code or fixture tests.
Success criterion: Agent behavior is measured step by step with failure modes, not only success/failure.
Stage completion decision: Can agent traces explain why the model succeeds or fails?
Measure agent success and failure taxonomy on multi-step tasks.
Compute prerequisite: Stage 5's actual loop, identified model variants and held-out tasks; prepare taxonomy and task fixtures independently.
Success criterion: A reliable evaluation system categorizes why the agent fails.
Stage completion decision: Do the evals reveal actionable failure patterns rather than just pass/fail rates?
Measure safety behavior by quality and balance, not just refusal rate.
Compute prerequisite: Identified checkpoints/agent variants and validated contrast/rubric sets; rubric development need not wait for all model runs.
Success criterion: The model's behavior is measured on both risky and harmless requests.
Stage completion decision: Can the safety eval distinguish good refusal from lazy or excessive refusal?
Package Phase 1 as a reproducible, readable portfolio before Phase 2 begins.
Compute prerequisite: Reproducible actual results for stages 1–7; packaging code can be prepared earlier, but dashboards contain real runs only.
Success criterion: An external reader can reproduce one stage's main result from the repository alone.
Release decision: Which Phase 1 artifacts carry into Phase 2 unchanged, and which need a new version?
Phase 2 adds small project-trained models and a project-built inference engine. Independent code, fixtures and metrics may be prepared before Phase 1 packaging finishes; each actual experiment waits only for the specific model/data artifacts it needs.
Enter multimodal model work through screenshot and UI-understanding tasks.
Compute prerequisite: Identified open/project VLMs and held-out UI tasks; screenshot/metric development can proceed before the checkpoints exist.
Success criterion: The model's ability to understand visual interfaces is measured task-by-task.
Stage completion decision: Can the eval distinguish OCR success from actual UI reasoning?
Measure whether better data mixtures produce better behavior with the same compute.
Compute prerequisite: Versioned natural-text subsets and a tested trainer for one shared ladder. Data/curve-fitting code need not wait for İzci's full release.
Success criterion: The effect of data quality on behavior and compute efficiency is quantified.
Stage completion decision: Does data quality improve score/GPU-hour enough to justify the filtering pipeline?
Add systems and profiling knowledge to make research experiments more efficient.
Compute prerequisite: Actual comparable model/runtime/hardware runs; develop engine correctness and benchmark interfaces independently.
Success criterion: The main bottlenecks slowing model research infrastructure can be measured and improved.
Stage completion decision: Which bottlenecks most affect experiment velocity and cost?
Evaluate not only final answers, but where reasoning processes break down.
Compute prerequisite: Suitable checkpoints and recorded outcomes; reward/rubric and process-analysis fixtures can be developed first.
Success criterion: The system can show flawed processes despite correct answers, or identify breakpoints leading to wrong answers.
Stage completion decision: Do process signals add useful diagnostic value beyond final-answer grading?
Start analyzing model behavior with internal signals in addition to external scores.
Compute prerequisite: Selected real model activations and held-out failure splits; tiny-model probe/extraction tests are preparation, not results.
Success criterion: Behavior changes can be tracked not only from outputs but also from model-internal measurements.
Stage completion decision: Are internal signals useful enough to guide later experiments?
Turn the 13-stage work into a showable, reproducible, publishable portfolio.
Compute prerequisite: Actual accepted model/eval/report artifacts. Keep unrun full-scale and hardware branches explicitly pending rather than inventing completion.
Success criterion: An open-source project demonstrates an end-to-end open-model research-engineering loop.
Stage completion decision: Is the project understandable, reproducible, and credible to an external research-engineering reader?