Open Model Lab

Open Model Lab

A public engineering notebook for measuring open-model behavior across evals, post-training, agents, safety, monitorability, and systems work.

Next · artifact-gated

Eval Suite v2

The SFT pilot dataset is prepared. Development and resumable local compute now have separate backlogs: evaluator, dataset and test work can proceed without waiting for an entire model release. Production runs and reports still require their actual input artifacts and evidence.

Prepared
200 training / 40 validation examples
Next step
Evaluator/local-runner readiness, then v2 baselines and validated LoRA smoke training
Current plan
13 stages in two phases · local-first · milestones, not fixed future months

Completed: evaluation foundation

The first stage completed on July 29, 2026: 25 evaluation tasks, three local models, 75 scored outputs, a failure audit, and a frozen baseline for subsequent experiments.

Research roadmap

July 2026
Completed

Foundation + Eval Harness

Build the basic research infrastructure that can measure model behavior reliably before changing it.

completed
Artifact-gated
Artifact gate

Eval Suite v2

Make the evaluation suite sensitive enough to detect post-training effects before any model is modified.

planned
After v2 base runs
Artifact gate

SFT Pipeline + Data Quality

Measure the behavioral difference between a base model and an instruction-tuned model.

planned
After SFT checkpoint
Artifact gate

Preference Optimization / DPO

Use preference data to improve SFT behavior in a more controlled way, then measure behavioral side effects.

planned
After usable tool interfaces
Artifact gate

Agent Harness / Tool Use

Move the model from passive question-answering into a tool-using agent setup.

planned
After shared ladder inputs
Artifact gate

Data Efficiency + Scaling Ladder

Measure whether better data mixtures produce better behavior with the same compute.

planned

Research modules

completed

Eval Harness

Reproducible task execution, grading, metrics, and failure labels.

planned

Eval Suite v2

Versioned benchmark subsets and suitable İzci artifacts, with paired confidence intervals; development proceeds before production runs are ready.

planned

SFT Pipeline

Pilot data prepared; production LoRA training follows exact-base v2 runs and validated local smoke training.

planned

Preference Optimization / DPO

Chosen/rejected pairs, DPO training, and side-effect measurement.

planned

Agent Harness

A coding-agent harness built on İzci's open-source harness: file editing, code execution, test running, and replayable traces.

planned

Long-Horizon Agent Evals

Multi-step coding tasks and trace-level failure taxonomy.

planned

Safety / Red Teaming

Refusal quality, over-refusal, under-refusal, jailbreak robustness, and safe completion.

planned

Release 1

Phase 1 reports, dashboards, and reproducible scripts.

planned

Multimodal UI Understanding

Screenshot QA, OCR plus reasoning, UI grounding, and visual failure modes, including a small project-trained VLM.

planned

Data Efficiency / Scaling

Data mixtures, filtering, small scaling ladders trained from scratch, and score/GPU-hour.

planned

Systems Efficiency

Latency, throughput, batching, KV cache, quantization, and profiling, including a project-built inference engine.

planned

Reasoning Process Evaluation

Outcome scoring compared with observable process and self-correction signals, including models trained with RL on verifiable rewards.

planned

Monitorability

Internal-signal probes for failure prediction and representation drift.

planned

Final Integration

Reproducible scripts, reports, dashboard, README, and portfolio packaging.

Latest public artifacts

Prepared

SFT pilot dataset

200 training examples, 40 validation examples, and a 24-example smoke subset, prepared September 24, 2026.

Published

July evaluation suite

A versioned 25-task diagnostic suite across five categories; all task quality labels remain draft.

Published

July baseline

A frozen identity for the dataset, configs, graders, reports, and three local runs.

Published

Failure audit

A manual review of all 28 recorded failures without rewriting the original scores.

Claim boundary

Diagnostic baseline, not a leaderboard. The July results apply only to a small 25-task suite on one local runtime. They do not establish general model quality or statistical significance. Planned later reports and dashboards remain empty until their own evidence exists.

13-stage plan

The current local-first plan in two phases, with independent development/compute workstreams and artifact-gated stages rather than fixed future months.

Timeline

Research stages, target dates, and completed evidence.

Stages

Research stages, target dates, expected outputs, and decisions.

Reports

Planned and published report pages with explicit claim boundaries.

Report template

A compact reading and writing template for Open Model Lab reports.

Evals

Task schema, initial categories, graders, metrics, and failure taxonomy.

Runs

Published summaries for the three frozen July diagnostic runs.

Models

Recorded identities and caveats for models used in published evaluations.

Datasets

The July evaluation suite, the planned eval suite v2, the prepared SFT pilot, and planned later datasets.

Dashboards

Planned dashboards. No placeholder charts or synthetic metrics.

Decisions

Public decision log for naming, eval-first scope, and claim boundaries.

Glossary

Concise definitions for terms used throughout the lab section.