Completed
Foundation + Eval Harness
Build the basic research infrastructure that can measure model behavior reliably before changing it.
Open Model Lab
A public engineering notebook for measuring open-model behavior across evals, post-training, agents, safety, monitorability, and systems work.
The SFT pilot dataset is prepared. Development and resumable local compute now have separate backlogs: evaluator, dataset and test work can proceed without waiting for an entire model release. Production runs and reports still require their actual input artifacts and evidence.
The first stage completed on July 29, 2026: 25 evaluation tasks, three local models, 75 scored outputs, a failure audit, and a frozen baseline for subsequent experiments.
Build the basic research infrastructure that can measure model behavior reliably before changing it.
Make the evaluation suite sensitive enough to detect post-training effects before any model is modified.
Measure the behavioral difference between a base model and an instruction-tuned model.
Use preference data to improve SFT behavior in a more controlled way, then measure behavioral side effects.
Move the model from passive question-answering into a tool-using agent setup.
Measure agent success and failure taxonomy on multi-step tasks.
Measure safety behavior by quality and balance, not just refusal rate.
Package Phase 1 as a reproducible, readable portfolio before Phase 2 begins.
Enter multimodal model work through screenshot and UI-understanding tasks.
Measure whether better data mixtures produce better behavior with the same compute.
Add systems and profiling knowledge to make research experiments more efficient.
Evaluate not only final answers, but where reasoning processes break down.
Start analyzing model behavior with internal signals in addition to external scores.
Turn the 13-stage work into a showable, reproducible, publishable portfolio.
Reproducible task execution, grading, metrics, and failure labels.
Versioned benchmark subsets and suitable İzci artifacts, with paired confidence intervals; development proceeds before production runs are ready.
Pilot data prepared; production LoRA training follows exact-base v2 runs and validated local smoke training.
Chosen/rejected pairs, DPO training, and side-effect measurement.
A coding-agent harness built on İzci's open-source harness: file editing, code execution, test running, and replayable traces.
Multi-step coding tasks and trace-level failure taxonomy.
Refusal quality, over-refusal, under-refusal, jailbreak robustness, and safe completion.
Phase 1 reports, dashboards, and reproducible scripts.
Screenshot QA, OCR plus reasoning, UI grounding, and visual failure modes, including a small project-trained VLM.
Data mixtures, filtering, small scaling ladders trained from scratch, and score/GPU-hour.
Latency, throughput, batching, KV cache, quantization, and profiling, including a project-built inference engine.
Outcome scoring compared with observable process and self-correction signals, including models trained with RL on verifiable rewards.
Internal-signal probes for failure prediction and representation drift.
Reproducible scripts, reports, dashboard, README, and portfolio packaging.
200 training examples, 40 validation examples, and a 24-example smoke subset, prepared September 24, 2026.
The completed July milestone report, comparison, limitations, and SFT handoff.
A versioned 25-task diagnostic suite across five categories; all task quality labels remain draft.
A frozen identity for the dataset, configs, graders, reports, and three local runs.
A manual review of all 28 recorded failures without rewriting the original scores.
The current local-first plan in two phases, with independent development/compute workstreams and artifact-gated stages rather than fixed future months.
Research stages, target dates, and completed evidence.
Research stages, target dates, expected outputs, and decisions.
Planned and published report pages with explicit claim boundaries.
A compact reading and writing template for Open Model Lab reports.
Task schema, initial categories, graders, metrics, and failure taxonomy.
Published summaries for the three frozen July diagnostic runs.
Recorded identities and caveats for models used in published evaluations.
The July evaluation suite, the planned eval suite v2, the prepared SFT pilot, and planned later datasets.
Planned dashboards. No placeholder charts or synthetic metrics.
Public decision log for naming, eval-first scope, and claim boundaries.
Concise definitions for terms used throughout the lab section.