July 2026
Foundation + Eval Harness
Build the basic research infrastructure that can measure model behavior reliably before changing it.
completed
Open Model Lab
Open Model Lab
A public engineering notebook for measuring open-model behavior across evals, post-training, agents, safety, monitorability, and systems work.
Status facts
- Project
- Open Model Research Harness
- Repository
- GitHub
- Timeframe
- July 2026 - June 2027
- Current status
- July gate completed; August planned
- Latest completed gate
- July 2026 eval harness completed
- Next gate
- August 2026 SFT + data quality (planned)
- Claims
- Diagnostic baseline only; no general model-quality claim
July gate completed; August is next
Open Model Lab is a public engineering notebook for measuring open-model behavior across evals, post-training, agents, safety, and systems work. The July 2026 baseline is frozen; later experiments are compared against it rather than rewriting it.
The July Foundation + Eval Harness gate completed on July 29 with a frozen 25-task suite, three comparable local model runs, 75 validated scored outputs, deterministic reporting, and a manual audit of all 28 recorded failures. August SFT and data-quality work is planned but has not been marked active yet.
12-month timeline
August 2026
SFT Pipeline + Data Quality
Measure the behavioral difference between a base model and an instruction-tuned model.
planned
September 2026
Preference Optimization / DPO
Use preference data to improve SFT behavior in a more controlled way, then measure behavioral side effects.
planned
October 2026
Agent Harness / Tool Use
Move the model from passive question-answering into a tool-using agent setup.
planned
November 2026
Agent Evals + Long-Horizon Tasks
Measure agent success and failure taxonomy on multi-step tasks.
planned
December 2026
Safety / Red Teaming / Refusal Quality
Measure safety behavior by quality and balance, not just refusal rate.
planned
January 2027
Reasoning Behavior + Process Evaluation
Evaluate not only final answers, but where reasoning processes break down.
planned
February 2027
Monitorability / Interpretability Start
Start analyzing model behavior with internal signals in addition to external scores.
planned
March 2027
Multimodal / UI Understanding / Computer Use
Enter multimodal model work through screenshot and UI-understanding tasks.
planned
April 2027
Training / Inference Systems Efficiency
Add systems and profiling knowledge to make research experiments more efficient.
planned
May 2027
Data Efficiency + Scaling Ladder
Measure whether better data mixtures produce better behavior with the same compute.
planned
June 2027
Final Integration + Public Portfolio
Turn the 12-month work into a showable, reproducible, publishable portfolio.
planned
Research modules
Eval Harness
Reproducible task execution, grading, metrics, and failure labels.
SFT Pipeline
Instruction data cleaning, supervised fine-tuning, and behavior comparison.
Preference Optimization / DPO
Chosen/rejected pairs, DPO training, and side-effect measurement.
Agent Harness
Tool registry, file operations, test execution, and replayable traces.
Long-Horizon Agent Evals
Multi-step coding tasks and trace-level failure taxonomy.
Safety / Red Teaming
Refusal quality, over-refusal, under-refusal, jailbreak robustness, and safe completion.
Reasoning Process Evaluation
Outcome scoring compared with observable process and self-correction signals.
Monitorability
Internal-signal probes for failure prediction and representation drift.
Multimodal UI Understanding
Screenshot QA, OCR plus reasoning, UI grounding, and visual failure modes.
Systems Efficiency
Latency, throughput, batching, KV cache, quantization, and profiling.
Data Efficiency / Scaling
Data mixtures, filtering, small scaling ladders, and score/GPU-hour.
Final Integration
Reproducible scripts, reports, dashboard, README, and portfolio packaging.
Latest public artifacts
July eval harness report
The completed July milestone report, comparison, limitations, and August handoff.
July evaluation suite
A versioned 25-task diagnostic suite across five categories; all task quality labels remain draft.
July baseline
A frozen identity for the dataset, configs, graders, reports, and three local runs.
Failure audit
A manual review of all 28 recorded failures without rewriting the original scores.
Claim boundary
Diagnostic baseline, not a leaderboard. The July results apply only to a small 25-task suite on one local runtime. They do not establish general model quality or statistical significance. Planned later reports and dashboards remain empty until their own evidence exists.
Section index
12-month plan
The full July 2026 through June 2027 research-engineering plan.
Timeline
A compact month-by-month view of gates, themes, and report targets.
Months
Detailed monthly gates, focus areas, expected outputs, and decisions.
Reports
Planned and published report pages with explicit claim boundaries.
Report template
A compact reading and writing template for Open Model Lab reports.
Evals
Task schema, initial categories, graders, metrics, and failure taxonomy.
Runs
Published summaries for the three frozen July diagnostic runs.
Models
Recorded identities and caveats for models used in published evaluations.
Datasets
The July diagnostic eval suite plus planned training and later evaluation datasets.
Dashboards
Planned dashboards. No placeholder charts or synthetic metrics.
Decisions
Public decision log for naming, eval-first scope, and claim boundaries.
Glossary
Concise definitions for terms used throughout the lab section.