July 2026
Foundation + Eval Harness
Build the basic research infrastructure that can measure model behavior reliably before changing it.
planned
Open Model Lab
Open Model Lab
A public workspace for tracking open-model evaluation, post-training, agentic behavior, safety, monitorability, and systems-efficiency experiments.
Status facts
- Project
- Open Model Research Harness
- Repository
- GitHub
- Timeframe
- July 2026 - June 2027
- Current status
- Planned
- First gate
- July 2026 eval harness
- Claims
- No model-quality claims yet
Current gate: July 2026 - Foundation + Eval Harness
Before changing model behavior, the first milestone is to measure it reliably. The July gate is complete only when at least three open models can be evaluated on the same small task suite with score, cost, latency, output-quality, and failure-mode reporting.
12-month timeline
August 2026
SFT Pipeline + Data Quality
Measure the behavioral difference between a base model and an instruction-tuned model.
planned
September 2026
Preference Optimization / DPO
Use preference data to improve SFT behavior in a more controlled way, then measure behavioral side effects.
planned
October 2026
Agent Harness / Tool Use
Move the model from passive question-answering into a tool-using agent setup.
planned
November 2026
Agent Evals + Long-Horizon Tasks
Measure agent success and failure taxonomy on multi-step tasks.
planned
December 2026
Safety / Red Teaming / Refusal Quality
Measure safety behavior by quality and balance, not just refusal rate.
planned
January 2027
Reasoning Behavior + Process Evaluation
Evaluate not only final answers, but where reasoning processes break down.
planned
February 2027
Monitorability / Interpretability Start
Start analyzing model behavior with internal signals in addition to external scores.
planned
March 2027
Multimodal / UI Understanding / Computer Use
Enter multimodal model work through screenshot and UI-understanding tasks.
planned
April 2027
Training / Inference Systems Efficiency
Add systems and profiling knowledge to make research experiments more efficient.
planned
May 2027
Data Efficiency + Scaling Ladder
Measure whether better data mixtures produce better behavior with the same compute.
planned
June 2027
Final Integration + Public Portfolio
Turn the 12-month work into a showable, reproducible, publishable portfolio.
planned
Research modules
Eval Harness
Reproducible task execution, grading, metrics, and failure labels.
SFT Pipeline
Instruction data cleaning, supervised fine-tuning, and behavior comparison.
Preference Optimization / DPO
Chosen/rejected pairs, DPO training, and side-effect measurement.
Agent Harness
Tool registry, file operations, test execution, and replayable traces.
Long-Horizon Agent Evals
Multi-step coding tasks and trace-level failure taxonomy.
Safety / Red Teaming
Refusal quality, over-refusal, under-refusal, jailbreak robustness, and safe completion.
Reasoning Process Evaluation
Outcome scoring compared with observable process and self-correction signals.
Monitorability
Internal-signal probes for failure prediction and representation drift.
Multimodal UI Understanding
Screenshot QA, OCR plus reasoning, UI grounding, and visual failure modes.
Systems Efficiency
Latency, throughput, batching, KV cache, quantization, and profiling.
Data Efficiency / Scaling
Data mixtures, filtering, small scaling ladders, and score/GPU-hour.
Final Integration
Reproducible scripts, reports, dashboard, README, and portfolio packaging.
Latest public artifacts
July eval harness report
Planned report for the first eval-first gate. No public runs yet.
Eval task schema
A documented schema for small task suites and graders.
Run storage format
A reproducible structure for run ids, configs, scores, costs, latency, and caveats.
Failure mode taxonomy
A controlled vocabulary for output, grader, safety, and agent failures.
Claim boundary
Planned scaffolding only. This section tracks a public learning and research-engineering process. Planned pages and empty dashboards are scaffolding. Results will be marked as published only after the underlying run, config, dataset, and report are available.
Section index
12-month plan
The full July 2026 through June 2027 research-engineering plan.
Timeline
A compact month-by-month view of gates, themes, and report targets.
Months
Detailed monthly gates, focus areas, expected outputs, and decisions.
Reports
Planned and published report pages with explicit claim boundaries.
Evals
Task schema, initial categories, graders, metrics, and failure taxonomy.
Runs
Future run registry and run schema. No public runs yet.
Models
Future model cards and model categories. No model cards yet.
Datasets
Planned eval, instruction, preference, agent, and safety datasets.
Dashboards
Planned dashboards. No placeholder charts or synthetic metrics.
Decisions
Public decision log for naming, eval-first scope, and claim boundaries.
Glossary
Concise definitions for terms used throughout the lab section.