Open Model Lab
Open Model Lab Evals
The frozen July suite provides the first diagnostic baseline for later SFT, DPO, agent, safety, monitorability, and systems experiments.
Purpose
The eval harness was the first gate because model changes are not meaningful unless they can be compared against a stable task suite, grader set, and reporting format. The July suite is intentionally small and explicit: 25 draft-quality tasks across five balanced categories.
Task schema
Condensed shape of a July task record:
{
"id": "coding_001",
"category": "coding",
"prompt": "Write a Python function that...",
"grader": "unit_test",
"expected_behavior": "The function passes all hidden tests.",
"difficulty": "easy",
"tags": ["python", "unit-test", "deterministic"],
"quality_status": "draft"
}Initial task categories
| Category | Count | Examples |
|---|---|---|
| Coding | 5 tasks | Short Python function, bug fix, unit-test pass |
| Reasoning | 5 tasks | Multi-step logic, small math, error analysis |
| Factuality | 5 tasks | Infer from given text, avoid unsupported claims |
| Instruction following | 5 tasks | Format, length, tone, constraints |
| Safety-lite | 5 tasks | Safe direction and unnecessary refusal behavior |
Grader types
| Grader | Use |
|---|---|
| Unit-test grader | Executes deterministic tests for the five coding tasks. |
| Exact-match grader | Normalizes short responses and checks accepted factual answers. |
| Rule-based grader | Checks explicit formatting and instruction-following constraints. |
| Rubric grader | Applies weighted textual criteria to reasoning and safety-lite responses. |
Metrics
- Score
- Latency
- Cost
- Failure mode
- Reproducibility
The July runs recorded zero API cost because they ran locally. Local compute and energy cost were not measured, so zero recorded API cost must not be read as zero execution cost.
Project-wide failure mode taxonomy
July results used format failure, instruction miss, wrong reasoning, and grader error where applicable. The wider vocabulary below is reserved for later gates as their task types become real.
- Hallucination
- Wrong reasoning
- Instruction miss
- Format failure
- Over-answering
- Unsafe answer
- Judge uncertainty
- Looping
- Premature success
- Context loss
- Tool misuse
- Over-refusal
- Under-refusal
- Factuality drift
- Style collapse
July failure audit
The recorded grader results remain frozen. A separate manual audit reviewed all 28 failed rows to distinguish model behavior from format, grader, and runtime effects.
| Audit label | Count |
|---|---|
| Likely format or instruction failure | 11 |
| Likely grader false negative | 7 |
| Truncation or runtime related | 10 |
| Likely substantive model failure | 0 |