Open Model Lab
Open Model Lab Evals
The frozen July suite (v1) remains the first diagnostic baseline. Artifact-gated eval suite v2 becomes the comparison suite for SFT, DPO, agent, and safety experiments.
Purpose
The eval harness was the first gate because model changes are not meaningful unless they can be compared against a stable task suite, grader set, and reporting format. The July suite is intentionally small and explicit: 25 draft-quality tasks across five balanced categories.
Task schema
Condensed shape of a July task record:
{
"id": "coding_001",
"category": "coding",
"prompt": "Write a Python function that...",
"grader": "unit_test",
"expected_behavior": "The function passes all hidden tests.",
"difficulty": "easy",
"tags": ["python", "unit-test", "deterministic"],
"quality_status": "draft"
} Initial task categories
| Category | Count | Examples |
|---|---|---|
| Coding | 5 tasks | Short Python function, bug fix, unit-test pass |
| Reasoning | 5 tasks | Multi-step logic, small math, error analysis |
| Factuality | 5 tasks | Infer from given text, avoid unsupported claims |
| Instruction following | 5 tasks | Format, length, tone, constraints |
| Safety-lite | 5 tasks | Safe direction and unnecessary refusal behavior |
Eval suite v2 (planned · artifact-gated)
| Category | July v1 | v2 source (planned) | Grader |
|---|---|---|---|
| Coding | 5 tasks | HumanEval and MBPP subsets | Unit-test grader, sandboxed |
| Reasoning | 5 tasks | GSM8K test subset | Exact-match grader on the final answer |
| Factuality | 5 tasks | İzci's published English question sets with supporting passages: answerable, unanswerable, and changed-fact items | Exact match and F1, with credit for correct abstention |
| Instruction following | 5 tasks | IFEval | Rule-based grader |
| Safety-lite | 5 tasks | XSTest | Rubric grader for refusal vs. compliance |
About 400 tasks per category. Every comparison reports paired bootstrap 95% confidence intervals, computed with İzci's evaluation tooling. Generation limits are set so outputs are not truncated. Base, SFT, DPO, and agent variants are evaluated in the same runtime and precision. Each source's license is recorded in its dataset card, and eval leakage control applies to every training set.
Grader types
| Grader | Use |
|---|---|
| Unit-test grader | Executes deterministic tests for the five coding tasks. |
| Exact-match grader | Normalizes short responses and checks accepted factual answers. |
| Rule-based grader | Checks explicit formatting and instruction-following constraints. |
| Rubric grader | Applies weighted textual criteria to reasoning and safety-lite responses. |
Metrics
- Score
- Latency
- Cost
- Failure mode
- Reproducibility
The July runs recorded zero API cost because they ran locally. Local compute and energy cost were not measured, so zero recorded API cost must not be read as zero execution cost.
Project-wide failure mode taxonomy
July results used format failure, instruction miss, wrong reasoning, and grader error where applicable. The wider vocabulary below is reserved for later gates as their task types become real.
- Hallucination
- Wrong reasoning
- Instruction miss
- Format failure
- Over-answering
- Unsafe answer
- Judge uncertainty
- Looping
- Premature success
- Context loss
- Tool misuse
- Over-refusal
- Under-refusal
- Factuality drift
- Style collapse
July failure audit
The recorded grader results remain frozen. A separate manual audit reviewed all 28 failed rows to distinguish model behavior from format, grader, and runtime effects.
| Audit label | Count |
|---|---|
| Likely format or instruction failure | 11 |
| Likely grader false negative | 7 |
| Truncation or runtime related | 10 |
| Likely substantive model failure | 0 |