Open Model Lab

Open Model Lab Evals

The frozen July suite (v1) remains the first diagnostic baseline. Artifact-gated eval suite v2 becomes the comparison suite for SFT, DPO, agent, and safety experiments.

Purpose

The eval harness was the first gate because model changes are not meaningful unless they can be compared against a stable task suite, grader set, and reporting format. The July suite is intentionally small and explicit: 25 draft-quality tasks across five balanced categories.

Task schema

Condensed shape of a July task record:

{
  "id": "coding_001",
  "category": "coding",
  "prompt": "Write a Python function that...",
  "grader": "unit_test",
  "expected_behavior": "The function passes all hidden tests.",
  "difficulty": "easy",
  "tags": ["python", "unit-test", "deterministic"],
  "quality_status": "draft"
}

Initial task categories

Category Count Examples
Coding 5 tasks Short Python function, bug fix, unit-test pass
Reasoning 5 tasks Multi-step logic, small math, error analysis
Factuality 5 tasks Infer from given text, avoid unsupported claims
Instruction following 5 tasks Format, length, tone, constraints
Safety-lite 5 tasks Safe direction and unnecessary refusal behavior

Eval suite v2 (planned · artifact-gated)

No v2 results exist yet. The July suite and its results stay frozen.
Category July v1 v2 source (planned) Grader
Coding 5 tasks HumanEval and MBPP subsets Unit-test grader, sandboxed
Reasoning 5 tasks GSM8K test subset Exact-match grader on the final answer
Factuality 5 tasks İzci's published English question sets with supporting passages: answerable, unanswerable, and changed-fact items Exact match and F1, with credit for correct abstention
Instruction following 5 tasks IFEval Rule-based grader
Safety-lite 5 tasks XSTest Rubric grader for refusal vs. compliance

About 400 tasks per category. Every comparison reports paired bootstrap 95% confidence intervals, computed with İzci's evaluation tooling. Generation limits are set so outputs are not truncated. Base, SFT, DPO, and agent variants are evaluated in the same runtime and precision. Each source's license is recorded in its dataset card, and eval leakage control applies to every training set.

Grader types

Grader Use
Unit-test grader Executes deterministic tests for the five coding tasks.
Exact-match grader Normalizes short responses and checks accepted factual answers.
Rule-based grader Checks explicit formatting and instruction-following constraints.
Rubric grader Applies weighted textual criteria to reasoning and safety-lite responses.

Metrics

  • Score
  • Latency
  • Cost
  • Failure mode
  • Reproducibility

The July runs recorded zero API cost because they ran locally. Local compute and energy cost were not measured, so zero recorded API cost must not be read as zero execution cost.

Project-wide failure mode taxonomy

July results used format failure, instruction miss, wrong reasoning, and grader error where applicable. The wider vocabulary below is reserved for later gates as their task types become real.

  • Hallucination
  • Wrong reasoning
  • Instruction miss
  • Format failure
  • Over-answering
  • Unsafe answer
  • Judge uncertainty
  • Looping
  • Premature success
  • Context loss
  • Tool misuse
  • Over-refusal
  • Under-refusal
  • Factuality drift
  • Style collapse

July failure audit

The recorded grader results remain frozen. A separate manual audit reviewed all 28 failed rows to distinguish model behavior from format, grader, and runtime effects.

Audit labelCount
Likely format or instruction failure 11
Likely grader false negative 7
Truncation or runtime related 10
Likely substantive model failure 0