Open Model Lab

Open Model Lab Report Template

A compact template for reading and writing Open Model Lab reports: what is measured, what is frozen, what is audited, and what the report is allowed to claim.

Purpose

Open Model Lab reports are evidence records, not leaderboard pages. A report should make the measurement setup inspectable, preserve recorded outcomes, audit failures separately, and state the narrow claim the evidence can support.

Reports Evals Runs

Required sections

Section	Purpose
Research question	The narrow question this report can answer.
Eval identity	Dataset version, task suite, graders, prompts, model artifacts, and runtime settings.
Recorded results	Scores, pass/fail states, latency, tokens, cost fields, and grader metadata.
Failure audit	Manual labels that qualify recorded outcomes without overwriting them.
Known limitations	Measurement caveats, task-quality limits, runtime effects, and missing evidence.
Claim boundary	What the report does and does not establish.
Next decision	The concrete gate, experiment, or repair the result supports.

How to read scores

Pass rate is local

A pass rate applies to the named suite, model artifact, runtime, and grader set. It is not a general model-quality ranking.

Audit labels are separate

Manual audits explain recorded outcomes. They do not silently replace scores after the result is known.

Runtime effects count

Token limits, timeouts, extraction rules, and serving behavior can materially affect a recorded result.

Failure audit labels

Label	Use
Format or instruction failure	The output violated the requested interface or final-answer shape.
Grader false negative	The grader likely rejected an otherwise usable or correct output.
Truncation or runtime related	The fixed runtime settings materially affected the recorded failure.
Substantive model failure	The output appears wrong under the task definition, not only the grader protocol.
Uncertain / needs follow-up	The evidence is not enough to assign a stronger label.

When to create a new eval identity

A baseline can be improved later, but material changes should create a new identity instead of rewriting old results.

Task text, expected behavior, or dataset membership changes.
A grader, rubric, parser, or output-extraction policy changes.
Prompt template, system instruction, or answer-format policy changes.
Model artifact, quantization, serving backend, or runtime configuration changes.
Generation settings change, including token limit, temperature, seed, or timeout.

Claim boundary checklist

Before publication: state that the report is not a leaderboard, does not establish general model quality, applies only to the named suite/config/runtime, and keeps audit labels separate from recorded grader outcomes.