Open Model Lab
Eval Suite v2
Make the evaluation suite sensitive enough to detect post-training effects before any model is modified.
Gate status
- Execution gate
- Artifact-gated
- Status
- planned
- Report
- Results pending
Actual compute prerequisite
Versioned complete suite, selected model/runtime identities, repeatable local runner and smoke update. Prepare available subsets without waiting for an entire İzci release.
Independent code, fixtures and tests may proceed before this gate. Their readiness does not complete the measured experiment.
Success criterion
Repeated runs reproduce their scores, and per-category 95% confidence intervals are ±5 points or narrower.
Focus
- Public benchmark subsets for instruction following (IFEval), reasoning (GSM8K), coding (HumanEval and MBPP), and safety and over-refusal (XSTest).
- Context-grounded factuality from İzci's published English question sets, with the supporting passages given: answerable, unanswerable, and changed-fact items.
- About 400 tasks per category, paired bootstrap confidence intervals, and generation limits that avoid truncation.
- Built on İzci's evaluation tooling for intervals, test-split locking, judge-agreement checks, and contamination scans.
- Select a base/instruct pair with trainable open weights and evaluate both in one runtime and precision.
- LoRA smoke training to check the training pipeline end to end.
Expected outputs
- Versioned eval_v2 suite with a dataset card and recorded source licenses.
- Base and instruct baseline runs with confidence intervals.
- Report: can the suite detect effects of the size SFT and DPO are expected to produce?
Stage completion decision
Can the suite detect effects of the size SFT and DPO are expected to produce?