Model Grader
A model grader uses an AI model to score or label an output with a written rule. It can be wrong, so people must check it against known examples.
output + rubric + grader → score + reviewrubric: correct · cited · escalateA team grades 200 support replies with one rubric. People review disagreements and compare the grader with answers they already scored.0.74 / 1.00needs 0.8027 / 30 agreeitem + output + rubric + grader model → score or labelOPSAnother example
A team grades 200 support replies with one rubric. People review disagreements and compare the grader with answers they already scored.
Main parts
- 01Evaluation item
- 02Candidate output
- 03Scoring rubric
- 04Reference or examples
- 05Grader model and prompt
- 06Score, reason, and review
Use it when
Use it when exact matching cannot judge many open-ended answers. First test the grader against human-reviewed examples.
Do not use it when
Do not call its score objective truth. Hide the target verdict and clues about which system wrote the answer.
Name used in code
item + output + rubric + grader model → score or label Before you ship
Check the model and prompt versions, source data, versioned evaluation set, and measures. Verify links to evidence, tool permissions, privacy, common failures, decline-to-answer and fallback behavior, human review, monitoring, cost, speed, and rollback.
Outcome: Use or evaluate Model Grader to make AI behavior measurable and tied to evidence. User context: A team grades 200 support replies with one rubric. People review disagreements and compare the grader with answers they already scored. AI method or concept: Model Grader. Why it fits: Use it when exact matching cannot judge many open-ended answers. First test the grader against human-reviewed examples. Do not use it when: Do not call its score objective truth. Hide the target verdict and clues about which system wrote the answer. AI requirements: Define the input and source evidence. Set model and tool permissions. Use versioned evaluation data and measures. Define failure, decline-to-answer, privacy, speed, and cost limits. Operational safeguards: Keep a clear trace and hide sensitive log data. Show users a safe fallback. Mark steps that need human review. Define how to roll back the model or prompt. Acceptance criteria: Record baseline and target measures on a versioned evaluation set. Test edge cases and hostile inputs. Verify fallback, monitoring, permissions, and rollback. Evidence and limits (evidence boundary): No single official standard controls the whole term. Its meaning may change across teams, platforms, or frameworks. Unknowns to confirm: Target task, model and version, evaluation owner, source data, risk limit, tool permissions, and production fallback.
Check this request
Review the current use of Model Grader. Definition: A model grader uses an AI model to score or label an output with a written rule. It can be wrong, so people must check it against known examples. Release checks: Check the model and prompt versions, source data, versioned evaluation set, and measures. Verify links to evidence, tool permissions, privacy, common failures, decline-to-answer and fallback behavior, human review, monitoring, cost, speed, and rollback. Before changing code, report the evidence you found, gaps, severity, and the smallest safe fix.