Purpose
Use this lesson to judge AI output against observable criteria instead of a vague impression. The aim is to produce work that a colleague can review and act on, not merely text that sounds convincing.
Lesson Explanation
Scenario: Evaluate a draft customer FAQ.
Better instruction: Score the FAQ from 1–5 for factual accuracy, completeness, tone, scannability, and policy compliance. Require a cited reason and revision for every score below 4.
A rubric turns “Is this good?” into repeatable review. Define criteria, scale, evidence required, threshold, and what happens when a response fails. Use the rubric on examples before relying on it broadly. Every low score includes a reason and a specific revision.
Work in a short cycle: provide the relevant input, state the result and limits, inspect the draft against the acceptance criteria, then revise the instruction when a requirement is missing. Keep the final human decision with the person responsible for the work.
Practice Questions
-
A prompt asks an AI to “check if this draft is good” with no defined criteria. What does this lesson say a rubric turns this kind of vague question into?
View Answer
Repeatable review – a rubric turns “Is this good?” into a defined, consistent evaluation process.
-
What five elements does this lesson say a rubric should define?
View Answer
Criteria, scale, evidence required, threshold, and what happens when a response fails.
-
Write a prompt for a rubric to evaluate customer support replies, following this lesson’s guidance.
View Answer
Something like: “Score each reply from 1-5 for accuracy, empathy, clarity, and policy compliance. Require a cited reason and suggested revision for every score below 4.”
-
A rubric assigns numeric scores but doesn’t require any explanation for low scores. What does this lesson recommend adding?
View Answer
A cited reason and revision for every score below the defined threshold.
-
What does this lesson recommend doing with a rubric before relying on it broadly?
View Answer
Using it on examples first, to test that it works as intended before applying it at scale.
-
A rubric scores a document on five dimensions but doesn’t define what score counts as passing versus failing. What is missing, per this lesson?
View Answer
A threshold – the score level that separates acceptable from unacceptable.
-
A completed rubric defines five criteria, a 1-5 scale, evidence requirements, a passing threshold, and a required revision for scores below threshold. What does this reflect?
View Answer
A complete evaluation rubric following this lesson’s full guidance.
-
Why does this lesson say a rubric should specify “what happens when a response fails”?
View Answer
Without a defined next step for failing responses, the evaluation process doesn’t actually lead to improvement or resolution – just a score with no clear action.
-
A rubric is applied to fifty documents immediately after being drafted, with no initial testing on a smaller sample. What does this lesson suggest could have been done first?
View Answer
Using the rubric on a few examples first to verify it works as intended before applying it broadly.
-
Why does this lesson emphasize requiring “a cited reason” alongside any low score, rather than just the score itself?
View Answer
A cited reason makes the evaluation specific and actionable, rather than just a number with no explanation of what needs to improve.
-
A rubric for evaluating marketing copy includes a scale from 1-5 for “on-brand voice” but no examples of what a 5 versus a 2 actually looks like. What might this lesson suggest improving?
View Answer
Testing the rubric on real examples first to calibrate what each score level should actually look like in practice.
-
A team builds a rubric, tests it on five sample documents, adjusts the criteria based on what they find, and then rolls it out broadly. What does this reflect?
View Answer
Following this lesson’s recommended process of testing a rubric on examples before relying on it broadly.