How Preference Ranking Works in AI Evaluation

In a preference ranking task you see a prompt and two responses, A and B, and record which is better and by how much. Pairwise comparisons like this are widely used to evaluate language models and to train them from human feedback, because people are often more consistent at comparing two answers than at scoring one in isolation.

The seven-point scale

A common format, and the one used in our labs:

The direction (A or B) is usually the easy part. The magnitude is where evaluators differ, so it deserves a deliberate process.

A process for choosing a rating

1. Read the prompt first and list what the user actually needs — including constraints such as "one line" or "don't include subdirectories".
2. Check each response for disqualifying problems: wrong answers, code that doesn't run, ignored explicit instructions, unsafe advice.
3. Compare on the remaining criteria: completeness, clarity, relevance to the user's intent, appropriate confidence, efficiency.
4. Choose the magnitude using anchors like those below.
5. Write the justification, leading with the deciding factor.

Calibration anchors

• Strongly better (1 or 7): the other response fails at the core task — e.g. its main suggestion raises an error, gives a factually wrong answer, or violates an emphasised instruction.
• Moderately better (2 or 6): both are usable, but one is clearly more helpful — it addresses the user's real problem, includes a needed caveat, or avoids a meaningful weakness.
• Slightly better (3 or 5): small differences in clarity, concision or completeness that a reasonable reader might weigh differently.
• About the same (4): differences genuinely cancel out or don't matter for this user.

Three quick examples

"How do I reverse a string in Python?" A answers with s[::-1] and a correct note that strings are immutable. B recommends s.reverse() — which raises AttributeError, because strings have no reverse() method — before offering a working loop. A is strongly better (1); "moderately" (2) is defensible because B's fallback works.

"What's the difference between == and is? My x is 1000 check behaves oddly." Both responses are correct. B explains why the user's specific check is unreliable (integer caching is a CPython implementation detail) and recommends is only for singletons like None. B is moderately better (6). Rating 7 would overstate it: A contains no errors.

"One-line command to count lines in .py files, current directory only." A gives wc -l *.py. B gives find . -name "*.py" | xargs wc -l, which recurses into subdirectories — exactly what the user said not to do — and breaks on filenames containing spaces. A is moderately to strongly better (2 or 1).

Multiple ratings can be defensible

Good rubrics often accept a range — for example "2 or 1" — because reasonable evaluators weigh the same evidence slightly differently. What matters is that:

• the direction is right;
• the magnitude is within the defensible range; and
• the justification shows why.

A rating on the wrong side of 4 is a real error. A rating one step from the reference, with a sound justification, usually isn't.

Writing the justification

A strong justification:

• leads with the decisive factor ("B's main recommendation raises AttributeError");
• cites evidence from the responses, not impressions ("more professional", "better vibes");
• acknowledges strengths of the losing response where they exist;
• avoids length bias — longer is not better unless the extra content helps the user.

A is strongly better (1). B's primary suggestion, s.reverse(), fails because str has no reverse method, and B presents it as "the most Pythonic choice". A is correct, concise and notes immutability. B's loop alternative works, which is the only reason "moderately better" might be argued.

Common pitfalls

• Rewarding confidence. Confident wrong answers should score worse than cautious right ones.
• Rewarding length. Padding, repetition and unnecessary caveats are weaknesses.
• Ignoring the prompt's constraints in favour of what you would have asked.
• Defaulting to 4 to avoid committing. "About the same" is a real judgement, not a safe harbour.

Practise

The Preference Labelling lab uses the seven-point scale, compares your rating with a reference range and shows a criterion-by-criterion breakdown.