How to Evaluate AI-Generated Code

AI-generated code is often plausible: well formatted, confidently explained and correct for the obvious case. Evaluating it well means looking past that surface systematically. Here is the method we use in our practice labs.

The seven-step method

1. List the requirements.
2. Trace the code against concrete inputs.
3. Hunt for edge cases.
4. Check the response's claims against the code.
5. Assess complexity against the stated constraints.
6. Look for safety and side effects.
7. Write a specific, proportionate verdict.

Let's apply it to an example.

Worked example

Request: "Write a Python function median(values) that returns the median of a list of numbers. Don't modify the caller's list."

Response:

1def median(values):
2    values.sort()
3    return values[len(values) // 2]

"This sorts the numbers and returns the middle element, which is the median."

Step 1 — List the requirements

Write them down, including the ones that are easy to forget:

• return the median of a list of numbers;
• do not modify the caller's list;
• (implicitly) behave sensibly for any list the caller could pass, including even-length lists.

Step 2 — Trace concrete inputs

Pick small inputs and run them in your head (or, when allowed, for real):

• median([3, 1, 2]) → sorts to [1, 2, 3], index 1 → 2. Correct.
• median([4, 1, 3, 2]) → sorts to [1, 2, 3, 4], index 2 → 3. The median of an even-length list is the mean of the two middle values: 2.5. Incorrect.

Step 3 — Hunt for edge cases

Useful defaults for list-processing code: empty input, one element, two elements, duplicates, negative numbers, very large inputs.

• median([]) → values[0] raises IndexError: list index out of range. The request didn't specify empty-list behaviour, so the evaluation should note it rather than treat any particular choice as mandated. Raising a clear ValueError (or matching statistics.median, which raises StatisticsError: no median for empty data) would be better than an accidental IndexError.

Step 4 — Check claims against code

The explanation says it "returns the middle element, which is the median". That is only true for odd-length lists. It also says nothing about modifying the input — but values.sort() sorts in place, so the caller's list is changed. That directly violates a stated requirement.

Step 5 — Complexity

Sorting is O(n log n); the median can be found in expected O(n) with a selection algorithm, but for most uses sorting is a perfectly reasonable choice. Don't penalise a sensible trade-off just because a theoretically faster algorithm exists — judge against the stated constraints.

Step 6 — Safety and side effects

Here, the side effect is the bug: mutating the input. In other tasks, look for unvalidated input reaching a shell or SQL query, secrets in code, or broad exception handlers that hide errors.

Step 7 — Write the verdict

A weak verdict: "Wrong, doesn't work."

A strong verdict:

Incorrect. (1) Violates the explicit requirement not to modify the input: values.sort() sorts the caller's list in place. (2) Returns the wrong result for even-length lists: median([4, 1, 3, 2]) returns 3, but the median is 2.5. (3) An empty list raises IndexError; a clear ValueError would be better. The explanation's claim that it returns "the median" is only true for odd lengths. Fix: use sorted(values) and average the two middle elements when the length is even.

A corrected version:

1def median(values):
2    if not values:
3        raise ValueError("median() of an empty list")
4    ordered = sorted(values)
5    mid = len(ordered) // 2
6    if len(ordered) % 2:
7        return ordered[mid]
8    return (ordered[mid - 1] + ordered[mid]) / 2

In production you would usually just call statistics.median.

What makes a verdict useful

• Lead with the conclusion (correct / partially correct / incorrect), then the reasons in order of severity.
• Cite evidence: an input and its actual versus expected output, or the exact requirement violated.
• Separate requirements from preferences. "Should use a list comprehension" is style; "mutates the input" is a requirement failure.
• Don't invent problems. If the code is correct, say so. Over-criticising is as unhelpful as missing a bug.
• Stay within the evidence. If behaviour depends on something not shown (a library version, an environment), say what you would need to check.

Practise it

The Response Evaluation lab gives you real requests and AI answers with hidden defects, then compares your verdict and failing tests with a model evaluation. The free Code Comparison lab practises steps 2–5.