Screening and comparison are different problems, and treating them the same is why shortlist decisions often feel arbitrary.
Screening is absolute: does this person meet the requirements, yes or no. Comparison is relative: given five people who all meet the requirements, which one do we hire? The first has a defensible right answer. The second is a trade-off, and pretending otherwise is where the trouble starts.
AI is genuinely useful at the second problem — but only if you structure it correctly, because the default behaviour is subtly broken in a way that is easy to miss.
The failure mode: convergence
Ask a language model to compare four candidates and tell you who is best, and here is what typically happens. It forms an early preference — usually driven by whichever CV reads most impressively — and then evaluates every subsequent dimension in a way that supports that preference.
Candidate A becomes the strongest technically, and also the best communicator, and also the lowest risk, and also the best culture fit. Four dimensions, one winner on all of them.
Real candidates almost never look like that. When four people clear the same bar, they are usually strong in different directions. A comparison that hands you a clean sweep has not compared anything — it has picked a favourite and written four justifications.
The reason this matters more than it sounds: the output is fluent, specific, and reads like analysis. It is much easier to accept than a vague answer would be.
Three structural fixes
1. Compare by dimension, not by candidate
The instinct is to evaluate candidate A fully, then candidate B fully, then compare the summaries. This is exactly backwards, and it is the main source of halo effects — both human and machine.
Instead, take one requirement and assess all candidates on it before moving to the next. Requirement one across all five. Then requirement two across all five.
This forces genuine comparison, because you are holding a single standard in mind and applying it repeatedly, rather than forming five overall impressions and then trying to rank the impressions.
2. Ask for different winners on different questions
Do not ask “who is best.” Ask several separate questions that can have separate answers:
- Who is strongest on the single most important requirement?
- Who has the highest ceiling if this role grows in eighteen months?
- Who ramps fastest — who could be productive soonest?
- Who carries the most risk, and what specifically is the risk?
- What does each candidate bring that nobody else on the current team has?
Then instruct it explicitly not to converge. If the same name comes back for all five, that is information — either the field is genuinely lopsided, or the analysis has collapsed and you should re-run it.
3. Require evidence for every claim
Every comparative statement should cite the specific line from a CV that supports it. This is the same rule that works for human screening, and it does the same job: it separates what the document says from what the model inferred from the shape of it.
Without it, you get confident sentences about “strong leadership experience” that trace back to nothing more than a senior-sounding job title.
A prompt that works
Compare the attached candidates for this role: [paste your 3-5 must-have requirements]
Step 1: Build a table. One row per candidate, one column per requirement. Mark each cell strong / partial / absent, and give the specific line from that CV that justifies the mark. Assess all candidates on requirement 1 before moving to requirement 2.
Step 2: Answer these separately, and treat them as genuinely independent questions:
— Strongest on the most critical requirement, and why
— Highest ceiling if the role expands, and why
— Fastest to become productive, and why
— Highest risk, and what the specific risk isDo not converge on a single favourite. If one candidate genuinely wins on multiple dimensions, say so explicitly and state what evidence makes that the case.
Step 3: List what is unknown. For each candidate, name the one thing missing from their CV that would most change your assessment.
Do not recommend a hire. Do not score candidates on anything outside the listed requirements.
Step 3 is the part most people leave out and it is often the most useful output. The gaps tell you what to ask in the interview, and they surface where the comparison is resting on absence rather than evidence.
Two AI-specific problems to control for
Order effects
Where a candidate appears in the input can influence how they are assessed. If the comparison matters, re-run it with the candidates in a different order. If the ranking changes, you have learned that the difference between them is smaller than the confident output suggested — which is itself a useful finding.
Prestige convergence
Models have learned that recognisable employers, known universities, and certain career shapes correlate with strong candidates. In screening this is a background bias. In comparison it is sharper, because when two candidates are genuinely close, the prestige signal is often what breaks the tie — invisibly, and dressed as reasoning.
The countermeasure is the evidence rule. Prestige never produces a quote from the requirements.
What AI is genuinely good at here
Worth being specific, because the useful applications are narrower than the marketing suggests:
- Normalising vocabulary. Five candidates describe similar work in five different ways across different industries. Mapping all of them onto one set of requirements is real work, and models are good at it.
- Spotting absence. Humans notice what is written; models are better at noticing what is systematically missing across a set.
- Generating differentiating questions. Given a comparison, “what should I ask each candidate to resolve the thing I’m uncertain about” is a strong use case.
- Consistency. Applying the same standard to candidate five as to candidate one, which humans do poorly at the end of a long day.
And what it should not do: make the decision. Not because models are bad at judgment, but because the inputs it lacks — team dynamics, what the hiring manager actually needs, what you learned in the interview that never reached the transcript — are usually the deciding factors. A comparison is an input to a decision, not the decision.
Where this fits in the process
Comparison is a shortlist activity, not a screening one. Running it on 200 candidates produces noise, because relative comparison only means something once everyone in the set has cleared the same bar.
The sequence that works: screen against requirements to get a shortlist, compare within the shortlist to identify trade-offs, interview to resolve the unknowns, then decide.
CVScanner — full disclosure, we build it — handles the first stage: semantic ranking against a job description, with written rationale per candidate rather than a bare score. That rationale is what makes the comparison stage tractable, because you start with evidence-backed assessments on a common standard instead of five impressions. The comparison itself, and the decision after it, stay with you. There is a free tier if you want to see what the shortlist stage looks like with reasoning attached.
Frequently asked questions
Can AI decide which candidate to hire?
It can produce a recommendation, and you should not use it as one. Beyond the practical problem — the model lacks the context that usually decides these calls — fully automated decisions with significant effects are restricted in several jurisdictions, and hiring qualifies. Use it to structure the trade-offs, then decide.
Why does AI always seem to prefer the same candidate?
Two causes. Convergence: once a model forms an early preference it tends to justify subsequent dimensions in that candidate’s favour. And prestige signals: recognisable employers and universities correlate with strong candidates in training data, so they quietly break ties. Asking separate questions with separate answers, and requiring evidence for each claim, is the practical fix.
How many candidates should I compare at once?
Three to six. Below three there is nothing to compare. Above six or so, output quality degrades and details start bleeding between candidates — and honestly, if you have twelve finalists, your screening stage did not finish.
What’s the difference between screening and comparison?
Screening asks whether one candidate meets a fixed standard — an absolute question with a defensible answer. Comparison asks which of several qualified people to choose — a relative question that resolves into trade-offs rather than a right answer. Different questions, different methods, and comparison only works once screening is done.
Is AI candidate comparison fair?
It can be more consistent than unstructured human comparison, which is a real advantage, but it is not automatically fair — it inherits the prestige patterns in its training data. Constraining it to score only against written requirements, with quoted evidence, and re-running with reordered inputs to check stability, are the practical safeguards.
Getting to a shortlist worth comparing is the harder half. Try CVScanner free and see the reasoning behind each ranking.