Evidence, Not Impressions

You have sat in this meeting. Three interviews are done, the candidate has left the building, and the panel gathers to compare notes. Someone says, "He's clearly the guy." Someone else agrees: "Very impressive. Great energy. You can tell he's operated at scale." A third voice adds the clincher: "And culturally, I just think he fits." Heads nod. The meeting is warm, confident, and short.

Now notice something about that conversation, gently, because we have all led it. Not one sentence in it is evidence. Impressive is a report about the interviewers. Great energy is a report about ninety minutes of performance. You can tell is precisely what you cannot do. He fits is a feeling wearing the costume of a finding. The meeting produced a verdict without ever producing the material a verdict is made of, and it felt rigorous, because everyone was experienced, everyone was sincere, and everyone agreed.

This chapter is about what to put in that room instead. It is the most technical chapter in the book, and I want to be honest about that upfront: there is a table coming, and the table has decimals. But stay with it. The table is the closest thing our field has to a map of what actually works, it was redrawn quite recently, and the redrawing is good news for people who interview for a living. Everything else in this book stands on it.

What a validity number is actually telling you

When researchers say a selection method "has a validity of .42," they mean the method's scores correlate at .42 with how people actually perform in the job later. The scale runs from 0, where the method tells you nothing, to 1.0, where the method is a crystal ball. Nothing in hiring gets near 1.0. Nothing in any human prediction business does.

Here is a friendlier way to hold these numbers. Imagine two candidates, and your method scores one of them higher. With a validity of .42, the higher scorer really is the better performer somewhere around two times out of three. At .19, you are barely ahead of a coin flip. That is the whole game, and it is worth feeling the force of both halves of it. The difference between .42 and .19 is the difference between a professional practice and a ritual. And even .42 leaves a great deal unpredicted, which is not a reason for despair but a reason for two things this book will keep returning to: stacking methods that see different things, and holding every conclusion with the humility of someone who knows the map is not the territory.

One more piece of honest bookkeeping before the map itself. These are averages across studies, mostly of jobs that are not CEO jobs, mostly measured against supervisor ratings. Executive work strains this literature in specific ways, and we will flag each one as it matters. The numbers are a compass, not a GPS. But a compass, used honestly, beats confident wandering, and confident wandering is the industry default.

The table was redrawn, and the new version is better news than the old

For a quarter of a century, the field's map was the famous 1998 synthesis by Schmidt and Hunter, and its headline was that general cognitive ability was the king of predictors, with everything else arranged beneath it. If you did your reading in the 2000s, that is the table you learned.

In 2022, Paul Sackett and colleagues went back through the machinery and found a systematic problem: the classic figures had been overcorrected. The corrections in question were adjustments for "range restriction," the statistical narrowing that happens because you only observe performance for the people you actually hired, and they had been applied in ways the underlying data could not justify, inflating many estimates. When the corrections were redone conservatively, the whole table shifted downward, but not evenly. Cognitive ability fell hard. The structured interview fell least — and ended up on top.

Two things about this revision deserve a working assessor's attention. First, it is a rare and encouraging case of a scientific field publicly marking down its own most famous numbers because the method demanded it, exactly the discipline this book asks of you with candidates. Second, the practical conclusion is genuinely warm news for our profession: the best-validated instrument in personnel selection is not a test, an algorithm, or a proprietary battery. It is a well-run conversation — structured, behavioral, anchored, disciplined. The craft you already practice is the top of the table. The rest of this book is about deserving that ranking, because the same instrument, run without structure, sits near the bottom.

The map, read for executive work

The table below is the book's spine. The middle columns give the classic and revised estimates; the last column is what each line means at the altitude where you and I work.

Three lines deserve a moment of quiet, because together they indict the standard executive process. The standard process is: read the CV (heavily weighting years of experience, .07), hold a series of unstructured conversations (.19), and close with candidate-supplied references (.26 as practiced). It is possible to assemble a worse toolkit from this table, but you would have to try. Meanwhile the top of the table, meaning structure and verified evidence and disciplined referencing, is available to anyone willing to prepare. No proprietary technology, no license fee. Preparation and discipline.

And one line deserves defending from misreading. Work samples fell from .54 to .33, and I have heard people wave this as "simulations don't work." That is not what happened. What happened is that the old estimate was built on studies that called nearly anything a work sample. A well-designed executive simulation, a strategy case on the actual mandate, a board presentation under real challenge, remains one of the few ways to observe enacted judgment rather than hear it described, and it adds predictive signal over interviews precisely because it sees different things. The correction teaches design honesty, not abandonment.

What stacks, and what merely repeats

Validity numbers describe methods one at a time. Real processes combine them, and here the question changes from "how good is this instrument?" to "what does this instrument see that my others don't?"

The evidence gives a clear principle: combine methods that look at different registers through different channels, and stop duplicating the same channel. A structured interview adds a great deal to a track record, because narrative and record cross-check each other. A structured reference adds to both, because it sees the candidate from an angle no interview can reach: how they actually treated people when no assessor was in the room. A targeted work sample adds again, enacted behavior set against described behavior. A personality instrument adds a little, honestly used, mostly by sharpening what the encounters should probe.

Now recall Chapter 1's failed CFO: three interviews, all about experience. In combination terms, that process bought the same information three times. Three unstructured passes through one register, three samples of the same charisma, and zero passes through everything else. The redundancy problem is the quiet killer of executive assessment design: panels feel thorough because many people were involved, while the evidence base remains one channel wide. When this book's process architecture assigns different interviewers different registers with different methods (Ch. 19), it is applying this principle. Coverage beats repetition.

The other half of the problem: the judge

Everything so far concerns the instruments. But the debrief scene that opened this chapter fails at a second point: even good evidence is then combined in someone's head, and heads are noisy.

The judgment literature, to which Kahneman and colleagues gave the blunt modern name noise, shows that professionals given identical information reach disturbingly different conclusions, and the same professional reaches different conclusions on different days. Interviews amplify this with a specific pathology: impressions form in minutes, and the remaining hour is quietly spent gathering confirmation. None of this reflects badly on anyone's sincerity or intelligence. It is simply what unstructured human judgment does, and experienced judges are not exempt. Chapter 4 has uncomfortable news about exactly that.

The defenses are procedural, cheap, and proven, and this book will build them in wherever judgments get made: assessors rate independently, in writing, before any discussion (the first voiced opinion anchors the room); evidence is recorded per answer, not recalled at the end; comparisons are made against anchored standards, not against the glow of the last candidate; and the final synthesis is delayed and disciplined — held until all the evidence is in, then assembled by rules agreed in advance. Chapter 23 will go further, into findings about mechanical combination that most practitioners find genuinely startling, and into how an owned, defensible judgment coexists with them. For now, one sentence carries what matters: structure is not the enemy of judgment; it is the condition under which judgment gets something worth judging.

What counts as evidence: four tests

Let me end where the debrief began, with the language in the room, because the fastest cultural upgrade available to any hiring organization is to change what is allowed to be said in that meeting.

An item earns the name evidence when it passes four tests. Specificity: it names an episode, a decision, a number, a behavior — something that happened, not a quality someone radiates. Ownership: it is attached to what this person actually did, decided, or was responsible for — not to their team, their market, or their era. Verifiability: it could in principle be checked — against a record, an artifact, a reference who was there — and the best process plants those checks deliberately. Relevance: it bears on this mandate's actual demands, not on general magnificence.

"Impressive" fails all four. "In a recent restructuring she personally made the call to close a loss-making plant against her chairman's preference, and her former CFO confirms both the call and the fallout" passes all four, and notice it took one sentence, not a dossier. Evidence is not slower than impressions. It is just built from different material.

So install one rule in the debrief room, warmly and without exception, including for the most senior voice present: every claim about a candidate must be followed by its evidence, and claims that have none are parked, visibly, as hypotheses to test — not conclusions to share. You will be surprised how much of the usual conversation gets parked. That is the point. The parked list is your remaining assessment plan.

Instrument: the evidence hierarchy for one executive search

A one-page ordering to pin above the process design, integrating the table with the combination principle:

1. Verified track record — the scope audit and attribution analysis of Chapter 3; the foundation everything else cross-checks against.
2. Structured behavioral interviews, by register — different interviewers, different registers, the four-layer probe architecture; the top of the validity table, earned through preparation.
3. A customized work sample or simulation for finalists — where enacted judgment matters and the mandate justifies the cost.
4. Structured, comparative, episode-anchored references — the free upgrade: from .26 as practiced to ~.42 done properly.
5. Targeted psychometrics as prompt-sharpeners — hypotheses for the encounters, never verdicts.
6. Unstructured impressions — recorded honestly for what they are: data about chemistry, useful for exactly one question (what will this person's presence feel like?), and banned from carrying a hiring conclusion alone.

And the debrief rule, on one line: claims carry evidence, or they park as hypotheses.

Where the rules run out

The honesty paragraph, as promised, because this chapter's map has real edges. Almost all the validity evidence is concurrent, measured on incumbents rather than on candidates followed over years, which likely flatters some estimates. The criterion problem bites hardest at the top: "job performance" for a CEO is genuinely hard to define, entangled with markets, boards, and luck, so a correlation with "performance" means less the higher you go. Executive-specific samples are thin throughout; most of what we know is extrapolated upward from managerial populations. And the 2022 corrections, while clearly sounder than what they replaced, are themselves debated at the margins: the table will move again, and a book that lives by evidence should expect and welcome that. None of these caveats rescues the unstructured interview or the years-of-experience heuristic; the gap between the top and bottom of the table is far wider than the error bars. But they do mean the numbers deserve to be used the way this book uses them: as an ordering of methods and a discipline of humility, not as false precision.

Notes and sources

Evidence grades: [M] meta-analytic/systematic; [L] peer-reviewed primary; [S] credible practitioner/survey; [A] audited claim.

• The revised validity hierarchy and the overcorrection diagnosis: Sackett, Zhang, Berry & Lievens, "Revisiting Meta-Analytic Estimates of Validity in Personnel Selection" (Journal of Applied Psychology, 2022); follow-up matrix analyses (Berry, Lievens, Zhang & Sackett, 2024). [M]
• The classic table: Schmidt & Hunter, "The Validity and Utility of Selection Methods in Personnel Psychology" (Psychological Bulletin, 1998). [M]
• Work-sample correction: Roth, Bobko & McFarland (Personnel Psychology, 2005); adopted in the updated syntheses. [M]
• Structured vs. unstructured interview validity and the confirmation dynamic: McDaniel et al. (1994); Campion, Palmer & Campion (1997); Levashina et al. (2014). [M]
• Structured reference validity (~.42 corrected, comparative formats): McCarthy & Goffin (2001); Zimmerman, Triana & Barrick (Human Performance, 2010); web-based multisource evidence, Hedricks, Robie & Oswald (2013). [L]
• Noise, first-impression anchoring, and decision hygiene: Kahneman, Sibony & Sunstein, Noise (2021) and underlying selection studies. [M/S]
• Incremental validity of method combinations: the updated meta-analytic matrix literature (Sackett and colleagues, 2022–2024). [M]
• Executive-level range restriction and criterion ambiguity: Kaplan, Klebanov & Sorensen (2012) for the executive-assessment context; methodological discussions in the Sackett revision literature. [L/M]