The Subject-Expert Fallacy

A founder I worked with, an engineer of real brilliance, the kind who had personally written the code his company still ran on, once insisted on leading the interviews for his own successor as CTO. It was hard to argue with the logic. Who could possibly judge a chief technology officer better than the person who had built the technology?

The interviews were, by his account, excellent. With one candidate in particular the conversation had taken flight: two hours gone in a moment, deep in distributed-systems architecture, trading war stories about database migrations, finishing each other's sentences about a framework they had both suffered under. "Finally," the founder said afterward, "someone who speaks my language." A second candidate had been slower, almost plodding. She kept steering away from technical depth toward questions about the engineering organization, delivery predictability, how decisions got made. The founder found her conversation "fine, a bit managerial." The fluent one was hired.

You can write the next paragraph yourself, and you would be right. Within a year it was clear the new CTO was a superb architect who could not run an organization: releases slipped, senior engineers left, and the board conversation turned to the very weaknesses the "managerial" candidate had been quietly probing for in her own interview. She, having actually run engineering organizations through scale, knew where CTO roles die. The founder had run a wonderful conversation and a poor assessment, and the two had felt identical from the inside.

This chapter is about why that happens: not occasionally, not to careless people, but systematically, to the smartest person in the room, because they are the smartest person in the room on the wrong subject. It is the most contrarian claim in this book, so let me state it carefully, defend it properly, honor its limits, and then turn it, in fairness, on my own profession.

The claim, stated carefully

Here is what this chapter does not say. It does not say expertise is bad, that technical depth doesn't matter in a CTO or a CFO, or that ignorance is an interviewing qualification. Candidates must be technically excellent; someone must verify that they are; and domain knowledge has indispensable seats in the process, which we will name precisely.

The claim is about one specific chair: the interviewer's. It runs like this: beyond a threshold of functional literacy, subject-matter expertise in the interviewer stops adding assessment accuracy and begins subtracting it — through mechanisms that are documented, predictable, and invisible from the inside. The expertise that actually improves candidate assessment is a different expertise: the practiced, outcome-corrected craft of assessing candidates in a given market. A former CFO interviewing CFO candidates holds the first kind. A specialist who has assessed two hundred CFO candidates in that sector, placed thirty of them, and lived with the results holds the second. Our industry, and most hiring organizations, instinctively trust the first chair more. The evidence points the other way.

The intuition this claim offends is genuinely reasonable, which is why it survives everywhere: it takes one to know one. Only a pilot can judge a pilot; only a surgeon, a surgeon. And for verifying technical knowledge, the intuition is roughly right, which is exactly the trap. At senior level, technical knowledge is the threshold, not the differentiator (Chapter 1 made that case with the failure data), and the expert interviewer is drawn, by gravity, into assessing the threshold over and over, magnificently, while the differentiator sits unexamined. The founder spent two hours confirming that his candidate knew distributed systems. He learned nothing about whether the man could run one hundred and forty engineers. He wasn't measuring the wrong register out of negligence. He was measuring the register his expertise could see.

Six mechanisms

What follows are the six documented ways interviewer expertise degrades assessment. As you read them, you may recognize the founder in several. You may also recognize yourself, which is the uncomfortable, useful part. I recognize myself in at least three.

First: experts abandon structure. Chapter 2 established that structure is where interview validity lives. The evidence adds a bitter twist: confidence in one's own judgment is precisely what licenses departing from structure, and expertise manufactures confidence. Scott Highhouse's review of selection practice gave the phenomenon its blunt name, a "stubborn reliance on intuition and subjectivity", and the pattern is worst among those with the most standing to trust themselves. The expert doesn't skip the protocol out of laziness; the protocol feels beneath the occasion. "We're peers — let's just talk." And with that warm sentence, the best-validated instrument in the field is traded for the worst.

Second: experts drown their own signal. Dana, Dawes, and Peterson demonstrated experimentally what they called the illusion of validity in unstructured interviews: given valid background information plus an unstructured conversation, judges predicted outcomes worse than judges given the background information alone, yet felt more confident. The conversation added noise, and the human mind, expert minds emphatically included, wove the noise into a coherent story. In their most unsettling condition, interviewers rated conversations with candidates who were answering randomly as informative; sensemaking machinery built a person out of static. The expert's rich mental models make this worse, not better: more material with which to construct a compelling, confident, wrong narrative.

Third: shared jargon creates an illusion of understanding — in both directions. Cognitive scientists call the general phenomenon the illusion of explanatory depth: fluency with a vocabulary feels like comprehension of the thing. In an interview between two insiders, the vocabulary is shared, so the illusion is shared. When the candidate says the right words, and every weak senior candidate has the words, because the words are the cheapest thing about the job, the expert interviewer's own knowledge silently completes the reasoning behind them. He hears his own understanding and attributes it to the candidate. Note the cruel geometry here, because it is the deepest point in this chapter: fluency in the field's language is exactly what the marginal senior candidate possesses most of, and the expert interviewer is exactly the audience most disposed to mistake that fluency for competence. A thoughtful non-expert, unable to fill the gaps, has to ask the question the expert never asks: "Walk me through what you actually did." Readers of the research corpus behind this book will recognize the pattern, fluent output received as understanding, from a much larger contemporary argument. Humans invented it long before machines industrialized it.

Fourth: experts assess by resemblance to themselves. Similarity attraction is among the most robust findings in hiring research, and expertise supplies a rich similarity surface: shared career shape, shared battles, shared aesthetics of the craft. The expert carries a prototype of excellence, and the prototype has his own face; the literature calls it constructing merit in one's own image. So the candidate who came up the way I came up, who solves problems the way I solve them, reads as excellent, while excellence in an unfamiliar shape reads as "fine, a bit managerial." Two distortions for the price of one: the mirror-candidate is overrated, and, peer dynamics being what they are, occasionally the reverse, when the expert unconsciously interviews a rival rather than assesses a candidate. Either way, the score is about the resemblance, not the role. And recall from Chapter 1 what the role's world actually rewards: the founder's prototype was built for a company of twelve; the mandate was a company of four hundred.

Fifth: experts talk. The most prosaic mechanism, and in the transcripts often the most damaging. Expertise wants to be exercised; the expert interviewer explains, riffs, debates, demonstrates, and enjoys, in the fullest sense, the conversation. The founder's best interview was the one where the interviewer spoke half the time. Every minute of interviewer speech is a minute of evidence not collected, and worse: it converts the interview into a duet, which the socially skilled candidate gratefully joins, harmonizing rather than disclosing. The plodding candidate who kept asking about the organization was, in fact, the only person in either room collecting evidence.

Sixth: experts test recall. When the expert does drive the questioning, gravity pulls toward what expertise can grade: knowledge questions. What would you do about X, how does Y work, defend framework Z. These feel rigorous, and they measure the wrong register: knowledge, which at this level is threshold and Google-adjacent, rather than capability-in-role, which lives in verified episodes of organizational action. Job-knowledge quizzing has real validity for jobs where knowledge is the job. Two levels above that, it assesses the candidate's conference-circuit self.

What expertise actually is — and why the recruiter's kind counts

Underneath all six mechanisms sits one clean idea from the judgment literature, and it reframes the whole question. Kahneman and Klein, in their famous adversarial collaboration on intuitive expertise, converged on two conditions under which real intuitive skill develops: a sufficiently regular environment, and prolonged practice with prompt, clear feedback. Chess players get both. Firefighters get both. Anesthesiologists get both. Political pundits, notoriously, get neither, with Tetlock's expert forecasters underperforming simple rules, and neither do most interviewers.

Now apply the two conditions honestly to our two candidates for the interviewer's chair.

Having done the job builds superb intuitions about the job, through thousands of practiced, feedback-corrected cycles of doing it. It builds no intuitions about assessing candidates, because the executive's career contains perhaps a dozen serious hiring judgments, with feedback arriving years later, confounded by everything, and rarely traced back to the interview that produced them. Interviewing, for the subject-matter expert, is a low-volume, feedback-starved side activity: precisely the environment in which the literature says confident intuition grows without corresponding accuracy. The confidence is real. The calibration never had a chance to form.

Vertical recruitment experience, and here is the definition this book will use, is the craft built from repeated, outcome-corrected assessment cycles concentrated in one market: hundreds of structured interviews against known mandates; placements tracked into their second and third years; the sector's title-to-scope map learned the hard way (what "Managing Director" means at that company versus its competitor); compensation reality; which celebrated transformations were actually driven by whom; where the market's bodies are buried. This is not knowing the job. It is knowing what claims about the job mean, and what became of the people who made them — a regular environment, high volume, real feedback. It meets the Kahneman–Klein conditions. Doing the job, for all its genuine glory, does not.

That is the asymmetry hiding under "it takes one to know one." Judging performances and giving them are different crafts. The sports world has always known that great players rarely make great scouts, and scouting, not playing, is the craft the interviewer's chair requires. (Readers of Executive Search as Erfahrung will recognize this as that book's formation argument wearing work clothes: judgment is built by corrected undergoing in the practice of judging itself — there is no borrowing it from an adjacent excellence.)

The boundary conditions — where the expert is irreplaceable

Now the other side, at full strength, because the evidence gives the domain expert real and non-negotiable seats, and a version of this chapter that denied them would be as miscalibrated as the fallacy it corrects.

Experts must design and score the work samples. A strategy case or technical exercise built by a non-expert tests trivia or, worse, tests the wrong hard things. Only someone who has done the work knows which ambiguities are the real ones, what a distinguished answer contains, and what impressive-sounding nonsense looks like. Chapter 6's simulations are unbuildable without the expert at the drafting table.

Experts detect bluff — when pointed at it deliberately. The same knowledge that fills gaps charitably in open conversation becomes a precision instrument inside structure: a scripted technical deep-dive, designed in advance to distinguish depth from vocabulary, is where the expert's ear genuinely outperforms everyone's. The difference between this and the founder's two-hour duet is the difference between a laboratory and a dinner party. Same expertise, opposite epistemics.

Experts must brief and debrief the technical referees. A referee's claim that the candidate "owned the replatforming" yields its truth only to questions an insider knows to ask.

Experts scope the role. Before any candidate exists, someone must translate the mandate into actual technical demands, which is where Chapter 3's scope audit begins.

And experts confer legitimacy. A serious senior candidate expects to meet genuine peers; a process with no technical gravitas loses credibility with exactly the candidates it most wants. This is real, and the chair map below honors it.

Notice what every legitimate seat has in common: the expert operates inside structure, on designated targets, as a specialist instrument — and the synthesis, the weighing of all evidence into a judgment, sits elsewhere. That is the chapter's whole practical teaching in one sentence: buy technical literacy; do not give it the chair. Decouple the expertise from the decision.

The mirror: our conflict, and what answers it

It would be convenient to end there, with the client's expert gently reseated and the professional assessor installed in the deciding chair. But this book is written from inside executive search, and the evidence has something to say about our chair too. Fair is fair.

The sociology of our industry is not flattering. Rakesh Khurana documented how search processes helped inflate the market for charismatic saviors, candidates optimized for the boardroom performance our processes stage. Bidwell, Choi, and Fernandez-Mateo, studying managerial careers, found that moves made through search firms showed no evidence of better worker–employer fit than moves made through other channels, a direct empirical challenge to our core professional claim. And the incentive analysis writes itself: a contingent-fee intermediary is paid on placement, and the candidate who places, polished and presentable and immediately convincing, is not reliably the candidate who performs. Our industry's codes of conduct, like the AESC's, require conflicts to be managed and disclosed; codes are governance around an incentive, not the incentive's disappearance. The recruiter's chair has its own gravitational pull, toward charisma, speed, and the placeable: the same pull, in commercial form, that bends the expert toward jargon and the board toward polish.

So the argument of this chapter, honestly completed, is not "trust the recruiter instead of the expert." It is: trust no chair on its authority — including ours — and make every chair answerable to evidence and outcomes. For the expert, that means structure and designated targets. For the assessor, it means the calibration disciplines of Chapter 25: outcome tracking, prediction logs, being corrected in writing by one's own placements. And for the search firm as an institution, it means putting the fee where the claim is. The Proof-First model this book's publisher runs, with validated shortlist evidence delivered and tested before the main placement fee, exists precisely because assessment claims from a conflicted chair should be paid for the way all claims should be believed: after the evidence. Whatever firm you work with, the principle travels. Ask how the assessor's judgment gets corrected, and ask what they are paid for: the placement, or the proof.

Where the rules run out

The honesty paragraph. The six mechanisms rest on strong, replicated literatures, but most of that evidence concerns interviewers, experts, and judges in general; direct head-to-head studies of domain-expert versus assessment-expert interviewers, with executive outcomes as the criterion, barely exist. The chapter's thesis is a strongly supported triangulation, not a settled meta-analytic fact, and it could be tightened or complicated by research that has not yet been done. The boundary conditions cut both ways as well: in deeply technical individual-contributor hiring, further from this book's altitude, the expert's chair grows larger. And nothing here licenses the opposite fallacy, the assessment professional with no functional literacy at all, charming the client with process while missing the substance. The threshold in "beyond a threshold of functional literacy" is real and must actually be met. The claim is about which craft belongs in which chair, not about which profession owns the truth.

Instrument: the chair map

Five chairs in every serious senior search. The map says who sits where and what each chair may and may not do. Pin it to the process design; revisit it when a strong personality, client or consultant, starts drifting between chairs.

Three standing rules complete the map. No chair grades its own homework: whoever designed the exercise does not make the hiring recommendation alone. Chair 3 reports signals, not verdicts: "she scored 4 of 5 on the architecture case; here is the evidence" — never "hire her." And every chair's predictions are logged: when the placement's second anniversary arrives, the file is reopened and every chair, this book's readers included, finds out how they did. That last rule is the one that, over the years, quietly builds the only expertise the interviewer's chair has ever respected.

Notes and sources

Evidence grades: [M] meta-analytic/systematic; [L] peer-reviewed primary; [S] credible practitioner/survey; [T] triangulated inference from adjacent literatures, flagged as such.

• Stubborn reliance on intuition; resistance to structured methods among experienced practitioners: Highhouse, "Stubborn Reliance on Intuition and Subjectivity in Employee Selection" (Industrial and Organizational Psychology, 2008). [L]
• Illusion of validity; dilution; sensemaking of random answers: Dana, Dawes & Peterson, "The Illusion of Validity in the Unstructured Interview" (Judgment and Decision Making, 2013). [L]
• Illusion of explanatory depth: Rozenblit & Keil (2002); curse-of-knowledge literature. [L/T] (Application to shared-jargon interviewing is a flagged inference.)
• Similarity attraction and cultural matching; merit in one's own image: Byrne's paradigm; Rivera, "Hiring as Cultural Matching" (American Sociological Review, 2012). [L]
• Interviewer talk-time and evidence loss: interview-process research tradition; structured-interview component studies (Campion et al., 1997). [L/T]
• Job-knowledge testing validity and its level-dependence: the updated selection-validity matrix (Sackett et al., 2022). [M]
• Conditions for intuitive expertise: Kahneman & Klein, "Conditions for Intuitive Expertise: A Failure to Disagree" (American Psychologist, 2009); expert-judgment underperformance: Meehl's actuarial tradition; Tetlock, Expert Political Judgment (2005). [L/M]
• Search-firm placement fit: Bidwell, Choi & Fernandez-Mateo, "Brokered Careers" — no better fit through search-firm moves. [L]
• Charismatic-savior dynamics: Khurana, Searching for a Corporate Savior (2002). [L]
• Industry conflict governance: AESC Code of Professional Practice. [S]