A key purpose of interview questions is to differentiate between great and poor candidates. In other words, if you go through a series of interviews and every candidate emerges with equally great evaluations, your interview questions are ineffective.
Of course, there are the occasional exceptions; for example, if you’re in a final round of interviews and the two finalists perform pretty similarly. But in general, if you’re not seeing significant differences between candidates’ responses to your interview questions, those questions are not working.
And no, candidates aren’t performing equally well because your candidate pool is so terrific. Even the best collection of talent can still be meaningfully differentiated.
The 1992 United States men’s Olympic basketball team (aka the “Dream Team”) was the first American Olympic team to feature active professional players. While every member of that team was excellent, three of the ten players were Michael Jordan, Magic Johnson, and Larry Bird. Everyone there was a star, but only three of them were paradigm-shifting legends. If the greatest sports team ever assembled can be differentiated, so too can your candidate pool.
The problem in hiring is that rarely do companies and hiring managers assess whether their interview questions are adequately differentiating candidates.
For starters, most hiring managers can’t tell the difference between good and bad interview questions. In the report “Six Words That Ruin Behavioral Interview Questions” roughly 80% of hiring managers could not identify the glaring flaws in a series of attitudinal interview questions.
Then there’s the problem of consistency. In the study “The Worker Shortage Is Partially Self-Inflicted” 62% of HR executives believe that their company’s hiring managers are inconsistent in how they interview candidates.
To ensure that every interview question used in your company is actually differentiating between great and poor candidates, you need to conduct a simple exercise. First, you’ll need a consistent set of interview questions. You’ll never be able to refine your hiring practices if hiring managers fly by the seat of their pants in every interview.
Once you have a set of questions, your second step will be to evaluate how well candidates responded to each of those questions. Imagine you’re hiring for an open position; you have 10 viable candidates and five ostensibly decent interview questions. At the end of each candidate’s interview, you’ll rate how well they answered each of your five questions. Something like a seven-point scale, ranging from “Great Fit” to “Poor Fit,” typically works well for this.
When you’ve completed all 10 interviews, you’ll assemble the scores for each of the five interview questions; this will give you ten scores per interview question. Typically companies find that some of the questions have a decent spread of scores, ranging from the very low to the very high. But there are usually at least a few questions where the scores are all clustered in the very high range.
This indicates that these few interview questions are not working to differentiate your candidates. If everyone taking a test gets the same score, what is the point of administering that test? Your task is to eliminate all the questions on which candidates perform equally well; those questions are simply wasting your time. (And, of course, you’ll keep asking the questions that do effectively differentiate candidates.)
It often takes a few iterations of keeping the differentiating questions and eliminating the un-differentiating ones before you’ve got a highly effective set. However, this process of thinking more scientifically about your interview questions will not only select better talent but it will also make your interviews far more efficient.