Everywhere, people quest after the most important staffing question. For example, the other day a woman called wanting to know if I had a test for hiring control-room operators. It seems their mass-transit agency had been using the XYZ personality test to select train operators, and they wanted to buy a separate personality test for jobs requiring critical decision-making. It seemed like a good question. I tactfully replied, “There is no such thing.” She did not what to hear that. She repeated, “Our test was developed by the famous Dr. XYZ. He even added items to the test to make it more applicable to our position.” I asked, “Is it validated?” She said, “Yes. Dr XYZ gave it to three of our top operators and supervisors.” I asked, “Did he give it to a statistical sampling of high and low producers?” She replied confidently, “No, only the top three.” I replied, “Three people are insufficient to get a good test profile. What about EEOC guidelines?” She got huffy. “Dr. XYZ guarantees the EEOC reviewed and approved the test. Do you have a test or not?” This had all the earmarks of a problematic client. I responded, “No, I’m sorry. I don’t.” We both hung up. She probably thought I was a jerk and continued calling vendors until someone finally gave her the answer she wanted to hear. Not the right one ó not a good one ó but the one she wanted to hear. What was wrong with her responses?
- Personality tests are highly inaccurate measures of job skills. Think about it: Ever known dull people who insist they are smart?
- Giving a test to top people does not indicate whether a test accurately predicts high and low job performance. For that, you need to test both high AND low performers
- Three people are not representative of any large group.
- The EEOC does not “approve” or “review” tests (unless you consider an EEOC challenge or OFCCP audit to be a “review”).
- Vendor guarantees are not evidence of validity.
She could have saved a lot of time if she knew to ask “The Greatest Staffing Question in the Known Universe, Ever.” If she did, the conversation would have been entirely different. What is “The Greatest Staffing Question in the Known Universe, Ever?” Simply this: “Can you show me a professional-quality validation study that proves your test/survey/interview questions are highly predictive of performance in my job?” Yes, I know. You’re disappointed. You probably expected something more ‘meaty”? like, “What is you greatest strength?” “What was your most important accomplishment?” or “What makes you more qualified than the most qualified human being who ever thought about applying for this job?” You might even be thinking, “How could Williams write such nonsense?” (It’s not easy.) Such questions as those above are intuitively attractive, even intelligent. But there is one problem. The answers tell you more about a person’s ability to answer smoothly than they tell you about an applicant’s core job skills. Let me repeat: Interview questions tell you more about the person’s ability to answer questions than they tell you about core job skills. How can I make such a statement? Well, there are educated people in universities who conduct tightly controlled experiments, do their best to treat every subject the same, and then rigorously follow up on results. In short, the researchers start with the assumption that something is true, then try to prove it false. They even report results based on the percent probability it would not occur by chance. They don’t offer personal opinions; they try to provide fact-based, third-party-verified information. These dedicated folks have shown time and again that question/answer sessions do one thing effectively: They screen out people who cannot give good answers. They know this because people who “pass” interviews tend to sink or swim with equal probability. But this should not be earth-shaking news to anyone who has hired employees based on interview data. What Does the Highly Predictive Part of the Question Mean? Hiring is a crapshoot. Even the most promising employee can turn bad under a different manager, a marriage crisis, a new organization, or any number of other factors. The only thing recruiters and hiring managers can control are the “odds” of job success ó and the “odds” of job success depend in large part on having proven skills for the job. Organizations that do a good job of identifying job skills and hiring qualified people have reduced their turnover to about half and doubled their average productivity. We can demonstrate how, as follows. First locate a co-worker with bell-shaped nostrils. The bridge of his nose represents the mid-line representing the average employee’s performance. The nostrils on either side represent a typical hi-lo employee performance distribution. Now, put a drop of super-glue inside each nostril and pinch his nose. Did you notice it is much thinner now? (Quell complaints about breathing difficulty by reminding him science is serious work!) Next firmly grip the newly narrowed nose and quickly jerk it sideways. He’ll probably object; but remember, science is not for the faint of heart ó people naturally resist change, especially when it affects something on their face. If you have carefully followed instructions, the center of your co-worker’s nose will now be much closer to his cheekbone and his nostrils will be tightly pinched-together. This represents a change in position and shape of the bell-curve on the performance scale. That is, there will be an overall increase in average individual performance coupled with fewer performance differences between the best and worst employees. True, we are still playing craps, but the dice are now “loaded” in our favor. Thank the co-worker, give him an ice-pack, call the medics, and wash your hands. (Caution: Do not try this experiment with people who are larger and stronger than you are, your managers, anyone with a VP title, people likely to hold grudges, or someone with a tribal tattoo on his face). Moving the curve requires more than interview questions. It requires demonstrated performance. If the job requires learning, the candidate should clearly demonstrate he or she can learn; if the job requires coaching, the candidate should clearly demonstrate he or she can coach; if the job requires questioning, the candidate should clearly demonstrate he or she can question; if the job requires presentation, the candidate should clearly demonstrate he or she can present ó and so forth. But What About the Human Decision Factor? Only people who bought a lifetime subscription to Dionne Warwick’s Psychic Friends Network would argue in favor of the exclusive use of half-baked job requirements and interview data. These create human-decision “fog.” Hiring managers and professionals need less fog and more trustworthy data. They need to know how to rule-out candidates who cannot perform critical competencies right before their eyes ó not just tell stories about them, not just second-guess interview questions. Demonstrate the skill right then, right there. Then, and only then, can a hiring manager or recruiter move the bell-curve. What about the candidate who fails the pre-screen but could have performed if hired? If the candidate cannot demonstrate necessary skills during pre-screen, then the odds he or she can perform on-the-job are less than the chances of receiving a million-dollar bank-transfer from that exiled Nigerian Ambassador who keeps sending you emails. Moving the bell-curve takes guts:
- Job specifications must be directly job related.
- Job skills must be clearly demonstrated.
- Test conditions must be tightly controlled.
- Scoring should be tightly controlled.
- Evaluators should be highly trained.
- Several methods should be used to measure the “whole” job and “whole person.”
- Each method should be known to be highly predictive of job performance.
- The number of good-natured co-workers with big noses is limited.
What about the woman and her test for mass-transit critical decision-makers? Well, I’ve done a few comparisons between self-reported personality and cognitive ability scores. In the majority of cases, personality seldom explains more than 1% to 2% of cognitive ability scores (i.e. percentage of variance). Bottom line: If your job needs “smarts,” personality tests are the wrong choice! Why worry? The lady wanted to use a personality test to hire critical decision makers. Test scores from a personality test will probably have little or no decision-making predictability. Would you want to ride a mass-transit system where the odds of employee high performance are only 50/50? By the way, Dionne’s Psychic Friends went out of business in 1999. Do you think they knew beforehand?