Do You Have a Test?

I don’t know where people get the idea that pencil and paper tests are the universal solution to all hiring problems. Tests are good for some things, but not others. But I’m getting ahead of myself. Let’s start by dividing tests into two groups: 1) those designed for hiring, and 2) everything else. Now, forget about the tests in the “everything else” pile. We only care about hiring tests. Hiring tests are based on a specific theory associated with job performance. No theory. No test. They use this theory to predict future job success. No prediction. No test. Developing a good hiring test requires investigating many jobs looking for common threads underlying job performance, using fancy statistics, running many controlled studies, and verifying that test scores are strongly correlated with performance ratings. This is called “validation” and it basically means test scores are accurate predictors of job performance. Validation can refer to the “content” of the job (e.g., typing tests = typing skills, programming tests = programming skills, problem solving tests = problem solving skills); or, it can refer to level of job performance (e.g., typing score = processing speed, programming score = lines of code, problem solving scores = profitability). There is a third form of validity called “construct” validity, but we won’t cover it here. No vendor claim is proof that his or her test is “valid” for your application. Each organization is responsible for its own validation work. If the test you use now was not designed explicitly for hiring, or if it is not validated, you need to find another test (don’t mean to be blunt, but it saves time). Hiring tests come in two main varieties: 1) the skills variety, 2) the self-descriptive variety. If you want to achieve accurate results, you need to know which is which and use them appropriately. Skills Variety Skills tests measure abilities. Yes, abilities. A spelling and grammar test measures spelling and grammatical ability, a Visual Basic test measures Visual Basic ability and so forth. The important thing to remember is that while ability tests are very “narrow,” they go very “deep.” Skill tests excel at measuring things like cognitive, technical, sensory or physical abilities. Licensure tests of law or medicine, certification tests of engineering, compensation and benefits exams or tests of programming languages are good examples of skills tests. These are measures of subject command. Other tests like hand-eye coordination, spatial ability, or mechanical ability also qualify as skills tests because they measure some kind of sensory or physical ability important to the job. I once recommended a test that required putting little pegs into little holes as a pre-requirement for people who assembled car radio circuit boards. Why? Because both tasks required fine motor control and fast finger movement – precisely the kind of skill needed to place little electronic components into teensy circuit board-holes. There are many other types of skills tests, but the main point is that skills tests are very hard to fake because they require demonstrating a specific ability. If you do well on a skills test, a hiring organization can be pretty sure you will do better than someone who fails the same test. As a matter of fact, the only time skill tests become “shaky” is when they are either too advanced or too basic when compared with job requirements or when the applicant practiced the test ahead of time. But mostly, it’s like testing basketball players for being tall. You might not always be right, but it sure helps play the game. Self-Descriptive Tests Self-descriptive tests are entirely different from skills tests. In many cases, they have almost nothing to do with actual skill levels either. A self-descriptive test is often termed a “motivation” test, “communication” test, “personality” test, “leadership” test, etc. Whatever the name, self-descriptive tests tend to have one thing in common – the questions ask people to “describe” themselves. Do you see any thing wrong with this idea? No? Let’s get personal. Have you ever made yourself look good on a test like this? Have you ever tried to second-guess what the organization was looking for? Have you ever changed your mind, and your answers, from one month to the next? Get the point? Self-reports are like that: they change. Bob Hogan, one of my dissertation advisors, and one of the leading researchers in the field of personality, calls this “socio-analytic theory”. Socio-analytic theory is Bob’s term for giving answers that correspond to how you want to present yourself to others. Bob says that before you put total faith in self-descriptive tests, you should think about whether the data represents absolute truth or only how the test-taker wants to describe himself to the prospective employer. Self-descriptions have very low statistical relationships with abilities. In my studies, I found that scores on a big-five personality test have about a two percent relationship with mental ability and an eight percent relationship with interpersonal skills. Pretty dismal correlations. Putting a self-descriptive test on a website might (if validated) be a useful tool for measuring job fit, but it should never be used to make an accurate prediction of job skill unless you know of some previously undisclosed research that shows scores on a self-descriptive test have a strong relationship with criterion scores on the job. This is why reputable self-report test vendors actively discourage the use of their test for selection or will insist on a scientifically conducted validation study prior to use. It is just good business practice. It is also ethical. When Self-Descriptive Tests Go Bad On first blush, it seems like teamwork would be a good thing. Let’s examine how one less than competent HR manager royally messed up his company by misunderstanding test use. A large truck assembly company wanted to put in a new plant in Ontario, Canada. They had the chance to hire about three hundred people from scratch (e.g., a “green field” operation). No dead wood. No legacy workers. Just an opportunity to hire good producers. The HR manager put in some tests. “Teamwork is good,” he must have thought, because he interviewed for teamwork, created simulations that measured teamwork and made hiring decisions based on teamwork. What did he get? Three hundred employees who would not hold a meeting unless everyone could attend, could not make a business decision unless everyone agreed and wanted to regularly socialize after work (to extreme spousal objections). In the meanwhile, the company expanded their main assembly building to fix quality problems caused by people who would not confront co-workers producing poor quality work. This example illustrates the power of misused hiring tools. The HR manager wanted teamwork. He got teamwork. He just forgot about all the other things that were important, like productivity and quality. I don’t know how long it took for his company to get the mix right, but it certainly didn’t happen overnight. The HR manager is no longer employed at the assembly plant. In another example, a very large organization put in hiring systems for plant start-ups of some of the world’s foremost auto manufacturers. They selected people who were very smart, highly motivated, good with people, and hard workers. All very good when starting up a plant, right? Now, fast-forward two years when all the start-up stuff is over and production has become routine. Can you tell me what kind of problems a plant with all leaders and no followers will have? Get the picture? Organizational needs change depending on growth stage. It wasn’t their fault; the client believed the consulting firm knew what it was doing. Too bad. Conclusion Good hiring practices involve more than just one kind of test and more than one kind of scores. Skills test must be carefully selected for level of difficulty and supplemented with different kinds of self-descriptive tests that measure a variety of attitudes, interests and motivations. There is some good news and bad news here. The good news is that if all you use are interviews or application profiles you can be assured that you will get a nice mix of people. The bad news is that half will be low performers. Of course, you can always use skills tests and self-descriptive tests conscientiously and follow best practices. The good news here is that you will get exactly what you measure. The bad news? Well, you just have to work a little harder to find them.