Walking the Talk

Every so often, we get an opportunity to conduct a little practical research to help make a point. This is a story about the value of different selection tools. A few months ago, a gentleman who was having a problem with his selection system contacted me. It seems that his company was using a well-known behavioral interview process to select new employees. As we talked, I learned that his operation was heavily oriented toward teamwork, and close interaction was critical to deliver high-quality product. An outside organization had done a job analysis, defined competencies, developed interview guides, and trained a large number of people to use behavioral interview technology. However, while his organization only hired people who passed the interview with good teamwork scores, a large number of employees still had decidedly un-teamwork “attitudes.” The interview technology was sound; the interviewers were trained; but the interview did not reliably screen out people unwilling or unable to work effectively in teams. My contact and I reasoned that we needed additional tools to measure teamwork skills, but we were not sure whether the problem was a lack of motivation or a lack of skill. So we decided to set up an experiment. We asked frontline managers to individually rank all employees based on their teamwork behavior (combining scores from many managers helped us reduce individual rating bias). This gave us a starting list of over 40 people. From that list, we asked for volunteers to participate in two short exercises. One exercise required completion of a 100-question measure of attitudes, interests, and motivations (AIM test), and the second exercise required participation in a controlled role-play requiring teamwork skills. Nineteen people volunteered. Two people from the organization were trained to administer and score the AIM test (attitudes, interests and motivations) and to administer and score the role-play. We were ready to go! All nineteen people completed the test and role-play. Scores from three of these people were excluded because their truthfulness test score was out of the acceptable range (this is why good selection tests have truthfulness scales). This left sixteen people to work with ? not enough to conduct a statistically valid study, but enough to rationally examine the data for large differences in scores. The remaining sixteen people were assigned to either a high-teamwork or low-teamwork group based on managers’ teamwork rankings. Scores from the AIM test and role-play were averaged based on group membership. Here are the results: Figure 1.

I have to warn you that statistical tools don’t work very well when using small numbers. In fact, all these averages could have occurred by chance, so be careful about generalizing the results to other situations. But let’s just set aside statistics for a while and examine score differences using logic. Teamwork role-play scores are a good 60% higher for the High group than the Low group. This means that members of the High group actually demonstrate better interpersonal skills than the Low group. This makes sense. High members are able to resolve interpersonal differences more effectively than Low members. This tells us that incorporating a simulation into the hiring process should help screen out applicants with low teamwork skills. We next looked at the score differences from the AIM test to see if there were differences in attitudes, interests and motivations. Five of the factor differences (Innovation, Administration, Teamwork, Impulsiveness and Perfectionism) were too small to consider (i.e., they had less than 10 percentile points difference). This was surprising, because one would expect Teamwork scores to show a big difference; however, members from both groups indicated they were “team oriented.” Probably not a good idea to trust Teamwork scores when predicting “Teamwork.” Bummer! But look! Interest in problem-solving activities, being more resistant to change, being more socially expressive, having a better work attitude, and being less self-centered are all associated with being a better team member. So it would seem that demonstrated teamwork is associated with maintaining high quality standards (e.g., Resistance to Change, Work Attitude) and being willing to do something about it (e.g., less Self-Centered, Problem Solving, Expressive). Interesting! The AIM test scores can probably help reduce hiring mistakes, but they need to be validated with performance to identify the right factors. So what does all this mean?

• As we said before, interviews ? even the best of them ? miss a significant amount of data. In this example, they totally missed Teamwork skills.
• Validated tests of attitudes, interests, and motivations should be able to identify good team members, but they are not always based on obvious scales. In this case, the Teamwork scale did not predict either high or low performers. Why? For one thing, we don’t know enough about manager ratings. That is, did managers base their ratings on the identical interpersonal factors measured by the two selection tools, or did something else influence their ratings? We need to find out. Most importantly, it points out the need for ANY pencil and paper test to be validated against performance.
• Simulations are better measures of Teamwork skills than interviews.

Like I said: we do research in the lab, but seldom have the chance to test our tools in real world conditions. This was a real treat. My only regret was that we had too few people to be able to use statistical tools that would assure us that these results did not occur purely by chance. Nevertheless, I think using a little “after the fact” rationale to examine score differences identified some important trends. This study was conducted in one small division of a large conglomerate. Company leaders were enlightened and intrigued by the results. Organizations move slowly, but a company-wide audit and wholesale redesign of hiring practices should be underway shortly. <*SPONSORMESSAGE*>