The Emperor’s New Metrics

Ptolmey (150 A.D.) used to teach his students that the sun went around the earth. After all, what could be more obvious? This seems to the prevailing approach when it comes to metrics for staffing. The most popular metrics relate to cost, time, and quality. We take it for granted that producing these numbers will improve the staffing function in any organization, despite any evidence to suggest that there’s value here. Quality is among the most popular ones in staffing circles ó metrics around quality get a lot of press in the HR journals. In the quest to be recognized as a “business partner,” staffing and HR have seized upon quality as a means to demonstrate that they “get it.” What better way to establish business credentials than to show a willingness to deliver “quality”? This is the concept that was a catalyst for transforming whole industries, and even a country ó the Japanese post war economic boom and the success of GE resulted from a commitment to quality. The best approaches to manufacturing and management all embody quality, be they Six Sigma, execution, or supply-chain management. Start throwing around names like Juran, Deming, and Crosby, interspersed with terms like “key performance indicators” and “analytics,” and you should be able to impress just about anyone. This would all be good if it was done right. Unfortunately, in our zeal to get a “seat at the table” staffing and HR have only demonstrated an inability to grasp a complex issue. We have metrics striving to establish quality of hire, applicant tracking systems that promise to improve the quality of staffing, and of course legions of consultants pitching their services to bring about quality. The emphasis on quality should be a positive for staffing, since the profession, by virtue of being the unwanted stepchild of HR, has long suffered from a lack of respect. But most discussions around quality in staffing miss the point entirely, either because of a wrong use of the concept or a fundamental lack of understanding about what constitutes quality. Since the goal is to improve quality (one would hope) then if the basic premise is wrong, it’s not likely to achieve this end. Defining Quality As with most things, quality is a far more complex concept than we would like it to be. But the desire to bring quality to staffing has resulted in a simplification that has distorted it beyond recognition. It’s easier to talk about it than to do something with it ó a metric is not an end unto itself ó but it’s worthless if it cannot be used to make improvements in the outcome it measures. The problem starts with the idea that quality can be managed by a metric such as “quality of hire.” No single metric can serve as a measure of quality. For that matter, no group of metrics can be aggregated to arrive at a measure of quality either. Quality is defined as being able to meet requirements; its definition was established by quality gurus Philip Crosby, Joseph Juran, and Edward Deming. In manufacturing, that means being able to build a product that matches the specifications and is within an acceptable range of tolerances. A component that has been precisely machined and does not fail if subjected to the conditions it was designed for. For a service, quality means having few errors in delivery ó within the Six Sigma threshold, for example. The basic idea behind all approaches to quality is to control a process at every step in order to deliver the product or service. It is not to try and prove at the end that the process produced a result or met some goals. That would be like saying that a car assembly line produced a car, so it must be a high quality vehicle. A process must meet quality standards to produce a quality product. Process consistency and repeatability are essential. In staffing, quality of hire has been interpreted to mean that hired candidates met the job requirements as specified in a job description. Longer term measures include metrics like hiring manager satisfaction or correlating job performance with evaluations made at the time of hiring. Many assume that if the hiring manager is satisfied, then quality has been achieved. These are not measures of quality. First, for all but those jobs with the simplest requirements, there is no meaningful way to determine the level of match between the requirements and the candidate’s skills. Assessments are often used as a surrogate measure of quality, but these are predictors of job performance, not measures of quality. In any event, the best any assessment can do is explain about 40% of the variance in job performance. The use of subjective interviews in the hiring process further muddles the picture, since there is no way to predict the impact of any particular interview on how a candidate will be evaluated for a particular job. Measures like satisfaction are too imprecise and confounded by too many unknown variables to be of any value as quality indicators, beyond being a general indicator. The same holds true for any correlation with job performance. So what to use? To get at the answer, start with the point of measuring quality in the first place. Quality measures as defined by Juran and others exist so that quality can be improved or at the very least managed. Since staffing is a service, understanding quality requires first knowing what constitutes an error. Whenever a good candidate is rejected prematurely in the hiring process, an error has occurred. Similarly, when a weak candidate moves forward in the hiring process, an error has been made. Knowing this is critical if staffing is to deliver quality results, since every step of the process impacts the outcome. It’s tempting to reduce staffing quality to just finding candidates with skills that seem to match the job description, but that’s a simplistic approach that demonstrates a lack of understanding of the fundamentals of a complex issue. Worse, it is hard to measure, and therefore to manage. So the staffing function fails to improve in a meaningful way, maintaining its traditional designation as a non-business function. A Multifaceted Approach The goal of delivering results that match requirements and are free of errors requires managing everything that can contribute to errors. It may never be possible to do so completely, but it’s a goal. To borrow a phrase from an automaker, it’s the relentless pursuit of perfection. That is what quality is about ó striving to do better, not trumpeting a metric. The staffing process is just one component of a larger quality staffing program. A quality process will reliably select quality candidates if such are fed into the machine. This requires quality sourcing: a quality sourcing program will consistently feed quality candidates to the machine. A quality staffing program requires both reliable sourcing and consistent input to create reliable output. Quality analyses of sources are not generated by those positioned to measure them ó that is, employers. Instead, employers want to rely on vendors (ad agencies, ATS providers, etc.), who, for a variety of different reasons, are not positioned to provide them. First, let’s examine ad agencies. These vendors place ads, and responses are routed directly to the client. Ad agencies are not in the loop on the receiving end, and are relegated to a weak partnership providing creative and media buying services. On the other hand, the ATS vendor is in a better position to provide sourcing analytics. But ATS vendors fail to do so for a different reason. An ATS is a piece of processing software, which is different from sourcing software. Processing software deals with materials that are brought to it, working them through a system. Sourcing software (posting tools, web spiders, parsing tools) generates responses that work outside the system and brings material back in some manageable form. An ATS vendor typically partners with sourcing providers, or skips them altogether. While response data are routed back to the ATS, the outgoing data streams (postings, newspaper ads, staffing company orders) are not captured by the ATS. Ironically, ad agencies tend to have outgoing data, and ATS have incoming data. The combination makes for some powerful information, but the only ones with access to both sets of data are the employers. And they don’t seem interested unless it is handed to them. Further, employers seem to feel that if a tool is not bundled into their ATS, it must not be important. ATS vendors have done a fine job marketing their wares as “end to end” solutions. But as a result, sourcing quality isn’t even on the radar. How can anyone take discussions of quality seriously if it is ignored in the initial step? This shortsightedness belies a lack of management acumen that haunts the HR field. The staffing function is best suited to absorbing these lessons and becoming an equal partner in the business enterprise, but given the stature of staffing in most HR departments, that’s about as likely as the New York Times saying anything good about George Bush. ATS vendors have done little to remedy this situation: there’s plenty of talk about analytics and metrics, but little of value being done. This is particularly unfortunate given how many ATS vendors like to boast about their supply-chain management credentials (some more than others). But the most popular metrics for staffing, available through an ATS, are historical measures like cost per hire and time to fill ó almost completely useless when it comes to managing results. On the other hand, it could be that the vendors know their audience. HR as a profession isn’t exactly known for attracting people with a quantitative bent. Trying to explain the finer points of advanced statistics and concepts pioneered by Deming et. al. in these circumstances is akin to discussing evolution with a creationist, for all the difference it’ll make. Still, it’s a poor excuse for perpetuating ignorance. Beyond Sourcing Sourcing is followed by multiple levels of screening, evaluation, and ultimately a hiring decision. Each of these steps requires controlling for errors if the entire staffing program is to deliver a quality result. This requires having clarity on what each step needs to accomplish ? not waiting till the end to find out whether the selected candidate does in fact come close to the job requirements. Obviously this is easier said than done, but then no one said it would be easy. As someone said ó if it was easy, everyone would do it. But, don’t do it because it’s easy. Do it because it’s important. HR is a profession that has long resisted accountability. For many HR professionals, it was an article of faith that what they did produced value, never mind the lack of proof. It was only pressure from executives and others to bring administrative costs down that forced HR to start using metrics and become more of a value provider than just overhead. Staffing has unnecessarily suffered as a consequence of being in HR’s shadow and not developing as a data-driven discipline. Focusing on quality is one way to change the situation, but it’ll take a lot more than producing a periodic report on cost per hire and time to fill. A consistent, repeatable, quality hiring process can be achieved, beginning with sourcing and by extending quality measures through each step in the process. It has already been done in many fields, and with the prevalence of automated staffing tools, should be happening here. By doing so, staffing will show measurable contribution to the enterprise, managing a greater value proposition. A likely side effect is acceptance as part of the business, instead of being lumped in with the overhead. It took 1,500 years for Ptolmey’s views to be dismissed. Hopefully, it won’t take HR and staffing that long to change.