Don’t Ignore What People Say Just Because It’s What People Say

Qualitative data can be tough for evidence-based professionals because the whole idea of evidence-based management is to get away from opinion and focus on fact. Yet qualitative data, such as anecdotes and opinions, can be seen as forms of evidence that are too valuable to simply ignore.

The key concepts we need to pull from the theory of evidence-based management are:

We want to consider all the available evidence
We want to assess the quality of evidence

For example, if our receptionist tells us that a lot of people are unhappy about a certain decision then we should add that observation to our evidence pool — even if data from the engagement survey indicates otherwise. The idea is to cast a wide net in gathering all the available evidence. The receptionist may be picking up signals the formal survey does not.

The second step is to assess the quality of evidence. If the receptionist is seen as reliable and is in a position to interact with many different people, then their observations should be treated as reasonably good evidence. If the receptionist complains about a lot of things and talks with relatively few people, then their observation will probably be considered low quality evidence.

Get all the evidence in front of you, consider the quality, and make an informed judgement. It is possible to do this in a highly disciplined way using Bayes’ Rule (see Chapter 12 in Evidence-Based Management: How to use evidence to make better organizational decisions by Eric Barends and Denise Rousseau); however, more commonly, analysts just make a judgement call.

It’s worth remembering that objective quantitative data can be of lower quality than qualitative data. An engagement survey that is two-years old may be less accurate than a receptionist’s observations. Objective data that doesn’t have predictive value is also low-quality evidence. For example, data on someone’s elementary school grades, objective though they are, may have very little value as evidence as to who will be good at sales.

Analytics professionals ought to push managers towards making decisions based on evidence. They will be more successful at that if they recognize there can be value in qualitative evidence; it’s just a matter of assessing the quality.