When are research scores comparable (and when are they not)?

"Let's not compare apples and oranges"
This phrase has led many a meeting astray. But it isn't completely misused. Before we compare any research findings, we need to agree on whether those findings are indeed comparable.

But, as we have seen earlier, whether we accept or reject findings depends as much on our prejudices as on our understanding of research. And if we disagree with the findings, we are more likely to believe the comparison itself is invalid.

So, how then, can we be sure that we are making like-to-like comparisons?

The guiding principle here comes from the way laws of economics are written - 'ceteris paribus'. Latin for 'All other things being the same', it sets the condition right at the beginning - that all variables other than the ones being compared are either constant, or their impact has been identified and accounted for.

Example 1: Comparing impact of an ad campaign across 2 bursts
If we see Burst 2 result in better sales than Burst 1, we need to first check if the distribution was already in place when Burst 1 was launched. Alternatively, when Burst 1 performed better, it could be a result of a retailer incentive program that didn't get rolled out along with Burst 2.

Example 2: Comparing an ad campaign across multiple geographies
Competition and Culture vary greatly across geographies. If a campaign for breakfast cereals does well in City 1 versus City 2, it could be a result of different breakfast habits, and different competing breakfast foods and not so much the ad itself.

Example 3: Comparing brand growth over long term using tracking studies
While working on evaluating a brand's long term ad campaign, we saw that while overall the ad had a positive impact, the sales actually dropped in some years when the ad was on air. A little digging revealed that the industry itself went through a slump during that time, and thanks to the ad campaign, this brand performed better than industry. Quite revealing how 'success' and 'failure' got redefined with just one additional factor included in the analysis!

What about your data?
Any interesting comparisons?

Why we still need research...

Our professor of business strategy used to ridicule research. He would say, "BMW doesn't go around carrying questionnaires, asking customers what want." Over the years, the standard example has shifted from automobiles to mobiles (specifically iPhone), but the argument has remained more or less the same.

But there's another point of view too. And it puts forward some very critical reasons why research is helpful.

1. Your customers live in all kinds of un-imaginable infrastructures
If you are reading this on LinkedIn, it is very likely that people in your city don't face problems with electricity and water supply. So, would you be able to understand if your consumers live with 4 hours of daily power-cuts, or get water for only a couple of hours a day? Would you be able to truly appreciate all the tricks and jugaad employed every 'day in the life of' your consumers?

2. You can't un-learn certain ways of life
For the longest part of my life, I would use public transport and auto-rickshaws for all travel, everywhere. But within a few months of buying a bike, I would feel odd about using public transport. Suddenly, travelling by auto-rickshaws felt ridiculously expensive. My outlook to money changed after I started earning. These are changes that you can't undo to imagine life without them. Can you imagine people of your age not owning a smartphone?

3. Everyone has their own unique combination of attitudes, values, and morals
What would you say about a mother who buys branded diapers for the son, and locally made stuff for her daughter? Or about a hospital that charges you 1.5% extra for paying through credit card?

Yes, perhaps the professor was right about questionnaires. We don't need to use them in every situation. But, research is a much larger, and wider field than just questionnaires. And it offers many tools that enlighten the world around us. Why stay in the dark!

Get more from your Brand Tracking Study

Most brands that spend a lot on advertising have some kind of a tracking study in place. And more often than not, use of the findings are restricted to checking whether the marketing team performed better or worse in the last week / month / quarter.

But once in a while a team of good researchers (or, as it happened with me, great researchers!) come along and teach others to do better.

Here's what I learnt:
1. Find what is generic to your category, and what drives the niches
A tracking study has a lot of image parameters and not all of them are equally important in the market. So how do we know which ones are the critical ones?

Take a dump data of fairly long period (say six months for a weekly tracking study) and run a Jaccard analysis on the image parameters and brand preference as the two sets of variables. What you will get is a score for each image parameter. These scores indicate how strongly the image parameters drive brand preference. Sort them in order and look at both the ends of the spectrum.

At one end will be the ones that drive preference the strongest across brands. These are the category generics. Your brand can't score low on these. At the opposite end, are the ones that drive preference for one or few brands, but not for others. These are the image parameters driving the niches. Pretty good data to look at while making brand portfolio decisions.

2. Track how consumer preferences change over time
Even the most boring categories will see changes in consumer preferences over time. To check this, just take two dumps of data over, say, two years and compare scores for image parameters. You can look at average score across brands, or average brands chosen for each image statement, etc. and figure out if certain factors are becoming more important over time.
There's a catch here. In some categories, these changes are a result (and not cause) of communication. For example, if deodorant brands talk about no-gas deos, it may lead to consumers asking for no-gas deos over time. Hence, discretion is advised. :)

3. Measure the lead time of communication on brand scores
The tracking study already tells you how much impact your communication has on the brands scores for you and competition. It can also give you an idea how much time it takes for your communication to show impact on the scores.

Take 2 trend lines - your advertising GRPs and one of your brand scores. Normally, we look at these trend lines as they happen in real life. But creating an overlap can provide a completely different analysis. What if you see the brand score with a one-week or a two-week lag? Maybe it shows a higher correlation between the two trend lines. This can be checked for promotions too. Then, you will theoretically have a better idea on when to begin your Diwali promotion, or back-to-school sale.

If you have already tried these or other such ideas, do share the hits and misses.
Because, well, it helps.

--
With thanks to Ipsita Arora for helping me with this post.