When are research scores comparable (and when are they not)?

"Let's not compare apples and oranges"
This phrase has led many a meeting astray. But it isn't completely misused. Before we compare any research findings, we need to agree on whether those findings are indeed comparable.

But, as we have seen earlier, whether we accept or reject findings depends as much on our prejudices as on our understanding of research. And if we disagree with the findings, we are more likely to believe the comparison itself is invalid.

So, how then, can we be sure that we are making like-to-like comparisons?

The guiding principle here comes from the way laws of economics are written - 'ceteris paribus'. Latin for 'All other things being the same', it sets the condition right at the beginning - that all variables other than the ones being compared are either constant, or their impact has been identified and accounted for.

Example 1: Comparing impact of an ad campaign across 2 bursts
If we see Burst 2 result in better sales than Burst 1, we need to first check if the distribution was already in place when Burst 1 was launched. Alternatively, when Burst 1 performed better, it could be a result of a retailer incentive program that didn't get rolled out along with Burst 2.

Example 2: Comparing an ad campaign across multiple geographies
Competition and Culture vary greatly across geographies. If a campaign for breakfast cereals does well in City 1 versus City 2, it could be a result of different breakfast habits, and different competing breakfast foods and not so much the ad itself.

Example 3: Comparing brand growth over long term using tracking studies
While working on evaluating a brand's long term ad campaign, we saw that while overall the ad had a positive impact, the sales actually dropped in some years when the ad was on air. A little digging revealed that the industry itself went through a slump during that time, and thanks to the ad campaign, this brand performed better than industry. Quite revealing how 'success' and 'failure' got redefined with just one additional factor included in the analysis!

What about your data?
Any interesting comparisons?