Significance Testing is not Magic

A ten-minute survey produces a lot of data. And if we’re trying to study the differences between different demographics, then you end up with a lot of comparisons. (A 20-question survey comparing men/women, three age ranges, and three income ranges yields 140 comparisons!).

In a typical data table, these comparisons will be “tested” for significance – usually denoted by a little superscript next to the percentage in a cell.

But significance testing like this is misguided. Here’s why…

Every significance test implies a hypothesis was tested

Generating 140 t-statistics implies that we had 140 hypotheses that we wanted the data to help accept/reject. That’s simply never the case in corporate market research. Hypothesis development is a careful and painful process. It requires a theoretical understanding of the subject matter followed by a painstakingly detailed experimental design and execution.

Corporate market research is usually much more exploratory. Deadlines and budgets mean we can’t always design the study exactly how we want. The result is surveys that are typically a little longer than we wanted in the hope that something in there will have the answer we’re looking for.

5% of 140 is seven

Conducting mass hypotheses tests puts the researcher at risk of interpreting random differences as true differences. (https://xkcd.com/882/)

If you are using a 95% confidence interval, then 5% of the time you’ll get a false-positive (type I error). That’s really dangerous when you have 140 comparisons because it means seven differences may not be real.

It doesn’t make the results any more scientific

Testing 140 hypotheses at once is like throwing a bowl of spaghetti against the wall and seeing what sticks. Putting on a lab coat doesn’t make what you did an “experiment” any more than computing 140 t-statistics does.

Survey questions are not independent

Experiments are supposed to be independent. The results of hypothesis 1 should not have any impact on the results of hypothesis 2. For example, if the survey results show men are more likely than women to do X. Then later in the same results you find men are less likely to do Y, by computing two t-statistics, you’re implying that these results are independent of each other.

Of course, there’s still value in the survey results. But a good researcher needs to recognize that a t-value of 1.96 or more doesn’t represent some magical boundary between reportable vs. non-reportable data.

Treat your data as a set, not an enormous combination of combinations.

Advertisement

Post-Election Doubts About Corporate Communications

Price tends to be the greatest equalizer.

There are somethings that I can overlook if I’m paying a good price, and I think many can sympathize with this view. We’ve all been in some retail store and thought, “wow! How is that pillow so inexpensive?”

If you think about it too much, you probably can talk yourself out of buying the pillow for one reason or another, but the price is just too damn low to start questioning yourself. So, you don’t worry about the Taiwanese labor conditions or whether the feathers are real or whatever…

That’s why, in industries in which price isn’t as big of a consideration, the softer attributes of the firm play a larger role in the purchase consideration. And we often see this in the form of companies investing in corporate communications.

To me, there’s two conditions that need to be met in order for the industry to invest in corp comm:

• The industry needs to be competitive so that PRICES ARE COMPETITIVE.
  • I don’t mean “perfect competition,” where there are a huge number of firms, is necessary. But the industry should behave as if it is competitive. So oligoplies, like energy companies, where we see pretty competitively low prices, apply.
• Firms need to sell a personality. (Think clothing)

Taken together, we can imagine a firm in a market with competitive prices that has an opportunity to communicate to its consumers a sense of identity. The luxury car market is a pretty good example. It seems that every luxury car commercial has something to say about standing out from the crowd… when everyone else has this grey car, you can stand out if you by this silver car!

All in all it would seem then that corporate communications is important. You want your consumers to identify with the good things that your company does so that your consumers can identify with your company itself. “That company gets me!” is what you want your customers to think.

So it’s not surprising that according to corporate communication specialists, it’s important to tell your consumers the good things that your company does. Not only that, it’s important how you say it – you don’t want to brag, but you want to make sure people know. It’s a tricky balance, but one that seems to be successfully towed regularly.

All this being said, I’m actually pretty skeptical about the value of corporate communications. And frankly, the fact that our electorate is willing to vote into office a person who would never, in a million years, be hired onto any company’s corporate communication team, does not help my skepticism.

So, yes, I am officially doubtful of the value of corporate communications. If someone can be so successful in an industry (politics) that is almost nothing more than communications, then why on earth would people care about what a CEO has to say about anything other that what their company sells.

Apparently it’s ok brag about sexual assault, brag about infidelity, be openly racist, misogynistic, immature, etc. because people will still figure out a way to justify purchasing your product anyway.