blue-growth-chartThe New Yorker published a lengthy editorial on how natural human behavior affects the scientific community when it comes to studies, in that selective reporting and significance chasing leads to ‘publishing bias’.

“the act of measurement is going to be vulnerable to all sorts of perception biases. That’s just the way human beings work.”

“researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance – the ninety-five-per-cent boundary invented by Ronald Fisher. The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,”

“…the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for.”

“The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe…”

Although the context is regarding the scientific community, I was thinking how there are direct parallels to the business and B.I community as well. perhaps we’re too focused on the pursuit of truth, when the ‘reality’ is that there no truth, only perception & interpretation.

So instead of using data to prove things, we need to look at data to be more of a guide. For example monitoring trends, inflection points, velocity of change over specific numbers. Not that specific numbers are unimportant (e.g. shareholders will continued to demand exact profitability statements), but in terms of managing a business not to exhaust enormous efforts on proving perceptions/suspicions/hypotheses but to define data driven decision supporting guides.

2 Responses to “B.I – Chasing significance and selective reporting”

  1. larry c. lyons says:

    Actually this is an old phenomenon. Even Fisher back around the 1920′s criticized the .05 significance label. That said I suspect that the so called effect size decline is something rather different. To give an example, in grad school I conducted a detailed meta-analysis of the relationship between hypnotic susceptibility and the Tellegen Absorption Scale (http://www.lyonsmorris.com/lyons/absStudy/index.htm). In the initial few years of research the correlation between the two was fairly strong, but it declined over time to a more modest correlation. In looking at this, I suspect that this is more akin to an initial enthusiasm (hey cool look at this relationship – other researchers are impressed one way or another and duplicate it). As time goes by the enthusiasm wanes and somewhat lower correlations are noted. However it does bottom out, in this case with r’s around .25-.4 depending on the hypnotizability measure. This is probably a more accurate estimate of the relationship within the population. Another case of where a healthy bit of skepticism may tease out the actual relationship.

  2. larry c. lyons says:

    Forgot to mention the File Drawer problem is an old issue with meta-analysis. Rosenthal (Rosenthal, R. (1991). Meta-analytic procedures for social research (rev ed). Beverly Hills CA: Sage.) gives a fairly effective way that estimates the number of additional studies, with mean null result necessary to reduce the combined significance to a desired a level (usually 0.05). If its a fairly low number then you may have a file drawer problem. If that number is very high, then there is no problem.

    In my own research I’ve had the opportunity to substantially tap a literature base – getting published, unpublished, studies presented at conferences, dissertations, theses, file drawer studies, etc., all that looked at a specific area of research. In comparing the different publication categories, I found no statistically significant differences among published, unpublished and other types of publication status.

Leave a Reply

Your email address will not be published. Required fields are marked *

* Copy This Password *

* Type Or Paste Password Here *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Powered by sweet Captcha