Posted on

On inference and statistical significance

Friday rabbit hole caveats be damned healed, a reference to p-values in my readings this week struck a chord for ready digression.

I interviewed a statistics researcher [podcast versions] a few years ago who served as an co-editor on special journal issue dedicated to the (mis)use of statistical significance in research, particularly psychology research and specifically the phenomenon know as p-hacking:

a little more explanation of the null hypothesis and how it relates to other flaws in the typical way psychology researchers have handled statistical analysis over the past several decades. Roughly, our research often comes down to finding differences between groups of participants on some variable of interest (Are boys better at math than girls?) or correlations among two or more variables (Does suicide risk rise as poverty increases?). The null hypothesis is the assumption we then make that there are no such differences or correlations in the populations from which our data were drawn, and we compute the probability that we would have found data with differences or correlations as large as we actually did if the null hypothesis were true. If that probability (called a “p-value”) is less than some specified value, typically .05, then we reject the null hypothesis and conclude that the differences or correlations we found are probably real. This is known as a statistically “significant” result. If the p-value is greater than .05, then we fail to reject the null hypothesis, for lack of convincing evidence otherwise.
Research that fails to reject the null hypothesis rarely gets published in scientific journals. Likewise, research that merely replicates rejections that were found in earlier research rarely gets published. As a consequence, there is a high professional premium on testing questions that have not been previously published, and on rejecting the null hypothesis.

The widespread adoption of these customs has unintentionally encouraged a whole series of inadvisable research practices, which have led to trouble. For instance, because most studies that do not find significant effects go unpublished, we don’t know whether the findings of the published research are real or just the mistakes that our chosen p-value allows to slip through (the “file-drawer problem”). This problem might be attenuated if we published efforts to replicate previous findings, but that kind of research is rarely accepted by journals.

Note the generosity embedded in the term ‘inadvisable.’ Hmm.

 

Posted on

Steady in One Desire

The Roman stoic Seneca (5 BC to 65 AD) was a philosopher and statesman whose writings have made to our own day in several forms, including a Penguin Great Ideas series. The slender volume On The Shortness of Life, Life is Long if You Know How to Use It is three sections full of gems. Is the life we get in fact not short – we just make it seem so? This from the first section, which is along letter to his friend Paulinus, On the Shortness of Life:

But life is very short and anxious for those who forget the past, neglect the present, and fear the future. When they come to the end of it, the poor wretches realize too late that for all time they have been preoccupied in doing nothing. And the fact that they sometimes invoke death is no proof that their lives seem long. Their own folly afflicts them with restless emotions which hurl themselves upon the very things they fear: they often long for death because they fear it. Nor is this a proof that they are living for a long time that the days seems long to them, or that they complain that the hours pass slowly until the time fixed for dinner arrives. For as soon as their preoccupations fail them, they are restless with nothing to do, not knowing how to dispose of their leisure or make the time pass. And so they are anxious for something else to do, and all the intervening time is wearisome: really it is just as when a gladiator show has been announced, or they are looking forward to the appointed time of some other exhibition or amusement – they want to leap over the days in between. Any deferment of the longed-for event is tedious to them. Yet the time of the actual enjoyment is short and swift, and made much shorter through their own fault. For they dash from one pleasure to another and cannot stay steady in one desire. Their days are not long but odious…

Even their pleasures are uneasy and made anxious by various fears, and at the very height of their rejoicing the worrying thought steals over them: ‘How long will this last?’ This feeling has caused kings to bewail their power, and they were not much delighted by the greatness of their fortune as terrified by the thought of its inevitable end. When that most arrogant king of Persia [Xerxes, ed.] was deploying his army over vast plains, and could not number it but had to measure it, he wept because in a hundred years out of that huge army not a soul would be alive. But he who was weeping was the very man who would bring their fate upon them, and would destroy some on the sea, some on land, some in battle, some in flight, and in a very short time would wipe out all of those for whose hundredth year he was afraid.

And what of the fact that even their joys are uneasy? The reason is that they are not based on firm causes, but they are agitated as groundlessly as they arise. But what kind of times can those be, do you think, which they themselves admit are wretched, since even the joys by which they are exalted and raised above humanity are pretty corrupt? All the greatest blessings create anxiety, and Fortune is never less to be trusted than when it is fairest. To preserve prosperity we need other prosperity, and to support the prayers which have turned out well we have to make other prayers. Whatever comes our way by chance is unsteady, and the higher it rises the more liable it is to fall. Furthermore, what is doomed to fall delights no one.