Posted on

On inference and statistical significance

Friday rabbit hole caveats be damned healed, a reference to p-values in my readings this week struck a chord for ready digression.

I interviewed a statistics researcher [podcast versions] a few years ago who served as an co-editor on special journal issue dedicated to the (mis)use of statistical significance in research, particularly psychology research and specifically the phenomenon know as p-hacking:

a little more explanation of the null hypothesis and how it relates to other flaws in the typical way psychology researchers have handled statistical analysis over the past several decades. Roughly, our research often comes down to finding differences between groups of participants on some variable of interest (Are boys better at math than girls?) or correlations among two or more variables (Does suicide risk rise as poverty increases?). The null hypothesis is the assumption we then make that there are no such differences or correlations in the populations from which our data were drawn, and we compute the probability that we would have found data with differences or correlations as large as we actually did if the null hypothesis were true. If that probability (called a “p-value”) is less than some specified value, typically .05, then we reject the null hypothesis and conclude that the differences or correlations we found are probably real. This is known as a statistically “significant” result. If the p-value is greater than .05, then we fail to reject the null hypothesis, for lack of convincing evidence otherwise.
Research that fails to reject the null hypothesis rarely gets published in scientific journals. Likewise, research that merely replicates rejections that were found in earlier research rarely gets published. As a consequence, there is a high professional premium on testing questions that have not been previously published, and on rejecting the null hypothesis.

The widespread adoption of these customs has unintentionally encouraged a whole series of inadvisable research practices, which have led to trouble. For instance, because most studies that do not find significant effects go unpublished, we don’t know whether the findings of the published research are real or just the mistakes that our chosen p-value allows to slip through (the “file-drawer problem”). This problem might be attenuated if we published efforts to replicate previous findings, but that kind of research is rarely accepted by journals.

Note the generosity embedded in the term ‘inadvisable.’ Hmm.

 

Posted on

Just in Time

is an inventory strategy implemented to improve the return on investment by reducing in-process inventory and associated costs. In order to achieve JIT [status] the process must have signals of what is going on elsewhere in the process.

This means that the process is often driven by a series of signals […] that tell production processes when to make the next part. [These] are usually ‘tickets’ but can be simple visual signals, such as the presence or absence of a part on a shelf. When implemented correctly, JIT can lead to dramatic improvements in a manufacturing organization’s return on investment, quality, and efficiency. Some have suggested that “Just on Time” would be a more appropriate name since it emphasizes that production should create items that arrive when needed and neither earlier nor later.

Semantics of ‘on’ vs. ‘in’ aside, what do these kinds of signals tell us? I alluded previously to complexity theory and the principle of indirect effects as constituent elements of systems ecology; one of the questions we need to reckon with in the ‘what’s next’ phase of our mourning is, how do we begin to untangle some of the many ways in which the American way of life and self-worth is connected to scams and schemes? A dangerous loss of legitimacy is waiting right around the corner for yet another cheek to be turned when a perp-walk might be warranted.

For a proper idea on the state of the reckoning, we can probably zoom past the savvy of green ads for a little while. Just sample/monitor the holiday editorials in your local paper and see what you come up with. Are they typical paeans to the year that was? Or are there little hints and allegations that actual people have had enough? Are people starting to ask questions with uncomfortable answers?