Error Statistics Philosophy
Summer Seminar in Phil Stat 2019
The eighth meeting of our Phil Stat Forum*:
The Statistics Wars
and Their Casualties
22 April 2021
TIME: 15:00-16:45 (London); 10:00-11:45 (New York, EST)
For information about the Phil Stat Wars forum and how to join, click on this link.
“How an information metric could bring truce to the statistics wars“
Daniele Fanelli
Abstract: Both sides of debates on P-values, reproducibility, and other meta-scientific issues are entrenched in traditional methodological assumptions. For example, they often implicitly endorse rigid dichotomies (e.g. published findings are either “true” or “false”, replications either “succeed” or “fail”, research practices are either “good” or “bad”), or make simplifying and monistic assumptions about the nature of research (e.g. publication bias is generally a problem, all results should replicate, data should always be shared).
Thinking about knowledge in terms of information may clear a common ground on which all sides can meet, leaving behind partisan methodological assumptions. In particular, I will argue that a metric of knowledge that I call “K” helps examine research problems in a more genuinely “meta-“ scientific way, giving rise to a methodology that is distinct, more general, and yet compatible with multiple statistical philosophies and methodological traditions.
This talk will present statistical, philosophical and scientific arguments in favour of K, and will give a few examples of its practical applications.
Daniele Fanelli is a London School of Economics Fellow in Quantitative Methodology, Department of Methodology, London School of Economics and Political Science. He graduated in Natural Sciences, earned a PhD in Behavioural Ecology and trained as a science communicator, before devoting his postdoctoral career to studying the nature of science itself – a field increasingly known as meta-science or meta-research. He has been primarily interested in assessing and explaining the prevalence, causes and remedies to problems that may affect research and publication practices, across the natural and social sciences. Fanelli helps answer these and other questions by analysing patterns in the scientific literature using meta- analysis, regression and any other suitable methodology. He is a member of the Research Ethics and Bioethics Advisory Committee of Italy’s National Research Council, for which he developed the first research integrity guidelines, and of the Research Integrity Committee of the Luxembourg Agency for Research Integrity (LARI).
Readings:
Fanelli D (2019) A theory and methodology to quantify knowledge. Royal Society Open Science – doi.org/10.1098/rsos.181055. (PDF)
(Optional) Background: Fanelli D (2018) Is science really facing a reproducibility crisis, and do we need it to? PNAS –doi.org/10.1073/pnas.1708272114. (PDF)
Slides & Video Links:
D. Fanelli “How an information metric could bring truce to the statistics wars” and D. Mayo’s “Casualties” (Video link).
Mayo’s Memos: Any info or events that arise that seem relevant to share with y’all before the meeting. Please check back closer to the meeting day.
*Meeting 16 of our the general Phil Stat series which began with the LSE Seminar PH500 on May 21
PhilStatWars 
I am still a a little confused about the concept of K, so apologies if the questions below are not clear enough.
On Page 33 of your article, you write that ‘an entirely fabricated study yields no positive knowledge and yields indeed negative knowledge.’ And that ‘if independent, genuine studies confirm the made-up finding’ it still follows that ‘the extra information costs of fabricating the entire study generate a net loss of information, even if the underlying claim is correct’. In light of this I have a couple of questions:
1. Does this mean that whenever genuine studies confirm a made-up finding, K will be less than 0 independently of how many genuine studies confirm the made-up result? If not, could you clarify what is the overall knowledge produced by several studies if one of them is completely fabricated but the other ones are genuine (and confirm the made-up result)?
2. On page 40, you claim that ‘It can be shown that a ‘true justified belief corresponds to
a system for which K >0’. Could you say a little bit more about this? Particularly, in relation to my previous question?