This answer focuses on biomedical research. The draft paper () referenced in the question, “What’s great and terrible about this draft paper on top journals publishing the least reliable science” is both great and terrible.
- It is great that it claims using data analysis that top journals publish the least reliable science since bold claims that foster open and vigorous debate are a hallmark of science.
- If independently verified, top journals publishing the least reliable science is terrible since doing so weakens both science itself and the public’s trust in it.
Obsessive & Relentless Focus on Scientific Novelty: The Perverse Incentive that Animates the Entire Biomedical Research Enterprise
That top journals publish the least reliable science shouldn’t be surprising since novelty rather than reliability is their prerogative. The prestige accruing to top science journals is precisely owing to their self-professed ‘high’ and ‘exclusive’ standards that mandate high degree of novelty in the research published by them. So vociferous and pervasive is this demand for novelty that it percolates down the scientific journal food chain. The degree of the novelty demand is the only difference in that standards for novelty are a lot more stringent with the top journals.
It’s self-evident that obsessive focus on novelty could harm biomedical research enterprise by causing irreproducible data to pile up. Problem is current biomedical research enterprise is locked into a vicious self-fulfilling cycle driven almost entirely by such perverse incentives. To understand how such an unfortunate state of affairs dominates biomedical research requires taking a step back to look at the whole picture and not just the scientific journal part of it.
The foundations of the present day scientific enterprise were laid in the post-WW II era. In the US this was largely the doing of mega-administrators such as. Details may vary from country to country but a largely similar biomedical research enterprise structure emerged the world over. The major stakeholders and decision makers within the modern day biomedical research enterprise are (again, details in this answer focus on the US),
Employers: Universities/research institutions along with their presidents, senior management, department heads, professors.
Funding agencies: NIH, NSF, non-profits, foundations, pharma, etc.
Scientific publications: Scientific output gatekeepers who publish original research findings after they’re peer-reviewed.
Today, a biomedical researcher wishing to become an academic starts by getting a Ph.D. (or M.D. or MD-PhD) followed by several years of post-doctoral training. Hopefully this process ends with a faculty position somewhere.
- How to be competitive in getting a faculty position? Scientific publications, the more prestigious the journal, the better the chances.
- How to be competitive in getting the grant funding necessary to become an independent researcher? Scientific publications, the more prestigious the journal, the better the chances.
- How to be a keynote speaker at a prestigious scientific conference? Scientific publications, the more prestigious the journal, the better the chances.
- How to become an editor on top scientific journals? Scientific publications, the more prestigious the journal, the better the chances.
- How to …? Rinse and repeat ad nauseam.
- And what is the most stringent stipulation of top scientific journals? Novelty, not reproducibility.
This in a nutshell is how the perverse incentive for novelty animates the present day biomedical research enterprise structure, one where each of these stakeholders has contributed to creating and maintaining a scientific culture girded almost entirely by this one perverse incentive.
Unrealistic Demand for Scientific Research Novelty Uncouples It from Reproducibility & Fuels Unethical Data Selection Practices
Is it inevitable for scientific novelty and reproducibility to be separate? Absent an a priori reason to separate them, how the scientific enterprise operates in practice makes them so. At its core, scientific research is a journey into the unknown, even sometimes without a navigator. Research results meeting stakeholder demand for novelty cannot be simply churned out periodically on cue and yet that’s what the current system demands. Predictably, biomedical research scientists respond to such obviously untenable pressure toby cutting corners.
Cutting corners entails doing all that’s necessary to make a viable paper, a veritable ‘sausage’, that will pass muster in the novelty stakes. This demand in turn drives practices such as
- Suppressing negative data, the so-called .
- Engaging in creative statistical analysis, the so-called p-hacking, .
Though falling short of outright fraud, such practices are pervasive throughout the biomedical research enterprise, simply a predictable response to relentless and unrealistic demand for far too frequent output of scientific research novelty.
This is also why efforts of self-proclaimed watchdogs such asare so much pablum and fluff. They miss the point almost entirely since outright fraud remains a small part of scientific enterprise and thankfully so. Rather the problem is far more consequential and far more pervasive, namely, that official watchdogs such as the (ORI) define scientific fraud far too narrowly, limiting it to three specific acts,
- Data Fabrication: Making data up from scratch.
- Data Falsification: Tampering with and improperly altering data.
- Plagiarism: Stealing another’s ideas and words without their consent.
Such a narrow purview leaves a titanic-sized hole that can be filled with the slew of pervasive unethical, not illegal, cutting corners practices outlined above that fall under the rubric of data selection or data cherry picking (colloquially referred to as ‘cooking’ or ‘massaging’ the data).
Twelve years sinceand his 2005 shot across the bow, ‘Why Most Published Research Findings Are False’, , what is it but absurd to still publicly debate the same issue of lack of reproducibility in biomedical research? Nothing has changed since reproducibility still isn’t a priority for any of the three major stakeholders (employers, funders, scientific journals).
With biomedical research scientists not being held accountable if their novel results turn out irreproducible, another generation of academic wannabes graduates from within the same ecosystem and operates in the same culture that demands fealty to scientific novelty and thus the cycle perpetuates.
The entire issue of irreproducibility at the heart of biomedical research thus naturally and automatically sloughs off of stakeholder and practitioner alike to become and remain someone else’s problem. That begs the absurd question of who that someone else is or could be, absurd since this tautology only exists from science stakeholders and practitioners alike being as yet unable to openly admit that their unwavering demand for unrealistically frequent output of scientific novelty is the main reason this problem exists in the first place, one that’s only exacerbated by official scientific watchdogs turning a blind eye to unethical practices that inevitably crop up to sustain such unrealistic demands for output.