Most published research is wrong!

The truth about the current state of scientific research

The current state of science is catastrophic. In the last 40 years we have practically witnessed the death of science. It's hard to say, but what is currently being produced in most laboratories bears little resemblance to real science. The truth about the current state of science can be summed up in three main themes, these are:

• The fact that most published research is almost certainly wrong
• The crisis of reproducibility, where over 90% of scientific work cannot be reproduced
• The flawed peer review process

Are Most Published Research Findings Wrong?

About 18 years ago, John Ioannidis published his now groundbreaking paper entitled Why Most Published Research Findings Are False. In it, he outlined a number of factors that, when present, reduce the likelihood of a given result being true. In reality, in most cases, many of these factors are present, leading Ioannidis to conclude that most of the results are wrong. At this point I would like to start. Buckle up! We're going to have a nice ride...

Ioannidis backs up his claims

There are increasing concerns that most of the research currently being published is incorrect.

Simulations show that for most study designs and settings, a research claim is more likely to be false than true. Moreover, in many contemporary scientific fields, the research claims that are claimed are often simply accurate measurements of prevailing prejudices.

The above quotes are from the summary part of Ioannidis' essay. He wasted no time in getting to the heart of the matter. According to him, there are simulations that show that a particular research claim is more likely to be false than true. It is even possible that many of them are only a measure of the prevailing bias. In his essay, he clearly backs up these seemingly bold claims. Much of what he highlights has been observed by scientists themselves in the course of their careers.

It's all about the p-value - at least that's what the scientists think

The first thing that has a big impact on the probability that a result is true is statistical significance and overuse of what is called the p-value. The New Oxford American Dictionary defines p-value as follows:

the probability that a certain statistical measure, e.g. B. the mean or standard deviation, of an assumed probability distribution will be greater than or equal to the observed results.

Scientists typically use a p-value less than 0,05 to determine statistical significance. They claim that if a given data set has such a p-value, then the result is real and probably true. Ioannidis had a different opinion on the subject.

Several methodologists have suggested that the high rate of non-replication of research results is a consequence of the convenient but ill-founded strategy of claiming conclusive research results solely on the basis of a single study assessed for formal statistical significance, typically at a p- Value less than 0,05.

Here he starts with the notion of non-replication, which we will discuss in one of the next posts. The remainder of the citation underscores his view that relying on formal statistical significance as a basis for evaluating a particular research result is poor strategy. Many scientists have relied heavily on statistical significance and continue to do so today, causing most of their results to be incorrect. Not only Ioannidis has made such a claim about the excessive use of the p-value.

Enter David Colquhoun

In 2014, David Colquhoun, a pharmacologist and statistician from UCL, published a review article in the Royal Society's journal Open Science. As we shall see, he backs up the claims of Ioannidis very clearly and starts with a bang.

If you use p=0,05 to assume you've made a discovery, you will be wrong at least 30% of the time. If, as is often the case, the experiments are not powerful enough, you will most likely be wrong.

For the first part to be correct, they would need to have an ideal test scenario. The sad truth is that in the vast majority of cases this is not the case and therefore most experiments end up being too weak, which in turn means you are wrong if you use p=0,05. Surely you are wondering what is meant by the power of an experiment or how underpower can come about, and I will get to that in a moment. Before returning to Ioannidis, I would like to quote one more quote from Colquhoun's work.

One makes oneself ridiculous when one claims to have discovered something, although one is only observing coincidence. From this point of view, what matters is the probability that a result that is "statistically significant" actually has an effect. If you find a "significant" result when nothing more than chance is involved, your result is a false positive, and the probability of getting a false positive is often frighteningly high.

It's hard not to agree with Colquhoun here. It highlights the problem of over-reliance on using the p-value when you have an underpowered experiment.

What's the deal with bias?

Next, Ioannidis targets bias. He defines it as follows:

First, we define bias as the combination of various design, data, analysis, and presentation factors that tend to produce research results when they should not be produced.

Bias can mean manipulation in the analysis or reporting of the results. Selective or biased reporting is a typical form of such bias.

Notice that he mentions the manipulation of analysis or reports as a means by which bias comes into play. I think this is a big problem. Many scientists do what they call "massaging" the data to eliminate outliers that could jeopardize their desired conclusion. He continues: "As the bias increases, the probability that a research finding is true decreases significantly." Therefore, bias is a key factor in determining the veracity of research results. Bias is ubiquitous in today's scientific research and takes many forms.

The Implications

From this Ioannidis derives several interesting conclusions which, when present, reduce the probability that a research finding is true. These are the following:

• conclusion 1: The fewer studies that are conducted in a scientific field, the less likely it is that the research results will be accurate.
• conclusion 2: The smaller the effect sizes in a scientific area, the less likely it is that the research results will be accurate.
• conclusion 3: The larger the number and the smaller the selection of the investigated connections in a scientific field, the less likely it is that the research results are correct.
• conclusion 4: The greater the flexibility in terms of design, definitions, results and methods of analysis in a scientific field, the less likely it is that the research results will be accurate.
• conclusion 5: The greater the financial and other interests and prejudices in a scientific field, the less likely it is that the research results will be correct.
• conclusion 6: The hotter a scientific field (with more scientific teams involved), the less likely it is that the research results are true.

Based on these conclusions, I would like to make a few observations. The first relates to sample size. This would be the case when only a limited number of replicate samples are tested. In a number of Covid articles I have found that when analyzing patient samples, the size of the patient groups is quite small. This is particularly surprising given that this is supposed to be a global pandemic and there shouldn't be a shortage of patients to take samples from.

The sample size is also a factor affecting the power of the experiment. Another is the effect size. Corollary 2 clarifies this. If you have a small sample size and observe a small effect, this would be an underpowered experiment, and if you use a p-value = 0,05 to determine if you have a valid result, then Colquhoun would say that you wrong in most cases.

For many years it has also been observed that the scientists are offered a great deal of flexibility in the planning of experiments and the methods of analysis. When two different lab mates perform the same experiment, in most cases they get different results. Consequently, all results are not valid. Corollary 5 shouldn't surprise anyone. In scientific research today, there is a large and growing number of conflicts of interest.

Another important aspect is funding. I know scientists who go directly to those who have the money to tell them what to do or what to find in their experiments. This is not science, of course, and it clearly affects whether a finding is true. The last conclusion is particularly applicable to the current situation with Covid. I mean, has there ever been a hotter field with new data being released all the time.

We are now even seeing a lot of data being published as preprints. That is, a work is published without going through the full peer review process. That's not to say there isn't a lot wrong with the peer review process, but we'll get to that in a future article. The point is that the data are released so quickly that they are not even verified and yet are accepted as true by many in the scientific community. I'm sure you can understand the problem. Here's a little more from Ioannidis that highlights the problem with data coming out of a hot scientific field like Covid.

This may explain why, in areas that attract a great deal of attention, we occasionally experience great excitement, which is quickly followed by great disappointment. With many teams working in the same field and large amounts of experimental data being produced, timing is of the essence to beat the competition. Therefore, each team can prioritize tracking and disseminating their most impressive "positive" results.

Ioannidis concludes his essay by asking how these problems can be solved. Unfortunately, many didn't heed his advice. There are a few more quotes that I would be remiss if I didn't mention them.

A big problem is that it's impossible to know with 100% certainty what the truth is in a research question. In this respect, the pure “gold standard” is unattainable.

Contrary to what "the experts" would have you believe, the fact is that science does not provide absolute truth. You can never know anything with 100 percent certainty. It is never final and is constantly changing.

The editor of the Lancet sums it up

Richard Horton is the Editor-in-Chief of The Lancet, a respected medical journal. In 2015, ten years after the publication of Ioannidis' essay, Horton published a powerful commentary. It was about something he had heard at a symposium "on the reproducibility and reliability of biomedical research" organized by none other than the Wellcome Trust. If you are unfamiliar with the Wellcome Trust, you should do some research. This group was instrumental in enforcing many tyrannical measures, particularly in the early days of Covid. Back to Horton's article. He begins with a remark he overheard which I think sums up the whole situation very well.

Much of what is published is wrong.

Short and sweet, but very to the point. It shouldn't fall on deaf ears that this was probably said by someone with considerable influence in the scientific world. I consider this a frank admission, even though many scientists claim the system isn't flawed. Horton continues by presenting his own thoughts, which clearly support Ioannidis' claims. He also seems to agree with the reasoning used.

The arguments against science are clear: much of the scientific literature, perhaps half, is simply untrue. Plagued by studies with small samples, tiny effects, invalid exploratory analyses, and glaring conflicts of interest, as well as an obsession with following fashion trends of dubious importance, science has strayed into obscurity.

There's not much to complain about, but Horton continues.

In their quest to tell a compelling story, scientists too often shape data to fit their preferred theory of the world. Or they adapt their hypotheses to their data.

This is now very common and many scientists have experienced this many times in their careers. It's about generating data that fits a preconceived hypothesis or settling on the story a work is intended to tell before all the data has been generated. This is clearly not science, and doing so will no doubt result in invalid results being presented. Horton concludes by asking if these bad practices can be fixed and then addresses one of the larger parts of the problem.

Part of the problem is that no one has any incentive to be right. Instead, there are incentives for scientists to be productive and innovative.

This is 100% true today as it was then and likely will continue to be as no one has any incentive to work towards a solution. The business of science has ruined real science. It has become no more than an innovation machine.

What did the CDC say?

I would like to conclude with what I believe is a fairly candid statement from a group at the CDC who published a 2007 response to the Ioannidis paper.

Essentially, they claimed to be able to demonstrate that a study that can be repeated increases the probability that a particular result is true, and that this would indeed solve Ioannidis's problem with statistical significance. They begin by admitting that what Ioannidis claims is probably true.

He elegantly showed that most of the claimed research is wrong.

Just to be clear, this is a group at the CDC that admits most published research is wrong. Read that a few times to make sure you get it. Then they talk about replication and how this problem could possibly be solved.

As part of the scientific operation, we know that replication - conducting another study that statistically confirms the same hypothesis - is the cornerstone of science and that replication of results is very important before any causal conclusions can be drawn.

Again, it is the CDC (the same one that foisted draconian measures on us during the Covid) that maintains that a repeat is crucial before determining whether a finding is true. Well, we have a little problem with replication. So much so that it has been called a crisis, a "reproducibility crisis". Just out of curiosity, has the CDC used studies that have been replicated to justify any of the actions they are proposing? I don't believe.

Discernment is key

I hope you can see from this why it is so important that we be very critical when evaluating new scientific claims, especially those that have significant implications for the way we live. Stay tuned next time we delve deeper into the replication crisis. You don't want to miss this. I really hope that you find your calm in this storm that is raging around us. And of course, don't stop questioning the science! It's never complete!

(Von Mike Donio, MS translated from English)