One of the more interesting media COVID panic media plays is to hype a bad research paper because it’s published in a high impact journal and fits a predestined narrative. Today we’ll examine two instances where this happened in the prestigious medical journal, The Lancet. Given that this type of publication is often relied on to drive public policy, this trend is downright scary.
What Is “The Lancet”?
The Lancet is one of the world’s most prestigious medical journals, founded in 1823. It’s the second-ranked journal in the world out of 160 journals in medicine. Basically, it’s right up there with the New England Journal of Medicine.
Media Panic and the Hydroxychloroquine Study
I’ve blogged in the past on the world’s most controversial COVID-19 drug, hydroxychloroquine. This Malaria and arthritis drug continues to be studied in clinical trials for treating COVID-19, but about a month ago, a paper was published in The Lancet which seemed to show that the drug had dangerous side effects (1). This was bizarre to physicians familiar with the drug because it had already been extensively studied showing it was quite safe.
The paper purported to have data on 96,000 COVID-19 patients from hundreds of worldwide hospitals and about 1,800 patients who had received hydroxychloroquine. This publication caused an explosion of COVID panic stories in the media and as a result of the pressure, multiple critical hydroxychloroquine studies around the world were halted. The authors claimed that heart-related deaths were sharply higher in the patients that got the drug, but questions soon arose about the accuracy of the data.
Eventually, the Guardian newspaper (which gets a hat tip) had the temerity to look into the data (2). Regrettably, the small company that claimed to have extracted this information from medical records using an AI system wouldn’t release the source data. There were also serious questions about whether the company had access to accurate data as some of the information that was reviewed was clearly wrong. This caused both The Lancet and the New England Journal (who had also published a paper based on data from this company) to issue retractions (3,9).
The New Lancet Disaster Paper
This week a spate of headlines from the panic fanning COVID media read:
First-of-its-kind study shows face masks greatly reduce risk of COVID-19-ABC Today Show (6)
- Physical distancing, masks can dramatically reduce COVID-19 transmission: Study–
“These fundamental public health measures are going to remain important.”-ABC News (7)
- The Lancet: Most comprehensive study to date provides evidence on optimal physical distancing, face masks, and eye protection to prevent spread of COVID-19-First review of all available evidence including 172 observational studies looking at how physical distancing, face masks, and eye protection affect the spread of COVID-19, SARS, and MERS in both community and healthcare settings across 16 countries.AANS EurekAlert (8)
The rest of the headlines were basically the same. Masks and physical distancing had been PROVEN to slow the spread of COVID-19 based on this new meta-analysis! Stop the presses, we in the COVID panic media have been right all along!
Not so fast lazy science journalists. There is, of course, reality to deal with, so let’s dig into this paper.
What the Heck Is a Meta-analysis?
A meta-analysis is when a scientist takes the raw data from many different studies and combines that data to try and answer a question. They are almost always done looking at data from high-level randomized studies. Why? The problem is that the review can be garbage in/garbage out depending on the accuracy of the data you’re reviewing. Meaning, in this case, the quality of the information that went into this meta-analysis must be stellar to earn these headlines.
Pre-print vs. Published
Before we dive into the stinking pile of horse dung called a meta-analysis which is sure to be the Lancet’s next big COVID retraction, let’s learn a little about the games that scientists and the media have been playing during the pandemic. One of the bizarre phenomena has been the meteoric rise of pre-print servers. What’s that?
Normally scientific papers go through peer-review. That means that there is a rigorous process where other random experts (scientists and doctors) look at the veracity of what the paper claims versus what the data will support. Having been through that more than 30 times, it’s often brutal. The paper that may have taken years to create is often viciously torn to shreds by nameless strangers and hopefully, it survives this gauntlet and gets published.
However, during the pandemic, doctors and scientists have been taking a short-cut around this arduous peer-review step. Instead, they just take their paper and upload it to a pre-print server, no fuss no muss. However, since that’s really skipping a critical screening process, here’s the warning that most pre-print servers use:
“This article is a preprint and has not been peer-reviewed [what does this mean?]. It reports new medical research that has yet to be evaluated and so should not be used to guide clinical practice.”
Meaning, whatever is sitting on a pre-print server without going through peer-review is garbage is until proven otherwise.
What Would be Required of a Meta-Analysis to Earn those Glowing Headlines
Given that the headlines say that this groundbreaking new study has confirmed that public mask use and social distancing are our tickets to a COVID free world, the meta-analysis would have to:
- Only use peer-reviewed studies (i.e. no pre-print server studies)
- Use studies that are high-level, randomized controlled trials or similar. Meaning the research needs to have tested large groups of the public with and without masks and the public with and without social distancing. Or the researchers had better have tried social distancing at one meter (and measured their compliance accurately with proximity sensors) and two meters and then looked at infection rates.
- Have studies that are all focused on COVID-19
- Be focused on the general public (i.e. NOT healthcare workers in a controlled hospital setting)
So how did his new meta-analysis fare? As you might have guessed by now, VERY poorly.
Digging into the New Lancet Disaster
The paper is by the COVID-19 Systematic Urgent Review Group Effort (SURGE) study authors (10). Wow! That sounds really snappy! Regrettably, that cool acronym is the only thing this paper has going for it.
The authors claim to have reviewed 172 studies that included tens of thousands of people, but it looks like they largely settled on 44 existing research studies, some of which have studied as few as a single-digit number of patients. The number of subjects reported by the SURGE authors in these studies is often wildly inflated by contact tracing. Meaning, if the study had one COVID-19 patient and then you traced 50 contacts back to that one patient, is the study of one person or of 50? These authors went with the larger number.
So what were these 44 research papers about? Only 7/44 were published on COVID-19, the rest were on SARS and MERS (i.e. not COVID)! Also, only 8 of the 44 papers were about the general public and the rest were about healthcare workers. Only 3 of those papers on the general public were about COVID-19. As you’ll see below, almost all of the COVID papers were not capable of supporting the author’s conclusions.
The House of Cards Supporting the Meta-analysis
Let’s dive into the source papers on COVID-19 because if they aren’t stellar, this meta-analysis is a joke.
The first study listed has many issues. First, it was never actually published, but just posted on a pre-print server. Second, it never even mentions anything about mask use or social distancing and concludes that nurses working double shifts and/or who were more stressed out were more likely to get COVID-19 (4).
The next study is frankly even worse! It also has never been published, but just slapped up on an online server. It’s the contact tracing results of 9 (yes you read that right) patients with early COVID-19 in the U.S. (5). It contains no details on mask use or social distancing that would allow the authors of our review paper to create any kind of recommendations or crunch any data.
We then go on to yet another study which has never been published! Are you seeing a trend here? This is a study of 32 confirmed patients again looking at contact tracing (11). It concluded that COVID is more contagious in the first 5 days of symptom onset and among close contacts (i.e. family and friends). Again, this paper contains no data regarding masks or social distancing that could be analyzed by the authors to draw the conclusions they ultimately made.
The next picadillo is a case report of a single patient (which is actually published in a real journal!) and this person’s contacts in the healthcare system who had no idea that he had COVID-19 and took no precautions (12). Again, nothing in this paper would allow the authors to draw any of the conclusions that they ultimately reported.
Then we finally find our first paper that was actually about masks! Hallelujah! It is published, sort of… It’s not an actual research paper, but a letter to the editor from some Chinese physicians that got published in a journal (13). They looked back at their experiences of about 500 medical staff working with COVID patients. Those that wore N-95 masks and who washed their hands frequently were less likely to get infected. While all of this sounds good, this study is what’s called retrospective, or the lowest quality data because it’s what people remember happened. For example, how close really did each worker get to the COVID-19 patient? Nobody ever measured this on video surveillance or with proximity sensors, so how would you know if it was less than 1 meter, 1-2 meters, or more than 2 meters? The answer is that most of this data would be garbage.
The next study was also never published, but just posted online (14). It looked back on healthcare workers in the neurosurgery departments of multiple Chinese hospitals. In the end, 120 healthcare workers out of more than five thousand became infected. Healthcare workers that used n-95 masks were less likely to be infected. It has all of the same issues as the study above. Nobody actually ever measured anything with regard to physical distance or looked at the compliance of face mask use. For example, Dr. Wang was issued an N-95 respirator, did he use it 100% of the time while seeing COVID patients? Or did he really keep it stuffed in his lab coat pocket most of the time because he was young and invincible?
Are you seeing a trend here? This is a meta-analysis of mostly of SARS and MERS papers. Most of these COVID-19 studies are NOT PUBLISHED! Most of those studies are about healthcare workers, NOT THE PUBLIC. The few that are about the public never tested mask use versus none, or how far people were from COVID patients and then looked at which people got sick, they are simple reports of contact tracing. All contact tracing is someone’s poor recollection of what happened.
Let’s try a thought experiment. Think back on the last week prior to the shutdown. You went about your business, went to work, went to stores, restaurants, malls, or elsewhere. Now tell me how many people coughed or sneezed while they were within 6 feet of you. How many people came within 3 feet, 3-6 feet, or were more than 6 feet from you? I don’t know about you, but I have no clue. In addition, outside of my common haunts, I couldn’t tell you where I went. That’s the type of poor data that contact tracing relies on…
You really can’t make this stuff up.
How Did the Thing Get Published in One of the World’s Top Journals?
If you haven’t guessed it by now, this paper can’t justify its conclusions that we all need to wear masks and social distance to prevent the spread of COVID-19. In fact, the authors acknowledged this:
“The primary limitation of our study is that all studies were non-randomised, not always fully adjusted, and might suffer from recall and measurement bias (eg, direct contact in some studies might not be measuring near distance).”
Frankly, this paper never should have been written in the first place. Why? There simply isn’t enough published data on mask use and social distancing on COVID-19 to warrant a meta-analysis. It’s basically a garbage in-garbage out opinion piece that uses poor quality information and fancy-looking meta-analysis math to draw conclusions that fit a narrative.
The bigger question is how did this thing get published in the Lancet? How did the two recently retracted papers from The Lancet and the NEJM get published? These top-tier journals have obviously let their publishing standards fall to levels equivalent to predatory journals on COVID-19 topics. Why? To get papers out there that will fit a media narrative and get attention? To keep hits up on websites? To get in front of the world on COVID? Your guess is as good as mine, but it’s sure sad to see the reputation of centuries-old journals get ruined in a few crazy months.
WHO Funded this Paper
Almost like the Abbott and Costello sketch “Who’s on first” (see below), WHO funded this paper? Now that’s WHO and not “Who”. Yes, the World Health Organization paid SURGE for this really bad paper. So WHO is on first and SURGE is on second…
The upshot? This paper on mask use is an unmitigated disaster. No public health policy decisions should be based on this work. Frankly, in my opinion, it’s the Lancet’s next retraction in the making, and really disheartening to see a prestigious journal stoop so low.
(1) Mehra MR, Desai SS, Ruschitzka F, Patel AN. Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis [published online ahead of print, 2020 May 22] [published correction appears in Lancet. 2020 May 30;:]. Lancet. 2020;S0140-6736(20)31180-6. doi:10.1016/S0140-6736(20)31180-6
(2) The Gaurdian. Covid-19: Lancet retracts paper that halted hydroxychloroquine trials. https://www.theguardian.com/world/2020/jun/04/covid-19-lancet-retracts-paper-that-halted-hydroxychloroquine-trials#maincontent Accessed 6/4/20.
(3) New England Journal of Medicine. Retraction: Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19. June 4, 2020
(4) Bai Y, Wang X, Huang Q, et al. SARS-CoV-2 infection in health care workers: a retrospective analysis and a model study. medRxiv. 2020; (published online April 1.) (preprint). DOI: 10.1101/2020.03.29.20047159
(5) Burke RM, Balter S, Barnes E. et al. Enhanced contact investigations for nine early travel-related cases of SARS-CoV-2 in the United States.
medRxiv. 2020; (published online May 3.) (preprint). DOI: 10.1101/2020.04.27.20081901
(6) ABC Today Show. First-of-its-kind study shows face masks greatly reduce risk of COVID-19. https://www.today.com/health/do-face-masks-work-against-coronavirus-study-finds-masks-social-t183018 Accessed 6/4/20.
(7) ABC News. Physical distancing, masks can dramatically reduce COVID-19 transmission: Study. https://abcnews.go.com/Health/optimal-physical-distancing-facial-protection-reduce-covid-19/story?id=71019436 Accessed 6/4/20.
(8) American Academy for the Advancement of Scinece-Eurek Alert. The Lancet: Most comprehensive study to date provides evidence on optimal physical distancing, face masks, and eye protection to prevent spread of COVID-19. https://www.eurekalert.org/pub_releases/2020-06/tl-pss060120.php Accessed 6/4/20.
(9) The Lancet. Retraction: “Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis”. https://www.thelancet.com/lancet/article/s0140673620313246 Accessed 6/5/20.
(10) Chu DK, Akl EA, Duda S, et al. Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis [published online ahead of print, 2020 Jun 1]. Lancet. 2020;S0140-6736(20)31142-9. doi:10.1016/S0140-6736(20)31142-9
(11) Cheng HY, Jian SW, et al. Taiwan COVID-19 outbreak investigation team. High transmissibility of COVID-19 near symptom onset. medRxiv 2020.03.18.20034561; doi: https://doi.org/10.1101/2020.03.18.20034561
(12) Heinzerling A, Stuckey MJ, Scheuer T, et al. Transmission of COVID-19 to Health Care Personnel During Exposures to a Hospitalized Patient – Solano County, California, February 2020. MMWR Morb Mortal Wkly Rep. 2020;69(15):472‐476. Published 2020 Apr 17. doi:10.15585/mmwr.mm6915e5
(13) Wang X, Pan Z, Cheng Z. Association between 2019-nCoV transmission and N95 respirator use. J Hosp Infect. 2020;105(1):104‐105. doi:10.1016/j.jhin.2020.02.021
(14) Wang Q, et al. Epidemiological characteristics of COVID-19 in medical staff members of neurosurgery departments in Hubei province: A multicentre descriptive study. medRxiv 2020.04.20.20064899; doi: https://doi.org/10.1101/2020.04.20.20064899