As The COVID-19 pandemic has spread. Scientific experts and policy makers have had to work closely to keep up with the latest scientific evidence and implement guidelines to prevent further transmission of the virus. In the meantime, the pace at which COVID-19 research has been released has meant that misinformation has spread widely and sometimes crept into the upper echelons of government.
To see how well policymakers around the world have incorporated sound COVID-19 research into their decisions, the researchers examined scientific citations in 37,725 policy papers from 114 countries and 55 intergovernmental organizations drawn up between early January and late May last year. In a study published in science yesterday (January 7th) they discovered that these documents related to COVID-19 papers, which contained 40 times more citations on average than other COVID-19 papers. Peer-reviewed articles from leading scientific journals were far more prominent in political documents than preprints, the researchers found.
The scientist spoke to Northwestern University economist Benjamin Jones, a co-author of the new study, about why these results surprised him.
KELLOGG SCHOOL OF MANAGEMENT, NORTHWESTERN UNIVERSITY
Benjamin Jones: First, conceptually there are a number of questions and concerns about the extent to which science is used in politics – is politics even paying attention to science? But does it also pay attention to good science? This goes back to CP Schnee’s investigation into World War II of the importance of science to the war effort and who was heard. [Editor’s note: Chemist and novelist C.P. Snow wrote The Two Cultures, an influential essay in which he lamented the cultural divide separating the arts and sciences.] I think there is a wide range of questions and concerns about how politics relies on science and this has long been explored.
The second channel is that big data has researched all sorts of questions. We have made increasing use of these incredible datasets in every scientific publication. And more recently we see an attempt to see how science is used in other areas. I’ve done some work on how science is used, for example in patenting, and we’re seeing work on how science is used in the media, but now we’re actually seeing better and better data opportunities for science referenced in the directive.
We had a new record, Overton, which allowed us to go into it, and then of course we have COVID on our mind. So this is a chance to really see this interaction in real time.
BJ: In a sense, we’re doing something very basic – the very first thing you would do – which is to see this explosion of science on COVID, and then we have the data produced with pretty much real-time policy documents that are in place right now. It was an opportunity to see if all of the policies that have been developed around COVID actually know and point to all of this science related to COVID. Basically, we are bringing this huge global database of policy documents together [with] this huge database of developing science. We can both see if it’s linked, and then we can see if it’s linked to the good science or the less scrutinized science, and we can see who’s making the link – which institutions in the policy area seem important to to bridge this science and bring these ideas and insights into politics.
BJ: The classic idea is that [policymakers] are just a completely different community of people with different expertise. Does the average policymaker understand the physics, chemistry, or biology of infectious diseases? Probably not. How do you even know what to quote and judge? And in the current social media context or in certain political contexts [where] There is misinformation, there is even more concern that people are not even trying to access the correct knowledge, but rather that they choose what matters rather than thinking about what good science is saying.
We were quite pleasantly surprised that the main finding is that all of these new policy documents are able to connect to the new science – they don’t ignore it; They are leaning on – and more importantly, they seem to be leaning on the powerful science – the stuff that scientists themselves recognize as important, see peer-reviewed, and appear in the audited locations.
We were quite pleasantly surprised that the main finding is that all of these new policy documents are able to connect to the new science – they don’t ignore it; They fall back on it – and more importantly, they seem to fall back on highly influential science.
– Benjamin Jones, Northwestern University
BJ: Aside from the fact that there is a strong connection to quality which is comforting, I was impressed with how important the World Health Organization is in the picture. What we have seen is that different national governments have different tendencies to go directly to science. But, in general, governments use it less and these international intergovernmental organizations like WHO use it much more accordingly. In fact, much of the way science gets into policy documents at the national level is indirect. So basically, the World Health Organization, for example, is writing something, pulling out all of this latest science, and then other governments are quoting this WHO policy document instead of quoting the science directly.
This is interesting because we obviously see that different countries have taken very different approaches to the pandemic, and we see that there is tremendous heterogeneity in the way science is used in the data that not everyone is listening. The good comes into politics, but that doesn’t mean that all policymakers or governments are listening. We see that places like WHO, which of course have been denounced by some political actors in this pandemic, clearly play a very central role. I mean, they really are the center of this whole network that brings science into politics.
BJ: We saw a few [preprints] that got a lot of media attention and then it turned out to be maybe not right. One thing that we wondered about this question, what kind of science is being resorted to, was whether the political community would do the same and dive into these brand new papers that have not been reviewed. That might not be a bad thing because speed is important in the pandemic.
But what we are really seeing very strongly is that of all of the COVID things popping up on preprint servers, the likelihood that they will be used is far less. In the guideline documents, the accepted peer review work is very strongly emphasized. That’s interesting – I think it suggests that magazines continue to play a very important role in choosing the decision makers they want to fall back on.
BJ: The obstacle is not that policymakers simply do not necessarily know or cannot know about science. The obstacle isn’t that you can’t figure out what the right science is – when you quote it, you’re citing the good stuff. It seems like the barrier to effective use of science in politics will depend heavily on people in different governments and different parts of government really listening. This suggests that the use of science in politics is an idiosyncratic national obstacle. And that, of course, is in line with what we’ve seen and seen during this pandemic.
Y. Yin et al., “Political and Scientific Coevolution During the Pandemic,” science, doi: 10.1126 / science.abe3084, 2020.
Editor’s note: The interview has been edited for brevity and clarity.