On the Nature and Value of Consensus

About 20 years ago, I remember participating in a meeting where participants were actively debating a proposition. The moderator was enforcing parliamentary procedure, and speeches for and against the proposition were heard in turn - one speech for, then one speech against. This being the first meeting I had been in following this kind of procedure, I began to think that the room was evenly divided between the two positions. I was strongly in favor of it, and I was disheartened to see so much opposition to it. But when the vote was finally taken, I was more surprised by the results. The proposition passed by an extraordinarily large margin. In fact, it appeared that those speaking against the proposition made up almost all the people that voted against it, and the vast majority of the people in the room were like me. Most of us had essentially made up our minds, and we had no intention of making any speeches; we were just waiting for the debate to be over so we could vote. 

My impression of the room before the vote was subject to a kind of selection bias. The proportion of people speaking for vs against the proposition was not representative of the positions held by those in the room.

The rules of debate were set this way for at least two reasons: 1) to give all sides equal opportunity to argue for their position and 2) to help mitigate against people voting for the position that seemed most popular. You would have to make your decision based on weighing the arguments presented, not by simply considering which position was most popular. However, it would be fallacious to conclude anything about the proportion of the room in agreement with the proposition based on the proportion of people speaking in favor of it. The sample wasn't representative.

There is a similar dynamic affecting the climate debate. A subset of scientists are speaking out as proponents of or detractors to AGW. These scientists are showing up on TV news programs, giving congressional testimony, writing on blogs, and participating in various ways in the public conversation. Others are just doing their work and publishing their research in the peer-reviewed literature, teaching courses in major universities, and collecting the data that can be used to assess AGW and its impacts on our future. The proportion of people speaking in these venues is inevitably going to be subject to selection bias - it's not a representative sample. In the public conversation, those with positions against AGW are inviting scientists who share their views to speak on their shows. Likewise, those with positions for AGW are doing the same. This selection bias can create a false impression that climate scientists are more evenly divided in their positions than they actually are, and this is one reason why the topic of "consensus" has become such a significant and divisive component of the public AGW debate.

Defining Consensus

There are some who object to the very notion of consensus. We hear claims like "there is no consensus in science." This, of course is preposterous. While consensus is almost never 100%, scientists can and do come to an overwhelming consensus on many points. In Geology, for instance, the theory of plate tectonics would be an example of this kind of consensus. While it took some time for the evidence to generate this consensus, there is now undeniably a consensus among Geologists on plate tectonics, and it has been revolutionary in the way scientists have understood many of the Earth's geologically significant processes.

However, in order to assess the nature and value of a consensus, we need first to definite it, and there are multiple ways to do so. But it's important first to decide what we're going to measure - what is it that we want to know if there is a consensus about? For the purposes of this post, I'm going to define this as, "human activity is the dominant cause for global warming above the 1850-1900 mean." I'm going to refer to this simply as AGW. In other posts, I've shown that the best evidence we have is that natural forcings have been more or less negligible over this time frame, and warming has been dominated by human emissions of GHGs, masked in part by aerosol pollution. This is the overwhelming position in the scientific literature, including IPCC assessment reports.

Second, we need a method to measure the consensus on this point. There are at least two ways we can do this. First, we can survey the opinions of scientists. You can find the results of surveys and petitions online, either of scientists expressing concern about AGW or of professionals (a few of which are scientists) saying there's no climate emergency. I don't find this approach persuasive, partly because it's hard to get a percentage of all qualified scientists having an opinion on something and partly because it focuses on the people, not the evidence. A second approach is to attempt to quantify the positions expressed in the peer-reviewed literature on AGW. This gives us a way to come up with a reliable percentage. Searching indexes of the peer reviewed literature, we can find the vast majority of papers taking a position on AGW, and we can then assess the percentage of the literature that agrees with AGW. This has the benefit of focusing on the literature where primary research is published, so it gives us a sense of what the published scientific evidence is currently saying. It has the weakness of being time an labor intensive to quantify. Teams of people have to read papers (or at least abstracts) to to come up with the percentage consensus on AGW. This means any result is going to be an estimate with an error - there is the possibility of a paper being wrongly categorized, either as to whether it expresses a position on AGW or whether it agrees or disagrees with AGW. So any number we arrive at will be somewhat approximate. But this approach does the best at focusing on the published evidence concerning AGW rather than on the opinions of people.

Existence of Consensus

There are two papers recently published that attempt to quantify and assess the percent consensus about AGW following the second method. This method focuses more on the published evidence and less on the people. The first received a great deal of attention about 10 years ago and has been repeated (sometimes incorrectly) in political discussions. The second is is a follow up to the first, and in my opinion it is a better paper.

Cook 2013

John Cook[1] undertook the task of attempting to quantify the amount of published research that agrees with AGW. The study consisted of two parts. In part 1, Cook and his team collected and evaluated 11,944 abstracts to papers that mentioned "global climate change" or "global warming" published between 1991 and 2011. Most of these papers didn't address whether human activity was the dominant cause of global warming. Only about 33.6% took a position at all - "32.6% endorsed AGW, 0.7% rejected AGW and 0.3% were uncertain about the cause of global warming." So of those papers that took a position on AGW, 97.1% endorsed AGW.

In part 2, Cook and his team asked the authors of these papers to evaluate their papers. While the first part focused on the abstracts of papers, in the second part, authors were asked to evaluate the whole paper.  Here the consensus was even stronger. A smaller percentage of papers were rated as taking no position on AGW, and 97.2% of the author-rated papers that took a position agreed with AGW.  Cook plotted the percent endorsement per year and found a marginal increase in the strength of the consensus with time. The paper concluded that "the number of papers rejecting the consensus on AGW is a vanishingly small proportion of the published research."

For anyone familiar with the literature, theses results were unsurprising. Most of the climate scientists that object to AGW are retiring, and some that remain active are no longer attempting to publish research in peer-reviewed journals. Certainly a few remain, but any more their research rarely appears in prestigious or high impact journals. Yet this paper was met with an astonishing amount of criticism. Some incorrectly objected that Cook should not have excluded the papers that didn't address the attribution of global warming to human activity. Irrelevant papers were correctly not counted, since they were irrelevant, and the statistic in the paper was correct - 97% of the literature taking a position on AGW agreed with the consensus position. Others correctly objected at the sloppy ways this paper's findings were reported. For instance, some said 97% of scientists think AGW is dangerous, but Cook's paper only addressed the attribution of global warming, not the danger associated with global warming.

Richard Tol published a paper[2] criticizing the paper, and while his criticisms were strong, he did not argue against the main conclusion of the paper. In fact, his position was one that I want to return to later: "There is no doubt in my mind that the literature on climate change overwhelmingly supports the hypothesis that climate change is caused by humans. I have very little reason to doubt that the consensus is indeed correct." Despite the flaws he believed to be in this paper, the paper's conclusion was essentially correct, and there was no need for this paper to be written to establish that. Tol wrote in a comment about this paper, "The consensus is of course in the high nineties. No one ever said it was not. We don’t need Cook’s survey to tell us that. Cook’s paper tries to put a precise number on something everyone knows."[3]

Lynas 2021

Last year, Mark Lynas published an update to Cook's paper[4] to see if the consensus remained as strong in papers published since the end of Cook's study period. Lynas found 88,125 climate-related papers published in 2012 or later. Of those, he examined a random sample of 3,000 papers from the full 88,125 papers. Lynas found 4 abstracts in the 3,000 paper subset that were either explicitly or implicitly skeptical of AGW. He then used a sample-weighted search with keywords biased to find skeptical papers in the whole dataset. Using this approach he found 28 papers from the entire 88,125 paper dataset that were either explicitly or implicitly skeptical. Between the two approaches, they found a total of 31 skeptical papers (one paper was found by both methods). Lynas concluded "with high statistical confidence that the scientific consensus on human-caused contemporary climate change—expressed as a proportion of the total publications—exceeds 99% in the peer reviewed scientific literature."

I have one quibble about this paper. Lynas decided to count the "no position" papers as favoring the consensus. I disagree with this because 1) it makes Lynas' statistic a bit of an apples to oranges comparison with Cook's, and 2) it assumes that virtually all the no position papers don't state a position because they accept the consensus. This assumption is at least possibly false, so the stat should be taken from papers that address the attribution of global warming to human activity. However, Lynas' paper does calculate the consensus with those papers excluded. "If we... further exclude papers that take no position on AGW (i.e. those rated 4a), we estimate the proportion of consensus papers to be 99.53% with the 95% confidence interval being 98.80%–99.87%." Lynas' paper indicates that the percent consensus in the last decade has increased from ~97% to ~99%. The consensus on AGW is strengthening with time.

Value of Consensus

One of the biggest concerns I have about this whole enterprise is, why do we have to quantify the level of consensus there is on AGW? There are very few scientific issues where consensus is quantified. The closest equivalent I can think of is "Project Steve." In response to lists of scientists who doubt evolution, Project Steve responded with a tongue in cheek list of scientists named Steve (about 1% of scientists) who accept evolution.[5] But in most issues, there is no need to quantify any consensus. Once the evidence reaches a critical mass, scientists come to see matters as "settled," and move on to other matters. Those matters that had been "settled" can certainly be reconsidered if new, contradictory evidence comes in, but rarely if ever is there any need to quantify a consensus or how settled a scientific issue is. 

So why do we do this with AGW? I believe the strongest answer to this is that if AGW is correct and dangerous, it's critical that we act quickly to keep the most dangerous effects of AGW from happening. Since some are saying (wrongly) that AGW is still a matter of open debate and uncertainties are still large, Cook and others have undertaken the task of showing that this in fact is not the case. The scientific research has achieved a near unanimous conclusion on AGW. The consensus is being quantified for political reasons so that voters and policy makers can see that the impression they may be getting in media presentations and congressional testimony is subject to a strong sample bias. The debates we see on news shows and in government testimony are not representative of the whole of scientific research. And while it may be valid to hear the best arguments of both sides, the reality is that one side is vanishingly small and shrinking, and the evidence no longer supports continued scientific debate on AGW. The clear weight of the peer-reviewed evidence is nearly unanimous in favor of the consensus. However, would like to clarify what this does and doesn't mean.

1. The greatest value of these consensus papers is that it makes clear that AGW is an issue about which there is no longer any meaningful debate. Climate scientists, for the most part, are not debating whether human activity is the dominant cause of warming above the 1850-1900 mean. They have moved on to other aspects of climate science.
2. The fact that consensus exists and we can consider AGW "settled science" is not proof that it's correct. Further evidence could in principle come in that would overturn the current consensus.
3. The fact that consensus exists on AGW doesn't mean that there no aspects of climate science that are less certain. Scientists are still working to constrain estimates for ECS, understand how quickly ice sheets may collapse, how AGW will affect tropical storms, etc. Nobody is suggesting that all matters related to AGW are settled or that there are no remaining areas of uncertainty.
4. Aside from the political motivations behind Cook 2013, there was never any reason to quantify the consensus, as best I can tell. In that sense, Richard Tol is right. The fact that a near unanimous consensus exists was already obvious before any of these studies were conducted, and we didn't need this study to know that.
5. However, Tol was wrong to say that no one ever said the consensus was not in the high nineties. People say that frequently and still do, and the fact that they do is part of the reason why the consensus was quantified. The publication of a quantified consensus may well have increased the volume of those who say there is no consensus on AGW. Cook's paper was a response to a clear denial that a strong consensus exists already present among the public and politicians.
6. In debating AGW, it's always wrong to use these stats for or against the validity of AGW. AGW is almost certainly correct because the evidence overwhelmingly supports it, not because the consensus exists.
7. Focusing on the percent consensus can breed conspiratorial thinking and polarize the public. Some people get the impression that there are forces looking to enforce a consensus and  people fall in line with it, rather than simply quantifying a consensus that exists because of the evidence.
8. We should not expect policy makers to be scientifically literate or keep up with the literature, and as much as I dislike the notion of quantifying the consensus, I can see it as being valuable for those willing to listen to the evidence. However, it seems to me generally speaking that the 97% statistic has only been persuasive for those that were already persuaded. Those who reject the consensus for political reasons are not likely to accept it because they're told they reject what almost all scientific research supports.
9. Consensus may well exist on matters of climate science that were not quantified in the above studies, particularly in areas of the expected consequences of AGW.
10. The best consensus documents are still the IPCC assessment reports, which do more than just survey the literature for statements affirming AGW. They quantify the certainties with respect to the evidence we currently have regarding various aspects of climate change, including its attribution to human activity, ECS and the various dangers associated with AGW.

We have to live in a world where the consensus on AGW has been quantified. I think it pretty obvious that these papers are essentially correct in their assessment of the literature. However, I suspect that the way this consensus has been advertised has only served to polarize people in the U.S. Those who already accepted the evidence may see this as reason to become more antagonistic towards the opponents of AGW and the need to replace fossil fuels with carbon neutral sources of energy. Those that reject AGW may see this as an attempt by their political adversaries to enforce a consensus and punish detractors, fueling conspiratorial thinking. Cook and Lynas are undoubtedly correct; what I question is whether advertising the quantification of consensus is good or bad.

References:

[1] John Cook et al, 2013, Environmental Research Letters 8 (024024). DOI 10.1088/1748-9326/8/2/024024

[2] Richard S J Tol, 2016, Environmental Research Letters. 11 (048001). DOI 10.1088/1748-9326/11/4/048001

[3] https://andthentheresphysics.wordpress.com/2013/06/10/richard-tols-fourth-draft/#comment-822

[4] Mark Lynas et al 2021, Environmental Research Letters 16 (114005). DOI 10.1088/1748-9326/ac2966

[5] https://ncse.ngo/project-steve