Has the "Hockey Stick" Been Disproven? Part 1 - Energy & Environment Politics

After Michael Mann and his colleagues (MBH) published their first "hockey stick" paper in 1998[1] and its follow up in 1999[2], scientists had compelling evidence that recent warming, especially warming following 1950 or so, was unique in recent centuries. Geological evidence revealed that recent increases global and NH temperatures were detectable at rates beyond the natural variability observed over the last several hundred years. The resulting "hockey stick" graph was used in the IPCC's third assessment report, and the contrast between this graph and the previous schematic in the 1990 report was visually evident. The 1990 schematic was not a true reconstruction of global or even hemispheric temperatures; it was indistinguishable from a reconstruction Central England temperatures published in Lamb 1965. The 1990 graph was only a schematic with no temperature scale, and it ended in ~1950. The MBH "hockey stick" reconstruction was the first time the IPCC had access to a reconstruction that was based on a multiproxy database capable of reconstructing hemispheric temperatures, and it showed that recent warming was exceptional over the past several centuries.

Red line is IPCC-1990 fig. 7.1 with inferred temp scale. Blue line is MBH98/IPCC-2001.
Black line is Moberg 2005. Dashed line is the instrumental record.

The MBH98/99 reconstructions were not perfect, but the basic shape of the reconstructions have been replicated numerous times with better methods and more proxies with greater resolution. Literally nothing about climate science today depends on the validity of MBH98/99. However, the findings of this paper (and every global and hemispheric reconstruction of temperatures following MBH98/99) provides compelling evidence that recent warming exceeds natural variability. If correct, the case for attribution of recent warming to human activity is much more compelling. Consequently, attacks on MBH98/99 have been severe and malicious, to state it mildly. In the seven years following its publication, criticisms of this paper escalated to what can only be described as a political witch hunt, with the authors of this paper being called to testify before congress, not only to their findings but to their personal information, funding, etc.

I figured it would be helpful to produce a chronological accounting of how this progressed in the literature, because these developments detail the lengths to which politicians have attempted to insert themselves into science and control it from the outside. The chronology is too long to cover in a single post. I'll begin with what I consider the initial criticism in 2003 that appears to be the catalyst that lead ultimately to the political witch hunt against MBH. Let me be clear. At this point in 2003, I don't think there was any intention on the part of Stephen McIntyre and Ross McKitrick to initiate such a response. But a likely unintended consequences was precisely that. The tl;dr for this series: while MBH98/99 can be and has been improved, the "hockey stick" shape to the last 2000 years of global temperatures has not been disproved; it has been vindicated by multiple replication studies, and continued objections to the "hockey stick" appear to be motivated by concerns that have little if anything to do with the proxy data and proper statistical analysis of it.

Publication of MM03 in Energy & Environment

In 2003,[3] Stephen McIntyre and Ross McKitrick published a criticism of MBH98 claiming to be an audit of MBH98 that uncovered mistakes in MBH's handling of data from their proxies that, when corrected, allows them to conclude that "values in the early 15th century exceed any values in the 20th century," thus undermining the "hockey stick" shape to the MBH98 reconstruction and any claim that recent warming is unique. In effect, they accused MBH98 of employing such poor quality control that their proxy data set contained numerous "collation errors, unjustifiable truncation or extrapolation of source data, obsolete data, geographical location errors" and incorrect calculations. They claimed that correcting for these errors allowed them to reconstruct a temperature history of the NH that lacks any hockey stick shape. Their "corrected version" looks like this.

They conclude, "Without endorsing the MBH98 methodology or choice of source data, we were able to apply the MBH98 methodology to a database with improved quality control and found that their own method, carefully applied to their own intended source data, yielded a Northern Hemisphere temperature index in which the late 20th century is unexceptional compared to the preceding centuries, displaying neither unusually high mean values nor variability."

This paper was published in the journal Energy & Environment, which was known to lack any robust peer-review. Articles were published in this journal because of their perceived political value rather than for their scientific merit. The editor for Energy & Environment, Sonja Boehmer-Christiansen, admitted that this was the case with MM03. She wrote to Tim Osborn of CRU that she rushed the paper to publication for political purposes and so dispensed with robust peer review. She is quoted as saying, "I’m following my political agenda – a bit, anyway. But isn’t that the right of an editor?" She was also quoted saying, "I was rushing you to get this paper out for policy impact reasons, e.g. publication well before COP9" and that the "paper was amended until the very last moment. There was a trade off in favour of policy." This by itself does not mean we should ignore the findings of this paper, but it does mean that we should not assume that this paper received any critical scientific scrutiny, let alone robust peer review, before publication. Energy & Environment published it for its perceived political benefits, not for its scientific merit.

The political motivation and rush to publication at least on the part of Energy & Environment may explain why customary academic courtesies were not followed in the publication of MM03. In a critical evaluation of other scientists' work, it is customary for the authors of the work being criticized to be given a prepublication manuscript for comment or for the publishing journal to ask the authors of the original work to be among the referees for the paper. But it is clear from later correspondence that neither of these things happened. Mann and Rutherford were contacted for data and information while M&M were preparing their paper, and Mann was asked to explain some things they didn't understand in the data they received (and M&M were not happy about Mann's responses). However, none of MBH were allowed to see the prepublication manuscript for comment and review. This simple protocol would have improved the paper and avoided much of the controversy surrounding MM03. Many of its mistakes they made could have been discovered prior to publication.

Critical Response to MM03

Mann's initial response to MM03 was summarized on David Appell's website. As best I can tell, these pages are now offline, and they cannot be found. However, the text from the website have been preserved by M&M along with a running log of the correspondence with commentary between M&M and Mann and Rutherford. The sheer volume of material written about this is too much for me to cover in one blogpost. I'm going to limit myself to statements by the major players in the resulting controversy that were written in the wake of the publication of MM03 - that is, statements and publications by M&M, MBH, Rutherford and Osborn. Much of this has been preserved on the ClimateAudit website, and I will quote from there frequently because many of these statements are now difficult to find and no longer exist at their original links.

The major critiques from MBH can be summarized in two parts, summarized in an AMS paper published in 2005 by Rutherford et al[5], which confirms that "reported putative 'errors'" in MM03 are actually "an artifact of (a) the use by these latter authors of an incorrect version of the Mann et al. (1998) proxy indicator dataset and (b) their apparent misunderstanding of the methodology used by Mann et al. (1998) to calculate PC series of proxy networks over progressively longer time intervals." In other words, MM03 applied the MBH98 methodology incorrectly to an incorrect version of the proxy dataset used in MBH98. I'll cover each of these in turn.

Incorrect Version of MBH98 Dataset

In one of his initial responses to MM03 on David Appell's website, Mann claimed that M&M had made mistakes in preparing their spreadsheet file from what they were given. And M&M's record of correspondence with Mann and Rutherford appears to confirm this. Here is a summary of Mann's response:

M&M asked an associate of Mann to supply them with the Mann et. al. proxy data in an Excel spreadsheet, even though the raw data is available here. An error was made in preparing this Excel file, in which the early series were successively overprinted by later and later series, and this is the data M&M used…. The spreadsheet file they used was a complete distortion of the actual Mann et. al. proxy data set, and was essentially useless, particularly in the earlier centuries.

In response, M&M point out among other things that "we had never asked for an Excel spreadsheet." And that's true; they didn't ask for or receive an Excel file. According to the log they preserved, they received a txt file from which they would have to prepare a spreadsheet or database table. M&M also expressed frustration with Mann and Rutherford not providing timely communication or helpful responses to their questions. This is impossible for me to evaluate, since we only have M&M's selective summary of their communication. But at the very least some of the problem clearly comes from M&M's failure to accurately evaluate the data and information they were given. The preserved correspondence shows M&M asking Mann for information that was already answered in the supplemental material to his paper. Mann also pointed out that others were able to use the publicly available data and implement Mann's methodology (he mentions Zorita et al).[4] In Mann's email, it's clear he believed that he and Rutherford had provided M&M with everything they needed and everything that others had used to evaluate his methodology. But the resulting data table that M&M used was wrong; the data table M&M used had only 112 columns, and it needed 159 columns to serve as a proper "audit" of MBH98:

Mann says that the crux of M&M's error is their use of a Excel dataset with only 112 columns (where each column represents one set of proxy data--tree rings, ice cores, historical temperature data, etc.), when in fact the full paleoclimatic data series requires 159 to be used properly in the analysis behind the Mann, Bradley, and Hughes 1998 paper whose results they're trying to check.

The response by M&M to that last paragraph is telling:

This is something new altogether. Where has it ever been disclosed there were actually 159 proxies behind MBH98? Their Nature paper (see for example page 781, “based on the full multiproxy network of 112 indicators” etc.) and all supporting material only mention 112. The data file we were sent only contained 112 series. Scott Rutherford’s April 11 email refers to “Do you want the raw 300+ proxies or the 112 that were used in the MBH98 reconstruction?” The file we sent back to Mann only contained 112. In the Zorita paper Mann refers to above they only refer to 112 proxies too (p. 1379.) It seems odd that 5 years after the publication of the paper it is now revealed for the first time that there were 159 proxies, not 112.

This is the text that ends M&M's log of correspondence with their commentary. But note what has happened here. Mann's criticism was that the spreadsheet they used contained only 112 columns, but the data required a 159 column spreadsheet or database table. M&M read "columns" as "proxies" and inferred that Mann was saying there were actually 159 proxies instead of 112. As we'll see later, this is a clear indicator both that M&M did not construct their dataset properly and that they did not understand the MBH98 methodology they were trying to replicate. The dataset MBH98 used contained 112 proxies but their data table required 159 columns with some proxies using more than one column (we'll see later why this is the case). M&M prepared their table with only 112 columns. M&M prepared a table of data from the txt file they were given, and in the process of doing that, they apparently wrongly assumed the table should have 112 columns with one column per proxy.

Misunderstanding the MBH98 Methodology

Shortly after the initial response from Mann, MBH composed a more substantial if still preliminary response with a more severe criticism of MM03. MBH noted that because of M&M's misunderstanding of the methodology used in MBH98, MM03 effectively eliminated several proxies from MM03's "corrected" reconstruction, especially between 1400 and 1500. The elimination of this critical proxy data from the first 100 years explained the spurious warming in MM03 during that time frame. MBH calculated that the 1400-1500 portion of MM03's reconstruction would have an RE score of -6.6, which "is far worse than even a typical random estimate, and such a result would have been discarded as unreliable based on the cross-validation protocol used by MBH98."

By truncating proxies from 1400-1500, MBH were able to replicate the spurious
warming produced by MM03

This also explained why it was so important for MM03 to use a data table with 159 columns. MBH98 employed a stepwise reconstruction with separate columns for sub-intervals of the same proxy. They used intervals for 1400-1980, 1450-1980, 1500-1980, and so on. This means they needed "159 independent time series to represent all indicators required for reconstructions of all possible subintervals, even though the maximum number ever used for a particular sub-interval is 112." MM03 clearly did not follow this procedure, and so by truncating their data table to 112 columns, it was simply impossible for MM03 to audit the MBH98 methodology as they claimed they had. And the consequences for the first 200 years of the reconstruction were severe. MBH estimated that M&M "appear to have eliminated in the range of 100 proxy series used by MBH98 over the interval 1400-1600." In Rutherford et al, they further estimated that by not following MBH98's methodology, M&M eliminated about 70% of the proxy data used by MBH98 prior to 1600, so their analysis "gave rise to spurious warmth during the fifteenth century in their reconstruction, sharply at odds with virtually all other empirical and model-based estimates of hemispheric temperature trends in past centuries." 

Rutherford et al also evaluated the impact of varying statistical methodologies on temperature reconstruction to see how sensitive reconstructions were to the statistical methodology. They concluded that

These evaluations suggest that differing methods of reconstruction (e.g., different CFR techniques or local calibration approaches) yield nearly indistinguishable results if differences in underlying proxy network, target season, and target region are controlled for. We conclude that proxy-based temperature reconstructions are robust with respect to a wide array of alternative statistical approaches.

These criticisms of MM03 appear to be sound, and they are confirmed in many ways by what M&M wrote and from the correspondence with Mann and Rutherford that they preserved (with their commentary). The errors that Rutherford et al describe are also mentioned in the Wegman Report that I'll explore in a future post. If there are errors in MBH98, MM03 did not qualify as an "audit" that was capable of finding them.

Rushed Criticisms and Responses 

One of the common threads running through the controversy over MM03, at least to me, is that a rush to judgment is almost always a bad thing, especially when it comes to honest evaluations and critiques of complex scientific issues. It's clear that Energy & Environment saw the political advantages of publishing MM03 ahead of COP9, and took shortcuts to publication that prioritized politics over accurate science. If the publishers had valued academic integrity, they would have included MBH as referees of MM03, and much of this would have been avoided. 

But it may well be that Mann's initial responses were also hasty and reactionary. Tim Osborn observed that while MM03 was likely rushed to publication and contained considerable errors, Mann should not rush out a response without doing a thorough analysis of M&M's paper. In doing so, Mann could commit the same errors committed by M&M (and Energy & Environment) and muddy the waters even further. In one statement, Osborn wrote that while the MM03 "audit" was able to replicate "clearly anomalous recent warming," they then made "many modifications to the proxy data set and repeat their analysis, and obtain a rather different result to MBH98." Osborn continues:

Unfortunately neither M&M nor the journal in which it was published took the necessary step of investigating whether the difference between their results and MBH98 could be explained simply by some error or set of errors in their use of the data or in their implementation of the MBH98 method. This should have been an essential step to take in a case such as this where the difference in results is so large and important. Simple errors must first be ruled out prior to publication. Even if the authors had not undertaken this by presenting their results to the authors of MBH98, the journal should certainly have included them as referees of the manuscript.

Osborn acknowledged that "a number of likely errors" had already been identified, but "rather than repeating M&M’s failure to follow good scientific practise," he wanted to withhold a final verdict on the MM03 paper until he could be certain of "what changes to data and method were made by M&M" and the effect of these changes on the reconstruction. He also advised Mann to do likewise, so that his response could be measured and accurate. "I really advise a very careful reading of M&M and their supplementary website to ensure that everything in the response is clearly correct." Osborn then suggests that some of Mann's initial responses were not as careful as they should have been.

MM04 Comment

In 2004, M&M attempted to publish a comment to MBH98 critical of the reconstruction and asserting that the MBH98 methodology would still produce warming during 1400-1500 comparable to recent warming. And to be clear, M&M were not asserting that this warming is real, only that the warming would be produced by MBH98 if their methodology was correctly applied. However, M&M did not condone their method and would later argue against it in 2005. MBH wrote a reply to this comment, and the MM04 comment was rejected by Nature and was never published in Nature (though it can still be found on Researchgate).[6] Accordingly, there was no need for the MBH reply to be published, but it is also available online.[7] The MBH reply points out that the MM04 comment made similar errors to the MM03 paper. By censoring proxies at various times in the dataset, MM04 produced spurious results that failed statistical tests of skill; MM04 had negative RE scores where it differed from MBH98.

However, MBH did acknowledge that M&M had discovered some mistakes made by MBH98 in their accounting of the proxy data used their study.[8] While these corrections did not affect the final results of MBH98, we can acknowledge the value of an audit that improves the quality of the final paper. I'll have more to say about this in Part 2.

Conclusion

I consider the MM03 paper and the MM04 comment to be a total loss. If there are any problems with MBH98, this analysis is incapable of discovering them, and the "hockey stick" emerges from proxy data with a wide array of statistical approaches. While Mann's main two criticisms of MM03 appear to be essentially correct and on point, perhaps the attitude he conveyed was not. In some of the CRU emails, Mann expressed his opinion that MM03 should be dismissed as a stunt. He writes, "The important thing is to deny that this has any intellectual credibility whatsoever and, if contacted by any media, to dismiss this for the stunt that it is." These of course are personal emails, and people vent. But the public response to MM03 should be focused on the merits of the paper accurately read and evaluated, rather than personal accusations.

In future posts, I'll consider further objections to MBH98/99 including M&M's follow-up papers published in 2005 and the infamous Wegman Report. Just to give a heads up for future posts, I'm not going to argue that MBH98/99 contain no flaws. Rather, it seems clear that none of the flaws that were found significantly alter the "hockey stick" shape of the last 2000 years of global climate history, and this shape has been replicated by multiple later studies, most notably the Pages 2K reconstruction. 

Part 2 of this series.

Part 3 of this series.

References:

[1] Mann, M., Bradley, R. & Hughes, M. Global-scale temperature patterns and climate forcing over the past six centuries. Nature 392, 779–787 (1998). https://doi.org/10.1038/33859

[2] Mann, M. E., Bradley, R. S., & Hughes, M. K. (1999). Northern hemisphere temperatures during the past millennium: Inferences, uncertainties, and limitations. Geophysical Research Letters, 26(6), 759–762. doi:10.1029/1999gl900070

[3] McIntyre S., McKitrick R., 2003. “Corrections to the Mann et. al. (1998) Proxy Data Base and Northern Hemispheric Average Temperature Series” Energy and Environment Vol. 14, pp. 751–771.

https://doi.org/10.1260/095830503322793632

[4] Zorita, E., F. Gonzalez-Rouco, and S. Legutke, Testing the Mann et al. (1998) approach to paleoclimate reconstructions in the context of a 1000-yr control simulation with the ECHO-G Coupled Climate Model, J. Climate, 16, 1378-1390, 2003

[5] Rutherford, S., M. E. Mann, T. J. Osborn, K. R. Briffa, P. Jones, R. S. Bradley, and M. K. Hughes, 2005: Proxy-Based Northern Hemisphere Surface Temperature Reconstructions: Sensitivity to Method, Predictor Network, Target Season, and Target Domain. J. Climate, 18, 2308–2329, https://doi.org/10.1175/JCLI3351.1.

[6] McIntyre S., McKitrick R., 2004. Global-scale temperature patterns and climate forcings over the past six centuries: A comment.

https://www.researchgate.net/publication/240021986_Global-scale_temperature_patterns_and_climate_forcings_over_the_past_six_centuries_A_comment

[7] Michael E. Mann, Raymond S. Bradley, Malcolm K. Hughes. REPLY TO "Global-scale temperature patterns and climate forcings over the past six centuries: A comment." By S. McIntyre and  R. McKitrick.

https://stephenschneider.stanford.edu/Publications/PDF_Papers/MannEtAl2004.pdf

[8] Mann, M., Bradley, R. & Hughes, M. Global-scale temperature patterns and climate forcing over the past six centuries: Corrigendum. Nature 392,779–787(1998).
http://www.meteo.psu.edu/holocene/public_html/shared/articles/MBH98-corrigendum04.pdf

[9] McIntyre, S., & McKitrick, R. (2005). The M&M Critique of the MBH98 Northern Hemisphere Climate Index: Update and Implications. Energy & Environment, 16(1), 69–100. https://doi.org/10.1260/0958305053516226