Is Patrick Brown Right that the Most Prestigious Journals are Biased?
Patrick Brown recently published a blogpost in which he suggested that in order to get published in highly prestigious journals, you have to follow a "not-so secret" formula. Another version of this shows up in the Free Press. According to Brown, "the formula is more about shaping your research in specific ways to support pre-approved narratives than it is about generating useful knowledge for society." Now I want to be clear about a few things before I get into this. First, it appears he's not attacking peer-reviewed journals in general or even highly respected journals a tier below the top echelon of Nature and Science. It also doesn't appear that he thinks these journals are forcing a narrative that isn't true; only that they are preferentially selecting the negative aspects of AGW over others factors, and science would benefit if these journals published more useful and comprehensive analyses. And he also seems to acknowledge that the kinds of more useful papers that he wants to publish can be published in other high impact journals. So for him, it's not that useful papers don't get published; it's that they don't get published in the two most prestigious journals, but those are the two journals that advance the careers of scientists the most, and that motivates scientists to publish on selective aspects of climate change to get published in the top two journals.
Second, I've never published in any of these journals. Brown certainly has more knowledge about how these journals work than I do, both in terms of publishing and in terms of participating in peer-review. I want to make sure that I don't presume to understand his field better than he does. All I can do is evaluate what he says. I think it's pretty clear that, if he's right about this publication bias in the two most prestigious journals, he didn't make a valid case that this bias exists and distorts the overall picture of climate science we have. I have several criticisms of what he wrote that I think don't depend on having more expertise than he does, but I think they do show that he hasn't made his case.
Motivations to Publish in Science and Nature
Brown says that since Nature and Science are so prestigious and the impacts of publishing in these on the careers of researchers are so profound, researchers will tailor their studies to maximize their chances of getting published in these journals. It at least makes sense to me that researchers motivated to achieve this kind of prestige would do this. And given that journal editors are people with expectations, those papers matching the expectations of the journals are probably more likely to be published. But he says,
My overarching advice for getting climate change impacts research published in a high-profile journal is to make sure that it supports the mainstream narrative that climate change impacts are pervasive and catastrophic, and the primary way to deal with them is not through practical adaptation measures but through policies that reduce greenhouse gas emissions.
Now if I had the opportunity to talk to Brown personally about this, I'd have lots of questions. Why would prestigious journals prioritize publishing the same thing that's already been published over and over again over new and innovative findings? What brings more prestige to a journal and to the researcher publishing in that journal - the paper that says, 1) here's more evidence for what we already knew or 2) here's new evidence that gives us new insights that we didn't have before? My sense given current concern for too few replication papers in scientific literature is it's much more prestigious for a journal to publish innovative research that adds new insights to current understanding or even challenges it, provided that the research is sound.
Brown seems to think that papers published in Nature or Science "are required to be short, with only a few graphics, and thus there is little room for discussion of complicating factors or contradictory evidence." That may be true, I suppose, but that doesn't mean that innovative discoveries that can be written up as short papers with a few graphics wouldn't get published. And I also question whether scientists must publish in these two journals to have a large impact in climate science. I can think if numerous papers published in "lower" tiered journals that have had a large impact on climate science, even featuring prominently in IPCC reports. Here is a list of some papers not published in Nature or Science but nevertheless have had a very significant impact on climate science.
Manabe, S., and R. T. Wetherald, 1967: Thermal Equilibrium of the Atmosphere with a Given Distribution of Relative Humidity. J. Atmos. Sci., 24, 241–259, https://doi.org/10.1175/1520-0469(1967)024<0241:TEOTAW>2.0.CO;2.
Mann ME, Bradley RS, & Hughes MK: Northern hemisphere temperatures during the past millennium: Inferences, uncertainties, and limitations. Geophys Res Lett 26(6):759–762, 1999. dx.doi.org/10.1029/1999GL900070
Sherwood, S. C., Webb, M. J., Annan, J. D., Armour, K. C., Forster, P. M., Hargreaves, J. C., et al. (2020). An assessment of Earth's climate sensitivity using multiple lines of evidence. Reviews of Geophysics, 58, e2019RG000678. https://doi.org/10.1029/2019RG000678
Erb, M. P., McKay, N. P., Steiger, N., Dee, S., Hancock, C., Ivanovic, R. F., Gregoire, L. J., and Valdes, P.: Reconstructing Holocene temperatures in time and space using paleoclimate data assimilation, Clim. Past, 18, 2599–2629, https://doi.org/10.5194/cp-18-2599-2022, 2022.
 |
| Erb et al 2022 |
And while I recognize that gaining prestige and advancing your career is a big motivator to publish in the two top journals, I question whether scientists should be publishing for those reasons. It would seem to me that researchers looking to publish research with comprehensive analysis of various features of the climate system still have plenty of options available to them that can still advance our scientific understanding of the climate system in profound ways, and if Nature and Science specialize in shorter papers, then maybe just try for publication those journals when your analysis fits the selectivity of those journals.
Brown's Formula for Publication in Science and Nature
Brown seems to think that in order to get published in Nature or Science, you have to "support the main stream narrative" and focus on mitigation policies rather than practical adaptation measures, and he gives four criteria that he believes are boxes to be checked to get published. These are:
- Show that climate change impacts something of value, without necessarily quantifying whether this impact is large compared to other influences
- Ignore or downplay near-term practical actions that could negate the impact of climate change.
- Focus on metrics that are not the most illuminating or relevant but generate impressive numbers.
- Clean, concise format of presentation (short papers with few graphics) that don't permit room for complicating factors in the analysis.
If this is the case, it should be hard for me to find papers published in Nature or Science that highlight other factors besides AGW that have large impacts on features of the climate system (or other other things of value). I shouldn't be able to find papers focusing on the importance of near-term adaptation strategies. So I did a quick search of papers in Nature or Science that might fit those criteria. I already knew of one published in Nature, and it took less than a minute to find another from Science. I've also used a third from PNAS before to show that scientists agree that human ignitions are significant to wildfires in the past. While PNAS is not one the two journals Brown is attacking, in my view it's just as prestigious. So I was able to quickly find 2 or 3 papers that got published in the most prestigious journals without checking the Brown's required boxes. Here's a list with excerpts from the abstracts so that you can see they don't check boxes Brown thinks should be checked:
"Globally, wildfire size, severity, and frequency have been increasing, as have related fatalities and taxpayer-funded firefighting costs . In most accessible forests, wildfire response prioritizes suppression because fires are easier and cheaper to contain when small. In the United States, for example, 98% of wildfires are suppressed before reaching 120 ha in size . But the 2% of wildfires that escape containment often burn under extreme weather conditions in fuel-loaded forests and account for 97% of fire-fighting costs and total area burned . Changing climate and decades of fuel accumulation make efforts to suppress every fire dangerous, expensive, and ill advised."
North, M. P., Stephens, S. L., Collins, B. M., Agee, J. K., Aplet, G., Franklin, J. F., & Fulé, P. Z. (2015). Reform forest fire management. Science, 349(6254), 1280-1281.
"We observe that human-ignited fires start at locations with lower tree cover and during periods with more extreme fire weather. These characteristics contribute to more explosive growth in the first few days following ignition for human-caused fires as compared to lightning-caused fires. The faster fire spread, in turn, yields a larger ecosystem impact, with tree mortality more than three times higher for fast-moving fires (>1 km day−1) than for slow moving fires (<0.5 km day−1). Our analysis shows how human-caused fires can amplify ecosystem impacts and highlights the importance of limiting human-caused fires during period of extreme fire weather for meeting forest conservation targets under scenarios of future change."
Hantson S, Andela N, Goulden ML, Randerson JT. Human-ignited fires result in more extreme fire behavior and ecosystem impacts. Nat Commun. 2022 May 17;13(1):2717. doi: 10.1038/s41467-022-30030-2. PMID: 35581218; PMCID: PMC9114381.
"Human-started wildfires accounted for 84% of all wildfires, tripled the length of the fire season, dominated an area seven times greater than that affected by lightning fires, and were responsible for nearly half of all area burned. National and regional policy efforts to mitigate wildfire-related hazards would benefit from focusing on reducing the human expansion of the fire niche."
Balch, J. K., Bradley, B. A., Abatzoglou, J. T., Nagy, R. C., Fusco, E. J., & Mahood, A. L. (2017). Human-started wildfires expand the fire niche across the United States. Proceedings of the National Academy of Sciences, 114(11), 2946–2951.
https://www.pnas.org/doi/10.1073/pnas.1617394114
In fact, in the Free Press, Brown's title and subtitle were, "I Left Out the Full Truth to Get My Climate Change Paper Published: I just got published in Nature because I stuck to a narrative I knew the editors would like. That’s not the way science should work." Perhaps a caveat is important here; no paper of finite length can cover the "full truth" of anything. I assume he means here the full amount of evidence that ought to have been included in the paper. If that's what he meant, then I agree that's not the way science should work, but it does appear to be how Patrick Brown worked, and it also appears that others that don't work that way get published in Nature and Science. So maybe the problem is more in his perceptions about these journals and not with the journals themselves?
I think in order for Brown to make his case, he needs to do more than he's done. His entire post appears to be largely anecdotal, dependent on his perceptions of his experiences with these journals - what he got published where, and what papers got rejected in the two most prestigious journals and accepted elsewhere. But as Feynman has famously stated, "The first principle is that you must not fool yourself and you are the easiest person to fool." How did Brown rule out the possibility that he has fooled himself? We don't know.
To be fair, Brown has qualified himself by saying, "My point is not that it is impossible to publish high-impact papers that deviate from the formula I describe but that it is much harder." So he's acknowledging it's still possible to publish in these two journals if you don't check all the boxes. And also in fairness, he's saying that you only need to cover the impact of AGW on something in order to get published in these two journals, not that you can't also cover other impacts. But some of the papers I cited have as their main subject these other impacts. But even with these qualifications, how did he determine that it's "harder" to publish? We don't know.
Brown's case would be better made if he did a systematic survey of wildfire papers published in Nature and Science and estimated the percentage of these papers that check his boxes. Then it would have been good to look at papers published in the next level of high impact journals to see what fraction of those check his boxes. Then if he can get this information, determine how many of those papers published in the latter journals were rejected from Nature and Science for reasons having to do with the boxes he says needed to be checked. I don't have knowledge of the inner workings of these journals, but I think I can safely say that my perception of the papers published don't match his, and if he wants to make a case for his perceptions of these journals, he needs to do a more thorough analysis than what he presented. It took me just a few minutes to find papers that don't match his perceptions. There's a non-zero chance that those papers are nearly unique in these journals, I suppose, but the point is that he didn't show that he's done the kind of analysis to justify his conclusions.
Conclusion
I suspect Brown is right about much of what he says about the competitive nature of publishing research and the desire to meet the expectations of the two most prestigious journals. I'm sure that has an effect on the types of papers that get published in Nature and Science - shorter papers with a focused argument may be favored in these journals. Looking back over papers published in these journals do seem to reflect those. But I also strongly suspect he's wrong about how difficult it is to get papers published in those journals that don't check his boxes. I also suspect he's very wrong about the need to publish in these journals to make a significant impact on your scientific field. There are plenty of highly influential papers with extensive analysis that get published in other high quality journals.
Now to be clear, I don't want this to be a kind of hit piece on Brown. I'm sure a lot of people I generally agree with on AGW may turn on him for his post. I'm also sure that many contrarians will misuse this post as well to promote conspiracy theories about AGW and the peer-review system. I'm hoping to to be fair to his view, respect his experiences and expertise that I don't share, and simply evaluate how well he made his case. I like Brown's posts normally, and his blog is on my blogroll here. But I respectfully disagree with him on these posts; I don't think he made his case.
Update: Sept 8, 2023
Yesterday after posting this, I came across an article in E&E News covering this issue, and the coverage contains a response by Nature's chief editor, Magdalena Skipper. Her response is pretty significant. She told E&E News,
“The only thing in Patrick Brown’s statements about the editorial processes in scholarly journals that we agree on is that science should not work through the efforts by which he published this article,” Skipper said in a statement to E&E News. “We are now carefully considering the implications of his stated actions; certainly, they reflect poor research practices and are not in line with the standards we set for our journal.”
I can understand her point, though, since I don't have access to the full correspondence between Nature, the peer-reviewers, and Brown in the process leading up to publication, so I'm not going to comment on that. But to the extent that Brown deliberately withheld relevant content from his paper that was detrimental to his paper in order to make a point following publication about publication biases, I think Skipper has reasons to be concerned.
Reading between the lines of this quote, one wonders if this could turn into an investigation of the paper itself. And it's starting to be reminiscent of the Alimonti et al 2022 paper, which was just recently retracted. That blew up because the paper received attention in the media, prompting an investigative response by the Guardian and AFP. Springer read those articles and started and investigation. Here, Brown kind of did the same thing to himself by publicly admitting that he deliberately published what he considered an incomplete paper in order to get published. Personally, I hope the paper stands or falls on its own merits, not on what he said about it afterwards, but I can see why Nature might be concerned that Brown's public confession in these posts may reflect poor research practices during the editing process. Skipper also noted three papers published in Nature that don't check Brown's boxes. Here are the citations to those papers:
Fredston, A.L., Cheung, W.W.L., Frölicher, T.L. et al. Marine heatwaves are not a dominant driver of change in demersal fishes. Nature (2023). https://doi.org/10.1038/s41586-023-06449-y
Gatti, L.V., Cunha, C.L., Marani, L. et al. Increased Amazon carbon emissions mainly from decline in law enforcement. Nature (2023). https://doi.org/10.1038/s41586-023-06390-0
Hussam Mahmoud. The causes of wildfires are clear. How they burn through communities is not. Nature 620, 923 (2023) doi: https://doi.org/10.1038/d41586-023-02687-2
And this reminded me of another study evaluating the results of previously published studies evaluating the impact of ocean acidification on the behavior coral reef fishes. The previous work had observed negative impacts on fish behavior, but Clark's team attempted to replicate those findings and could not do so. Nature published this work, which has the effect of reducing the estimated impact of climate change on coral reef fishes (again, this doesn't check Brown's boxes).
Clark, T.D., Raby, G.D., Roche, D.G. et al. Ocean acidification does not impair the behaviour of coral reef fishes. Nature 577, 370–375 (2020). https://doi.org/10.1038/s41586-019-1903-y
I strongly suspect that if Brown did the kind of survey of papers published in Nature and Science I recommend above, he would not find that the evidence actually fits his perceptions. And it would seem that that his perceptions regarding publication biases led him to publish a paper that has the problems he believes exists in the literature. In effect, to some extent his paper exemplified the problem he wanted to expose.
%20Hurricanes%20and%20All%20Hurricanes.png)
What a sad story. Whoever convinced Brown that he had to compromise the integrity of his submission just to get published in Nature did him a grave disservice. In a long career in science (not climate, but closely related), I never heard of anyone being denied tenure or a promotion because they had not published in Nature or Science, or of anyone losing an award or some recognition to a competitor because the competitor published in Nature or Science and they didn’t. (I counseled younger colleagues *not* to submit to Nature or Science, because the rejection rate is so high. Read their “Information for authors”.)
ReplyDeleteBrown says “…the biases of the editors (and the reviewers they call upon to evaluate submissions) exert a major influence on the collective output of entire fields.” Yes, editors and reviewers are influential. Hundreds, if not thousands of them. And Brown seems to think they’re all biased. He says “…my papers were rejected out of hand by the editors of distinguished journals…” And Brown was so desperate to get published in a “distinguished” journal that he “…sacrificed contributing the most valuable knowledge for society…” (his words!)
This is such a sad story, not for reasons that Brown might think.
It is really sad, and I can imagine few people will want to collaborate with him on these kinds of projects. He comes off here as someone who cares most about getting published in these two journals, then confesses that he compromised his own ethics to do so.
Delete