Correcting for Time of Observation Bias

You'll frequently see contrarian influencers on social media showing the differences between "raw" and "adjusted" temperatures for the United States that indicate that CONUS warming in the "adjusted" temperatures is greater than in the "raw" data. We're often told this indicates that scientists have adjusted CONUS temperatures to make them cooler in the past, thus making the amount of warming that has occurred in the US larger in the "adjusted" temperature data than in the "raw" data. It's then frequently just assumed that this is because "liberal" scientists need are conspiring with nefarious intent to tamper with and manipulate temperature data to create artificial warming trends in US temperatures. In another post I share some dishonest way in which contrarians exaggerate the difference between the "raw" and adjusted temperatures, but even in properly plotted comparisons, the final "adjusted" temperature data published by NOAA shows more warming than the "raw" data.

Graph from Jim Java on X

What's actually happening is somewhat complicated, but the adjustments are absolutely necessary if we want to accurately represent how CONUS temperatures are changing. All surface thermometers collect data that are subject to biases, and in all fields of science "bias correction" is used to remove systematic errors that can artificially influence what the data says. If you have a tape measure you use to measure heights of your children every month, and one month you notice all your children grew by over two inches, you might check your tape measure. Looking at your tape measure you find that someone cut exactly 2 inches off the end of your tape measure. So you realize that your measurements were off by 2 inches, and since you can quantify that systematic error, you can also remove it by subtracting 2 inches from the measurements of your children's heights. Here the difference between the "raw" and "adjusted" measurement is the result of honest bias correction with an interest in being truthful about the heights of your children, not data tampering or manipulation. The tl;dr here is that this is precisely what "adjustments" to the US temperature record do as well.

The most significant bias affecting CONUS temperatures has to do with changes in the time of observation (TOBs).  Historically, weather stations used max-min thermometers that would be manually read once every 24 hours and the data recorded. The thermometer would register both a maximum temperature (Tmax) and the minimum temperature (Tmin), and when the observation was made, the Tmax and Tmin would be recorded, and thermometer would be reset. For climate purposes, the observation time doesn't matter that much, as long as the observation time remained at roughly the same time of day. Consequently, up until the 1950s, the observation times for most stations took place in the afternoon. However, weather stations also recorded precipitation, and the US Weather Service wanted to standardize when precipitation measurements were taken to minimize evaporation loss, so they systematically changed the observation times at weather stations to standardize morning observations. Between the 1950s and 2000, therefore, there was a gradual and systematic shift of observation time from the afternoon to the morning.

Changes in TOBs

It's this change in observation time that introduces bias to the temperature record. To illustate, let's consider the effect of changing an observation time of a max-min thermometer from 5 pm to 6 am. 

1. With TOBs at 5 pm, let's say we record a Tmax of 98°F and reset the thermometer. At 5:20 pm, the temperature is still a blistering 96°F, but then a storm comes though, temperatures cool, and the following day is more temperate, with a high at 3 pm of just 84°F. Because the observation time was 5 pm, the Tmax for Day 1 (98°F) and Day 2 (96°F) actually occurs on Day 1, and the weather station has essentially double-counted a hot day. 
2. If we change TOBs to 6 am, we essentially eliminate the possibility of double counting hot days, but we introduced the possibility of double counting cool nights. For instance, let's say that at 6 am Tmin is recorded to be 34°F, and the thermometer is reset, but then it remains cold such that the temperature at 6:15 am is still just 36°F. Then the day warms and the next night only cools to 50°F. In this case, the Tmin for Morning 1 (34°F) and Morning  2 (36°F) occurred on Morning 1, and the weather station double-counted a cold night.

Double counting either hot days or cold nights doesn't add bias to temperature trends. What adds bias is the change of observation time, which systematically changes the frequency of double counting either hot days or cold nights. This bias was described and a methodology was created to correct for this bias in 1986.[1] Since the systematic error introduced by changing observation times can be quantified, it can can also be effectively removed. Bias correction effectively removes the systematic error and replaces it with a little bit of random error,[2] and random error is reduced as sample sizes increase. As a result, bias corrected ("adjusted") temperature data does a much better job of representing CONUS temperature changes.

USCRN Temperatures With (Red) and Without (Blue) a Change in TOBs

In fact, this bias can be synthesized and replicated with weather station networks that record temperatures more frequently than 24 hours. Using the USCRN stations, which record hourly temperatures, Zeke Hausfather synthesized the effect of changing observation time from PM to AM and found that his synthesized bias agrees with what had been described in the peer-reviewed literature.

Comparing Berkeley to NOAA adjustments to US Temperatures

Interestingly, since changes in TOBs occur at the station level, the change causes affected thermometers readings to "stick out" relative to surrounding stations, and so the bias can actually be corrected by homogenization without the procedure designed by Karl et al 1986.[1] The Berkeley Earth project relies on homogenization to remove the bias, and their temperature record agrees with NOAA's.[5]

It's indisputable that this correction to a systematic error increases the amount of warming that has occurred on land in the US with respect to the raw data. But because the systematic error affecting raw temperature data is so well-quantified and the correction procedures so effective, it's also indisputable that the bias-corrected ("adjusted") CONUS temperature data published by NOAA and Berkeley (among others) more accurately represent the changes in CONUS temperatures that actually occurred. Claiming that we should continue to use raw temperature data with this systematic error instead of the bias-corrected temperatures without this systematic error is exactly like saying that your children actually grew by over 2 inches in a month because someone cut 2 inches off the end of your measuring tape. It's downright silly.

For more detailed information, see a very helpful post from Zeke Hausfather, from which I got some the graphs above.

References:

[1] Karl et al, “A Model to Estimate the Time of Observation Bias Associated with Monthly Mean Maximum, Minimum and Mean Temperatures for the United States,” Journal of Applied Meteorology 25.2(1986):145-160. https://doi.org/10.1175/1520-0450(1986)025<0145:AMTETT>2.0.CO;2

https://www1.ncdc.noaa.gov/pub/data/ushcn/papers/karl-etal1986.pdf

[2] "Homogenization is the process of identification and removal of artifacts in station records such as those caused by changes in measurement equipment, relocation of stations within their local area, changes in time of day of measurements, and changes in methods used to compute monthly mean temperatures. Homogenization adjustments have been applied to the land station data included in HadCRUT4 [Jones et al., 2012]. Brohan et al. [2006] compared adjusted time series in the CRU archive to unadjusted records where unadjusted records were available. Through this comparison it was concluded that small discontinuities in station records were difficult to detect in the homogenization process and that a residual error in the homogenization process exists. This error was modeled as a zero mean Gaussian distribution with a standard deviation of of σh = 0.4°C. Recent studies of homogenization uncertainty report broadly similar magnitudes of homogenization uncertainty and so the model of σh = 0.4°C is maintained in CRUTEM4 and HadCRUT4."

Morice, C. P., J. J. Kennedy, N. A. Rayner, and P. D. Jones (2012), Quantifying uncertainties in global and regional temperature change using an ensemble of observational estimates: The HadCRUT4 data set, J. Geophys. Res., 117, D08101, doi:10.1029/2011JD017187.

[3] Vose et al, “An evaluation of the time of observation bias adjustment in the U.S. Historical Climatology Network,” Gheophysical Research Letters 30.20 (2003). https://www1.ncdc.noaa.gov/pub/data/ushcn/papers/vose-etal2003.pdf

[4] Menne, M.J. and C. N. Williams Jr. “Homogenization of Temperature Series via Pairwise Comparisons.” Journal of Climate 22 (2009): 1700-1717. http://dx.doi.org/10.1175/2008JCLI2263.1

[5] Williams, C. N., M. J. Menne, and P. W. Thorne (2012), Benchmarking the performance of pairwise homogenization of surface temperatures in the United States, J. Geophys. Res., 117, D05116, doi:10.1029/2011JD016761. https://www.ncei.noaa.gov/pub/data/ushcn/papers/williams-etal2012.pdf