Europe One Degree Warmer: How we got things wrong — and are working on fixing them
The European Data Journalism Network published an analysis of temperature trends in 558 cities and their surroundings in Europe on Monday, 24 September 2018, starting at 08:00 CEST, in partnership with several news outlets. A few hours after publication, several commenters pointed out that the temperature data for some cities in Sweden, which had been described in the analysis as the fastest-warming in Europe, were probably erroneous. Swedish blogger Göran Johnson, for instance, used the Swedish weather service’s data to point out the problems in our data for the city of Kiruna.
The mistake was identified on Tuesday, a day after publication. Data for 38 locations had been collected on a different machine from the rest and had not been properly checked. The methodology for all cities was to use the ERA-20C database for the period 1900–1978, the ERA-interim database for the period 1979–2017 and to correct the first time series using the second one (the code used for the reconciliation is available in a Jupyter Notebook). For the 38 locations with erroneous data, the reference data for the period 1979–1998 was ERA-20C’s (instead of ERA-interim’s) and no reconciliation had taken place. The result was that some locations were shown to be warming much more rapidly than they actually were, sometimes by more than two degrees Celsius. (The data for the 520 other locations was correct).
The graphic below shows the difference between the correct data and the erroneous one for the Swedish town of Kiruna and its surroundings, the place that was allegedly the fastest-warming according to our first release.
Unfortunately, no copy of the original weather files (called GRIB after their file format) were kept on our machines, so that we had to redownload the original data from ECMWF’s servers, which took close to 50 hours. We spent that time contacting news outlets outside our network that had written stories based on our data, to let them know that they could expect a correction.
The correct data was re-published on the onedegreewarmer.eu website on Friday, 28 September, slightly less than 70 hours after the incident was confirmed.
An apology and a lesson
We would like to apologise to all the media partners of EDJNet and their readers for the mistake in the data. The error was preventable, had we put in place more checks to ensure the integrity of the data.
The error was compounded by the fact that most of the erroneous data showed faster warming in sub-arctic regions, which was an outcome experts mentioned we could expect. A strong confirmation bias was at play. While we did check the results at the European level with climate experts, a talk with local (Swedish) experts would have allowed us to discover the error before publication.
Data errors happen from time to time. But it is completely unacceptable that errors of this magnitude are not discovered before publishing. Our review of what went wrong pointed to organisational and structural weaknesses within EDJNet. While each of the involved partners had strict internal routines for fact-checking and data validation, these routines were not automatically scaled up among partners. Having multiple partners sharing responsibilities with no overarching oversight allowed for a critical error to go through unattended.
Data for the affected locations now displays a disclaimer with a summary of the incident. All partner media outlets were informed of the incident and its resolution as we took steps to resolve it, which led to corrections and, in some instance, to retractations. We have also, to the best of our ability, tried to contact news organisations outside of EDJNet that published stories based on erroneous data.
The data and methodology is available in the following Gitlab Repository: gitlab.com/edjn/onedegwarmer-shared-data
Time series of yearly temperature data per location are also available in a more accessible format, in three spreadsheets:
We welcome all comments, suggestions and criticism regarding the data and the methodology.
The lessons learned from this error will help us improve and correct our internal processes, which includes introducing new coordinating roles, peer-review, and check-lists. We hope it will not affect the trust of our valuable readers and partners and that the many will be able to benefit from the mistakes of a few.
