Conflicting risk predictions presented black-box climate models

Enjoy free access to the best ideas and information – selected by our editors.

(Bloomberg) –Many companies offer detailed climate projections tailored precisely to the property level. These tools can allow property owners and investors to better anticipate the risks of flooding, wind, fire and other disasters. Insurers have been among the adopters of climate risk models, in a sign that these vulnerability predictions will inform decisions about whose premiums increase as the planet warms.

But these risk assessments take place in what they mean a bunch of black boxes. There is almost no way to compare or review the growing number of risk projections, which are usually protected from inspection as the intellectual property of private firms. Now there’s new evidence — as Bloomberg Green found in an exclusive analysis published in today’s Big Take — that risk models often disagree with each other on fundamental assessments of vulnerability.

In an analysis that used properties in Los Angeles County as a benchmark, there was wide disagreement between a private risk model that assesses future flood risk and an open-source alternative created by academics to measure the same. The two risk models assigned it similar vulnerability scores only about one in five properties in the Los Angeles area.

Other comparisons made by researchers faced the same lack of agreement between models. Analysts at the nonprofit CarbonPlan found substantial differences in data and projections between two private risk models, according to a report.published on Friday on the group’s website. An urban lot that one company labels as high risk may not be risky by another, raising questions about the reliability and transparency of tools that are now widely used in many sectors of the economy.

In December, CarbonPlan researchers asked nine climate analysis companies to participate in a “limited data assessment” by sharing information on 342 locations across the US. The researchers specifically requested small data samples from the millions that are potentially available to avoid burdening providers. Of the nine they approached, only two fulfilled the request: Jupiter Intelligence and XDI Pty Ltd.

Working with data from two firms was sufficient to allow comparisons that would not otherwise be available to anyone who did not purchase access to both tools. CarbonPlan found major variations in risk estimates, both in results from the mid-1990s and in projections to the end of this century. In one telling example, sample results from Jupiter and XDI agreed on only 12 percent of locations where fire risk will increase in the coming decades after comparing sample data for 128 California post offices.

The researchers also requested coastal flooding data for 90 New York schools, determined by the clear risk of sea ​​level rise in the region. Both models agreed that vulnerable properties are concentrated near the coast. But the share of locations where both Jupiter and XDI agree with increased flood risk is only 21%.

A third comparison used river flood and precipitation data for 124 post offices in New York State. Both companies agree on the broad overlap of locations facing low risk of river flooding by the end of the century. But among locations assessed with higher levels of risk, “there is only minimal consistency” between the two models, according to the CarbonPlan report.

Part of the variation comes down to Jupiter and XDI having different approaches and proprietary designs. CarbonPlan only looked at each firm’s risk scores, not the underlying methodologies. The purpose of the study was not to evaluate whose model is better, but only to see if or where the tools differ.

“What is perhaps most striking about this effort is how little data we have received” from most climate analysis companies, CarbonPlan noted in their report. First Street Technologies, Verisk Analytics, ClimateCheck, ZestyAI, Riskthinking.AI, Carbon4 Finance and Climate X either declined to participate or did not respond to the researchers.

In an interview before the release of the CarbonPlan report, XDI co-founder Karl Mallon said he expected the comparison of different risk models to prove useful. He said one advantage companies have over academic researchers is the ability to build a team that draws on diverse areas of expertise, such as climate science, building codes, insurance, engineering and hydrology.

“When we have really big differences, it kind of indicates that we’re doing different things,” Mallon said. Even though modelers agree on the key drivers of fire weather, such as temperature, humidity and wind, they can diverge if they include grass fires.

Josh Hacker, Jupiter’s co-founder and chief scientist, said it’s hard to comment on CarbonPlan’s results without seeing the full analysis. He generally disagreed with the idea that private climate models are black boxes. “When you’re in business, you have to protect some things,” Hacker said, noting that Jupiter’s team publishes peer-reviewed papers and presents at conferences when possible.

Hacker also noted that Jupiter is working with third-party experts to help validate its work and is careful to alert clients of the results, which Jupiter says include Norwegian oil company Equinor as well as insurance giants. Aon PLC and Zurich Insurance Group AG. “We communicate uncertainty,” he said. “We communicate where we have less confidence. We communicate where we have more confidence. But it’s against our business interest to communicate everything widely.”

Many of XDI’s clients, which the company says include the world’s largest money manager BlackRock Inc. and the banking giant HSBC Holdings Plcthey want to stay ahead of insurers’ decisions and that encourages XDI to think like an insurer. “In a way, part of our challenge is trying to think about how an actuary will think,” Mallon said. “And they’re pretty tight-lipped about how they do things.”

CarbonPlan’s mission is to promote transparency, and its new Risk Models report echoes that imperative. His findings highlighted that risk patterns can agree in a region, while diverging substantially at the property level. In other words, different risk models might determine that 5 percent of homes in a state are at high risk of disaster—just not the same 5 percent.

Another takeaway concerns how people, companies and governments respond to this kind of information. With climate adaptation a growing priority, “risk estimates from analytics companies are likely to affect billions of lives and trillions of dollars,” CarbonPlan warns in its findings. “If the climate risk industry continues in the private sector without any transparency requirements, black-box models that cannot be compared will become the norm in many industries, including housing and insurance.”

Climate science has progressed for decades comparing models on a global scale. In a sense, CarbonPlan is now calling for similar practices to benchmark private risk analyzes at a very local level.

“There’s a lot of room for there to be variation — even for there to be variation and for nobody to get it wrong,” said Oriana Chegwidden, a scientist at CarbonPlan. But the tighter the geographic focus, the more uncertainty in the prediction, she noted. “And we just don’t know how much there is.”

To contact the author of this story:
Eric Roston from New York at [email protected]