📰 Newspapers

LLMs will provide accurate climate info, except when they won’t

LLMs will provide accurate climate info, except when they won’t

Some models fare better, but all fail in similar ways.

Many studies over the years have examined how people encounter climate science information. Some focused on the top results that Google surfaced (back when Google was still in the business of serving links), while others examined social media content or newspaper and TV news coverage. With large language models (LLMs) now forming a share of the public information diet, it's worth looking at how they handle climate change.

But there’s no direct way for researchers to look at the information people have seen. It is served ephemerally to individual users, confined to their chat session. A study led by researchers from University College London and the United Nations Educational, Scientific, and Cultural Organization (UNESCO) tried to tackle this question from multiple angles, testing LLMs with different prompts and evaluating their output. They started with benchmarks of simple factual questions (benchmarks that might even be in models’ training data at this point) but also analyzed more complex interactions—including when users think they’re right to push back on an answer.

CO2 Q&A

LLM research often faces (no-true-Scotsman-tinged) criticism for focusing on a particular model rather than some other frontier model that may have been released after the experiment concluded. In this case, the study used 33 models, including multiple versions of the major frontier and open-weight models.

In brief

Many studies over the years have examined how people encounter climate science information. Some focused on the top results that Google surfaced (back when Google was still in the business of serving links), while others examined social media content or newspaper and TV news coverage. With large language models (LLMs) now forming a share of the public information diet, it's worth looking at how they handle climate change. In this case, the study used 33 models, including multiple versions of the major frontier and open-weight models.

This is the opening of the story, shared with credit to Ars Technica. The rest of the report — with every detail — continues on the publisher’s website.

Continue reading at Ars Technica ↗

Cookie settings

Choose which cookies we may use. You can change this at any time from Cookie settings in the footer. Details of each cookie are in the Cookie Policy.

Strictly necessary

Needed for sign-in, security, payments and remembering these choices. Always on.

Preferences

Remember settings such as dark mode.

Analytics & performance

Help us count visits, understand which pages, devices and countries our readers use, and measure how well the site performs – anonymous and aggregated.

Advertising & marketing

Lets advertising partners show and personalise adverts, measure impressions, clicks and visits, prevent ad fraud and pay us – this is how the free site is funded.

🔔 News notifications

Get headlines from NewsHub on this device — even when the site is closed.

You can switch this off at any time. Works on Android, Windows, Mac and Linux browsers; on iPhone/iPad add the site to your Home Screen first.