📰 Newspapers

The most misunderstood graph in AI just got a major update

The most misunderstood graph in AI just got a major update

A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory—but it also indicates that models aren’t quite as good at tasks that require judgment as they are at coding.

The report is an update to the organization’s time horizon plot, which was widely shared—and widely misinterpreted, due to the complexity of the measures it used. METR evaluated models by having them attempt a large suite of tasks, each of which had also been completed by humans. Each task’s difficulty was scored by how long it took the human testers to finish it—and then models were rated according to the length of the tasks that they could complete. That plot showed that AI systems were rapidly improving—every four months, the length of the tasks that frontier models could successfully complete doubled.

The updated plot adds more tasks—some that are estimated to take humans far longer than the tasks in the original suite, and others that are more open-ended. One of the biggest questions in AI right now is the extent to which progress on “verifiable” tasks—where it’s possible to check automatically whether an AI system has succeeded or failed—carry over into tasks on which success or failure is harder to measure.

In brief

A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory—but it also indicates that models aren’t quite as good at tasks that require judgment as they are at coding. METR evaluated models by having them attempt a large suite of tasks, each of which had also been completed by humans.

This is the opening of the story, shared with credit to MIT Technology Review. The rest of the report — with every detail — continues on the publisher’s website.

Continue reading at MIT Technology Review ↗

Cookie settings

Choose which cookies we may use. You can change this at any time from Cookie settings in the footer. Details of each cookie are in the Cookie Policy.

Strictly necessary

Needed for sign-in, security, payments and remembering these choices. Always on.

Preferences

Remember settings such as dark mode.

Analytics & performance

Help us count visits, understand which pages, devices and countries our readers use, and measure how well the site performs – anonymous and aggregated.

Advertising & marketing

Lets advertising partners show and personalise adverts, measure impressions, clicks and visits, prevent ad fraud and pay us – this is how the free site is funded.

🔔 News notifications

Get headlines from NewsHub on this device — even when the site is closed.

You can switch this off at any time. Works on Android, Windows, Mac and Linux browsers; on iPhone/iPad add the site to your Home Screen first.