The end of benchmarks: why GPT-6 Astra makes leaderboards unreadable

By AI Secret (Magna & Ben) · September 3, 2026.

OpenAI made it official this week: the company is presenting GPT-6 Astra as the entry into the 'Era of AGI', pitching it as a generational leap with major advances in computer use, coding, research and the kind of long, messy work that until now required constant human supervision. The newsletter's authors use that marketing as a starting point to raise a question that, they say, the industry has long been dodging: how do you actually measure this?

According to the article, AI has run on benchmarks for years: every launch comes with a table showing the new model is 3% better here, 7% better there, and mysteriously 12% smarter on a benchmark that didn't exist the previous spring. Models get ranked, tables get color-coded, leaderboards pile up and a winner is crowned. The piece argues that this system worked reasonably well when models were basically answering questions, but it stops working as soon as models start *doing* things.

The central argument is that a benchmark is nothing more than a question bank: the model is given a problem, produces an answer, and that answer is checked against an answer key. Even the most sophisticated 'computer use' benchmarks, the authors note, are at bottom a set of pre-written tasks running inside equally pre-built environments. That is useful, but it bears no resemblance to what happens when an agent is let loose on a real machine and asked to sort something out, because real life doesn't come with a benchmark file: nothing on a real desktop tells you exactly what 'done' looks like. A real computer is a browser that crashes mid-task, a website redesigned the day before without warning, a spreadsheet with three contradictory versions, a permission nobody mentioned, a PDF buried six folders deep, an API that times out on precisely the worst request, and a person who says 'take care of this' and walks off without defining what 'this' is. The agent has to figure it out, and the part almost no benchmark measures well is its ability to recover when the first attempt fails, a completely different skill from getting an answer right.

That is why, according to the article, reports that OpenAI has bought tens of thousands of consumer Macs for reinforcement learning training and computer use make more sense. The authors themselves caution that those specific figures have been reported, not audited, but they say the underlying direction is unmistakable: the major AI labs want their models operating on real machines, not locked inside a chat window, and that quietly breaks the benchmark game as it existed.

The newsletter illustrates the problem with an example: two agents, the same computer, the same vague brief — 'research this market, pick three promising companies, build a financial comparison and prepare a presentation for the investment committee'. There is no single right way to do it; there are hundreds. One agent browses for two hours, another writes a script, a third stumbles on a better data source halfway through and scraps its original plan, a fourth makes a mistake, spots it and fixes it, and a fifth delivers a slick presentation built on completely wrong numbers. The benchmark author cannot write down the right answer in advance, because there isn't just one.

According to the authors, this raises a genuine discomfort: the hardest part of evaluating an agent may no longer be 'was the answer correct?' but 'was this actually useful?'. To argue this is no minor nuance, they cite the example of an agent that scores over 95% on a computer-use benchmark and, when deployed for a week at a real company, builds the wrong folder structure, overwrites a production spreadsheet, wastes six hours chasing a dead end, confidently cites a 2023 statistic and gets stuck at a permissions wall it never thinks to ask how to get around: it aces every benchmark task, but nobody would keep it on for a second week. By contrast, an agent that scores only 85% but knows when it is confused, asks the one question that unblocks it, switches strategy when something breaks and delivers usable work is harder to rate on a leaderboard, which simply has no way of capturing that difference.

The authors argue that independent benchmarks are hitting a wall not for lack of effort by researchers — who are building ever harder and more realistic tests — but because reality remains bigger than any test: you can make the benchmark longer, add more tools, more sites, more applications, more steps, more randomness, and nothing fundamental changes, because it is still a game with rules and, as soon as a model learns to play it, you are back to square one. The result, they say, is that we will keep seeing scores like 91.4%, 94.7%, 97.2% or 98.1%, people will argue over whether Model A beats Model B, someone will launch another leaderboard and yet another to rank the leaderboards, while the actual model keeps doing things on a real computer that none of those figures describe. They frame this as the 'benchmark paradox': the closer AI gets to general-purpose intelligence, the less a bank of pre-written questions tells us about it.

As a way out, the article does not propose 'a better benchmark' but something fundamentally different: stop giving the agent a question and give it a goal; stop offering it a controlled environment and give it a messy one; stop scoring whether it followed the expected steps and score whether it achieved the outcome; stop repeating the same public test and send it into constantly changing environments, with private tasks it has never seen; and stop asking only whether it succeeded, and also ask how reliably it does so, what it cost, how long it took, how many times it needed a human, how safely it behaved and whether the work survived contact with reality. Their formula: the next generation of AI evaluation should look less like an exam and more like a professional internship — not asking whether the model passed the test, but giving it a laptop, an account, a budget, a goal and a week, and watching what it does. The authors acknowledge this would say far more about the system's real intelligence, though they admit it would be a nightmare to standardize, and they close with the idea that, alongside the 'Era of AGI', the 'Era of Unscorable AI' is beginning.