From 99% to 37%: the real reliability of AI chains

Perplexity says it now lets a model manage its systems with barely any supervision. Before approving something similar in your company, you should understand a three‑line multiplication.
A model gets it right 99% of the time. You assign it a process of a hundred chained steps. How many times does it finish the whole thing without a single error?

It’s not 99%. It’s 37%.

That multiplication — 0.99 raised to the 100 — is the gap between the demo they show you and the bill you pay. And this week it stopped being a technicality.

What has been announced

OpenAI published the Perplexity case, the conversational search engine, using its latest model internally. Johnny Ho, co‑founder of the company, sums it up: “We can hand over end‑to‑end systems and supervise them far less frequently than in previous generations.”

Three concrete uses: drafting communications, making changes to software, and monitoring systems in operation.

It's an important piece of news, and you should read it knowing who authored it: it's a customer case study published by the model provider, and it contains not a single metric. It doesn't say how often a person reviews it, what controls remain in place, or which decisions still require a human signature. It's a statement of confidence, not a reliability report.

The interesting part is that the question it doesn't answer is actually measured, and by people who don't sell models.

How to truly measure what a system like this can endure

The reference is METR (Model Evaluation & Threat Research), an independent organization that evaluates model capabilities. In a paper authored by Thomas Kwa, Ben West and collaborators, with participation from Anthropic, they proposed a metric that is now standard and understandable without any technical background: the task time horizon.

The question they ask is: what is the longest task—measured by how long it would take an expert professional to complete—that the model successfully completes at least half the time?

Their two findings together explain the mismatch between expectations and what actually happens in a company.

The first is the one that circulates everywhere: that horizon doubles every 212 days, roughly seven months, consistently since 2019. It's an extraordinary pace and it's true.

The second is almost never mentioned, and it's the one that decides whether a project succeeds: if instead of being satisfied with getting it right half the time you demand getting it right 4 out of 5, the workload the model can handle shrinks by about a factor of five. In its measurement, a model that solved tasks in about 59 minutes with 50% reliability dropped to around 15 minutes when you required 80%.

Translated into executive‑committee language: the capacity figures that are published are measured at a reliability level that no company would accept in a real process. Hitting 4 out of 5 times remains untenable for almost anything that touches a client or an account.

The multiplication that almost nobody does

The reliability of a chain is not averaged: it is multiplied. Each additional step erodes the result of the previous one, and the whole falls much faster than intuition suggests:

• 99% success per step, 10 steps: 90% overall success.
• 99% per step, 50 steps: 60.5%.
• 99% per step, 100 steps: 36.6%.
• 95% per step, 20 steps: 35.8%.

Turn it around and you get a budgeting rule. If you want a process to succeed 90% of the time:

• With 99% success per step, you can have 10 steps.
• With 99.5%, you can have 21.
• With 99.9%, you can have 105.

It’s a deliberately simplified calculation — assumes errors are independent, and in reality they sometimes chain and sometimes cancel out — but the order of magnitude is right. And from it comes the conclusion that saves the most money: half a point of reliability per step doubles the complexity of the process you can automate. Ten points more in an intelligence ranking, not.

Where the real money is

Here’s what changes an investment decision, and it’s that intuition leads to the wrong place.

When an automation project fails, the reflex is to ask for a better model. But if the problem is the upstream multiplication, a better model only nudges per‑step accuracy by a couple of points and doesn’t fix a fifty‑step process. What does fix it is the engineering around the model, which costs far less than the price gap between vendors and relies on four properties that the software industry has been applying for decades:

• That repeating a step causes no harm. If retrying a delivery duplicates the shipment, you can’t retry anything. This property, known in engineering as idempotency, turns a failure into a nuisance rather than an incident.
• That the process remembers where it left off. When something fails at step 40, it resumes at 40 instead of starting over.
• That it can distinguish a stumble from an error. A service that is slow is retried; an inconsistent datum stops and raises an alert.
• That each step validates its output. Checking the output against a set of rules before passing it to the next step prevents a small mistake from reaching the end as a disaster.

None of the four is a recent breakthrough. They are the playbook for any critical system, and they are precisely what separates a flashy pilot from something that can run without anyone watching.

The uncomfortable difference

With a nuance worth knowing before signing anything. In a traditional automated process, someone wrote the steps and knows what they are. With an AI agent, the model decides the steps on the fly. You cannot validate the output of a step that no one knew would exist, and a classic system fails loudly while a model can get it wrong silently and with complete composure.

Therefore the useful question isn’t whether the model is good enough.

The three questions before approving the project

If someone presents you with an automation that uses agents, these three questions separate a serious proposal from a demo:

1. How many steps does the process have and what reliability does each have? Multiplied, that’s the actual success rate. If no one has done that calculation, the project isn’t properly sized.
2. Which step is the most expensive to undo, and is there a person in front of it? Sending a communication to the entire customer base, publishing a price, executing a charge. The irreversible step cannot be retried, and therefore it is not automated without a signature.
3. How do we find out that it has failed? If the answer is “because we reviewed it,” there is no control system: it’s manual monitoring under a different name, and that’s exactly what the project promised to eliminate.

Takeaways

• The capacity advertised by vendors is measured at reliability levels your company would not accept. Ask for the figure at 80 %, not at 50 %.
• The real success rate of a process is the per‑step accuracy raised to the power of the number of steps. Do that calculation before approving the budget.
• When a pilot fails, the problem is rarely the model. Investing in the surrounding engineering is cheaper and yields more than switching vendors.
• Irreversible steps remain behind a human. Not out of moral prudence: because they cannot be retried.

The good news is that none of this is new. It’s what is already required of any system that the operation depends on. The only thing that has changed is that now the system writes its own steps, and that makes the questions above more necessary, not less.

Sources

• OpenAI, "Perplexity trusts GPT-6 Astra with end-to-end systems", September 2026, with statements from Johnny Ho, co‑founder of Perplexity. It is a customer case study published by the model provider and does not provide metrics — https://openai.com/index/perplexity-improving-accuracy-with-astra/
• Thomas Kwa, Ben West and collaborators (METR — Model Evaluation & Threat Research, with participation from Anthropic), "Measuring AI Ability to Complete Long Tasks", March 2025 and subsequent updates. From this comes the duplication every 212 days (95 % confidence interval: 171–249 days) and the 80 % horizon about five times shorter than the 50 % horizon — https://arxiv.org/abs/2503.14499 and https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
• Composite reliability calculations are proprietary and assume independent steps. In practice, errors are correlated, so they serve as an order‑of‑magnitude estimate rather than a precise prediction. METR also warns that extrapolating its trend involves considerable uncertainty.