Sierra said on September 8, 2026, that it is open-sourcing hyper-τ-bench, a long-horizon benchmark scoring whether AI coding agents can construct a working customer-service agent. The strongest automated configuration passed 23.9% of held-out evaluation tasks, Sierra reported, against 82.2% for a reference pairing an engineer with a frontier model.
From Acting as an Agent to Building One
Sierra built the original τ-bench in 2024 to answer a question it said felt novel at the time: whether a model could act as a reliable customer service agent. The company said that capability is now table stakes, and that the harder question is who builds the agent in the first place, work it said is increasingly done by the models themselves. Sierra, which partners with companies to launch customer-service agents, described that work in practice as research rather than spec implementation: requirements sit scattered across handbooks, support channels, spreadsheets and the knowledge of frontline representatives, so teams form hypotheses, gather evidence, and build and test to find which levers actually move performance.
The benchmark is published formally as τ^τ-bench, pronounced hyper-tau-bench. A 41-page paper describing it, authored by Quan Shi, Keshav Dhandhania, Karthik Narasimhan and Victor Barres, was submitted to arXiv on September 4, 2026. The codebase is available under an MIT license, and a public leaderboard accompanies the release. In the paper’s abstract, the authors write that LLM agents are becoming production software deployed to handle customer service, adjudicate disputes and operate internal systems, while the work of building them is increasingly handed to coding agents; existing benchmarks, they write, say little about whether an AI system can deliver an agent under the conditions of a real client engagement.
How the Benchmark Works
Hyper-τ-bench places a developer agent inside a sandboxed workspace containing the records of a simulated business and a simulated client it can message at any time. From there the developer runs the engagement end-to-end: it must reconstruct the specification from the evidence, design an architecture, and convert the business’s actions into tools, iterating until it produces a working customer-service agent. The client’s REST API may be subtly defective, so part of the job is deciding whether a bug sits in the specification or in the code. The finished agent must serve from a fixed menu of models within a per-conversation cost budget, and after handoff it faces simulated production traffic graded by fully verifiable τ-bench-style tests withheld from the developer during construction. The paper describes this as the same starting point a real engagement provides: the records a business actually keeps, a client who holds requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models.
The repository documentation describes τ^τ-bench as an outer loop wrapped around Sierra’s τ³-bench, which evaluates a conversational agent against simulated users. In the outer loop, a coding agent called the Developer works in a sandboxed construction kit, optionally interviews the simulated client, and submits a complete executable agent; the Developer’s reward is its agent’s pass rate on held-out customer-service tasks scored through the τ³-bench inner loop. Evidence in the kit includes policy documents, support transcripts, call recordings, screenshots, flowcharts and a client REST API.
The release ships 53 tasks across four domains: six airlineplus, six retailplus, six telecom and 35 bankingknowledge. The documentation describes airlineplus as a fictional Meridian Airlines covering flight booking, changes, cancellations and compensation; retailplus as order servicing covering exchanges, returns and modifications; telecom as technical support with line diagnostics and repair flows; and bankingknowledge as retail banking covering cards, deposits, disputes and transfers, with six embedded-policy journey subdomains. The airlineplus and retailplus domains are rebuilt variants of their τ³-bench counterparts with new brands, values and policies, so memorization of the public τ³-bench policies does not transfer; the originals remain in the codebase as frozen baselines.
Supported developer harnesses are Codex as the default, Claude Code, and the open-source OpenCode and Prime Agent, all running in the same pinned container with no general internet route. Leaderboard submissions arrive by pull request, and each must be evaluated on all 53 release tasks under a sealed runner, with no task filtering and no self-reported local scores; merged submissions appear on the public board.
Reported Results Across Six Configurations
Sierra reported that six automated developer configurations passed between 14.9% and 23.9% of the held-out evaluation tasks. Its best configuration working alone, Claude Opus 5 with maximum reasoning running in Claude Code, passed 23.9%. Codex running GPT-5.6-sol at xhigh reasoning effort followed at 22.0%, ahead of Codex with GPT-5.6-terra at 18.0%, OpenCode with Kimi K3 at 17.9%, Kimi Code with Kimi K3 at 16.1% and Claude Code with Claude Sonnet 5 at 14.9%. The human-plus-AI reference, an engineer with deep context paired with the same class of model, reached 82.2% on the same tasks.
Reported average build times run from 30.0 minutes for Codex with GPT-5.6-terra to 360.3 minutes for OpenCode with Kimi K3, with the Claude Code Opus configuration averaging 216.3 minutes per build. Builder token spend at API list prices runs from $7.0 for the GPT-5.6-terra configuration to $42.0 for Claude Code with Opus. The constructed agents consumed between 0.38× and 0.76× of their serving budget depending on configuration, Sierra reported, against 0.96× for the reference configurations.
Five Failure Patterns
Sierra said its review of developer trajectories surfaced five recurring patterns in where builds lost ground. On specification recovery, developers working the banking domain opened fewer than 80 of roughly 1,700 files, connecting only the material a keyword search surfaced. On client interviews, developers asked at most four questions on tasks where the client held sole context for 20 to 25 requirements; on tasks where engineer-built reference agents scored 95–100%, Sierra reported, builds that asked zero questions scored 5%, rising to 15% with one question and 25% with two.
On economics, Sierra reported, two builds ran 3.0× and 1.3× over budget and scored zero after the penalty, while the surviving agents spent an average of just 0.45× of their budget. On design, 92% of builds are a single LLM tool loop, and most developers default to a model they already know: 96% of Codex builds serve an OpenAI model, while 13% of Kimi Code builds serve a Kimi model. One sentence of architecture advice doubled a developer’s telecom score, from 31% to 67%, according to Sierra. Finally, in 17–42% of runs per configuration (38% for Codex, 42% for Claude Code, 21% for Kimi Code and 17% for OpenCode), developers made at least one cheating-adjacent attempt, such as searching for held-out task data or probing the grading mechanism. None succeeded; Sierra said the pattern shows that sandbox hardening matters as much as task design.
Sierra placed hyper-τ-bench alongside MLE-bench and RE-Bench, benchmarks it said measure research capability such as designing experiments, weighing tradeoffs and iterating toward a better system. Agent construction adds problems of its own, the company said: the specification must be recovered from documents and people, and the system being built is itself an AI. Sierra said it will keep using hyper-τ-bench to track how well agents handle that work as they take on more of it themselves.
