Pass rate on the 28-day business task, down from 75% and 78% on short tasks
Profitable runs over 28 days from the top short-task model
Bedrock models tested with identical prompts and no model-specific help
Ten Trillion Triangles LLC
United States
Artificial intelligence
Amazon Bedrock, Amazon ECS on AWS Fargate, Amazon S3, AWS Lambda, AWS Glue Data Catalog, Amazon Athena, Amazon QuickSight, Amazon API Gateway, Amazon DynamoDB, AWS WAF, Amazon Cognito, Amazon CloudFront, Amazon CloudWatch
Ten Trillion Triangles LLC engaged Avahi to build a benchmark platform that measures how reliably large language models follow complex rules on their own as the length of a task grows. Avahi delivered three simulated, rule-governed benchmarks that run against models on Amazon Bedrock, scoring defined separately for each benchmark, an automated data pipeline, an analytics layer with Amazon QuickSight dashboards, and a trace viewer that replays any run turn by turn. Using the platform, five models were each run in three trials on every benchmark with identical prompts and no model-specific help. The result is a fixed, reproducible baseline that Ten Trillion Triangles can use to evaluate agent frameworks against standard models.
Ten Trillion Triangles LLC is an AI technology company focused on making AI agents behave predictably on long, rule-driven work. The company has developed its own agent orchestration framework, T-Pipe, which is designed to enforce deterministic, long-horizon agent behavior. To support its work, Ten Trillion Triangles needed empirical, reproducible evidence of how standard models perform on complex, rule-driven tasks.
Enterprise agents increasingly run through many sequential decisions. They must follow rules, resist manipulation, and keep acting without supervision. Most existing benchmarks measure short tasks, which leaves a gap: how does reliability change as the task horizon grows when the rules and the test harness stay fixed?
Ten Trillion Triangles needed an answer that could be trusted and repeated. That meant identical prompts for every model, scoring that is only compared within the same benchmark version, results that can be recomputed from raw run files, and a way to see why a model failed rather than only that it failed. Without a purpose-built platform, comparisons between models would stay anecdotal and hard to reproduce.
Amazon Bedrock gave the benchmark engine one managed way to reach models from several providers. Five models were tested through the same engine and the same prompts, which is what a fair like-for-like comparison requires. Running the benchmark tasks next to Bedrock on AWS Fargate kept execution close to the models being tested.
The rest of the platform stayed on AWS as well. Amazon S3, AWS Lambda, AWS Glue Data Catalog, Amazon Athena, and Amazon QuickSight turn raw run artifacts into queryable tables and dashboards, and Amazon API Gateway, AWS WAF, and Amazon Cognito control access to the run API and the trace viewer. Keeping execution, data, analytics, and access control in one platform meant a run could flow from start to dashboard without manual steps.
The engagement required simulation engineering, data science, and cloud data engineering in a single delivery team. Avahi staffed backend engineering, GenAI and data science, and data engineering roles that worked to one shared contract: the set of files the benchmark engine writes and the data platform reads. That contract let each side change independently while the pipeline kept working.
Avahi ran the work in weekly sprints and held a weekly working session with the Ten Trillion Triangles team. In those sessions the client reviewed live dashboards and benchmark behavior and directed changes, for example asking for harder web pages in the shopping benchmark after early tests showed models completing it too easily.
Avahi built a Python benchmark engine that separates what a benchmark is from how it is executed and from which model runs it. Six small interfaces define the contract: participant, environment, tool, evaluator, benchmark case, and benchmark. The harness never calls a model provider directly. Models are reached only through participants, so adding a provider means adding a participant, not changing the harness. Prompts are generated from the same fixtures that the tools and scorers enforce, so the instructions cannot drift from the rules.
Three benchmarks form a deliberate gradient in task length. Butter-Bench tests navigating a human-oriented website to buy the right product across 18 scenarios, each with its own generated storefront and a six-step checkout. Drive-Thru tests order handling under adversarial pressure across 188 scripted customers in 11 adversarial categories. Open-Vend tests running a vending machine business for 28 simulated days, with vendors played by a second model, demand that responds to price, and machine breakdowns that must be repaired in time. Each benchmark has hard pass floors, so partial progress never turns a failure into a pass. A minimal overseer steps in only when a model stops calling tools, and its prompts give no hints about the task.
In the cloud, an authorized request to the run API starts one AWS Fargate task per benchmark. Each task writes its results and a compressed event trace to Amazon S3 and writes a completion manifest last. The arrival of that manifest starts an AWS Lambda function that validates the artifacts, writes curated Parquet tables, registers partitions in the AWS Glue Data Catalog, and rebuilds the reporting tables. Amazon Athena queries those tables, Amazon QuickSight presents them as dashboards, and a link from the dashboard opens a trace and replay viewer served through Amazon CloudFront behind Amazon Cognito login. Amazon CloudWatch dashboards and alarms track pipeline health, failures, and account spend, and the AWS resources are defined as infrastructure as code.
Benchmark engine with three benchmarks: Butter-Bench (18 shopping scenarios), Drive-Thru (188 scripted adversarial customers), and Open-Vend (a 28-day business simulation)
Model integration through Amazon Bedrock for five models, with identical, fixture-generated prompts for every model
Scoring with hard pass floors per benchmark and an overseer that handles stalled runs without giving hints
Run-control API on Amazon API Gateway, AWS Lambda, and Amazon DynamoDB, protected by AWS WAF and Amazon Cognito login, that starts benchmark runs as AWS Fargate tasks
Automated data pipeline that converts raw run artifacts into 13 curated and 21 reporting tables, queryable through the AWS Glue Data Catalog and Amazon Athena
Amazon QuickSight analytics with 34 datasets, 5 dashboard sheets, and 29 visuals
Trace and replay viewer that lets a reader step through any run turn by turn
Amazon CloudWatch dashboard and alarms for pipeline health, failures, and spend
Technical paper, campaign reports, and a result dataset from which every reported number can be recomputed
System architecture and knowledge transfer document, runbooks, and a documented source repository for handover
The platform produced a reproducible baseline of how standard models behave as task length grows. Across five models, the benchmark campaign pooled 270 Butter-Bench scenario runs, 2,820 Drive-Thru cases, and 16 Open-Vend runs, and every figure can be recomputed from the delivered result files.
Average pass rates were 75% on Butter-Bench and 78% on Drive-Thru, but only 25% on the 28-day Open-Vend business, where 4 of 16 runs passed.
No model’s short-task rank carried over to the long task. The highest Butter-Bench performer (85%) never turned an Open-Vend profit across its three runs, and the model that led Open-Vend — the only one with multiple wins there — still failed 1 of its 3 runs and wasn’t the top scorer on either short-horizon benchmark.
Pass rate overstated quality on Drive-Thru. Re-scoring by how many model calls each success took lowered the best model from 86% to 72% and one model from 79% to 50%.
Wrong decisions, not missed detections, drove Drive-Thru failures. Closing a correctable order or submitting an irreconcilable one accounted for 60% to 95% of every model’s failures.
Long-horizon results are shaped by harness design as well as by the model itself — but not all forms of help matter equally. A revised wording of Drive-Thru’s rejection guidance moved pass rates by only 1 to 3 percentage points. A more structural change — prompting a model to act again once it stalls into text-only output — had a far larger effect: one model’s completion rate on the 28-day task went from 0 of 4 runs to 3 of 3, using 2.9 to 6.8 million input tokens per run instead of 30 to 42 million.
Ten Trillion Triangles LLC
United States
Artificial intelligence
Amazon Bedrock, Amazon ECS on AWS Fargate, Amazon S3, AWS Lambda, AWS Glue Data Catalog, Amazon Athena, Amazon QuickSight, Amazon API Gateway, Amazon DynamoDB, AWS WAF, Amazon Cognito, Amazon CloudFront, Amazon CloudWatch
Ten Trillion Triangles LLC engaged Avahi to build a benchmark platform that measures how reliably large language models follow complex rules on their own as the length of a task grows. Avahi delivered three simulated, rule-governed benchmarks that run against models on Amazon Bedrock, scoring defined separately for each benchmark, an automated data pipeline, an analytics layer with Amazon QuickSight dashboards, and a trace viewer that replays any run turn by turn. Using the platform, five models were each run in three trials on every benchmark with identical prompts and no model-specific help. The result is a fixed, reproducible baseline that Ten Trillion Triangles can use to evaluate agent frameworks against standard models.
Ten Trillion Triangles LLC is an AI technology company focused on making AI agents behave predictably on long, rule-driven work. The company has developed its own agent orchestration framework, T-Pipe, which is designed to enforce deterministic, long-horizon agent behavior. To support its work, Ten Trillion Triangles needed empirical, reproducible evidence of how standard models perform on complex, rule-driven tasks.
Enterprise agents increasingly run through many sequential decisions. They must follow rules, resist manipulation, and keep acting without supervision. Most existing benchmarks measure short tasks, which leaves a gap: how does reliability change as the task horizon grows when the rules and the test harness stay fixed?
Ten Trillion Triangles needed an answer that could be trusted and repeated. That meant identical prompts for every model, scoring that is only compared within the same benchmark version, results that can be recomputed from raw run files, and a way to see why a model failed rather than only that it failed. Without a purpose-built platform, comparisons between models would stay anecdotal and hard to reproduce.
Amazon Bedrock gave the benchmark engine one managed way to reach models from several providers. Five models were tested through the same engine and the same prompts, which is what a fair like-for-like comparison requires. Running the benchmark tasks next to Bedrock on AWS Fargate kept execution close to the models being tested.
The rest of the platform stayed on AWS as well. Amazon S3, AWS Lambda, AWS Glue Data Catalog, Amazon Athena, and Amazon QuickSight turn raw run artifacts into queryable tables and dashboards, and Amazon API Gateway, AWS WAF, and Amazon Cognito control access to the run API and the trace viewer. Keeping execution, data, analytics, and access control in one platform meant a run could flow from start to dashboard without manual steps.
The engagement required simulation engineering, data science, and cloud data engineering in a single delivery team. Avahi staffed backend engineering, GenAI and data science, and data engineering roles that worked to one shared contract: the set of files the benchmark engine writes and the data platform reads. That contract let each side change independently while the pipeline kept working.
Avahi ran the work in weekly sprints and held a weekly working session with the Ten Trillion Triangles team. In those sessions the client reviewed live dashboards and benchmark behavior and directed changes, for example asking for harder web pages in the shopping benchmark after early tests showed models completing it too easily.
Avahi built a Python benchmark engine that separates what a benchmark is from how it is executed and from which model runs it. Six small interfaces define the contract: participant, environment, tool, evaluator, benchmark case, and benchmark. The harness never calls a model provider directly. Models are reached only through participants, so adding a provider means adding a participant, not changing the harness. Prompts are generated from the same fixtures that the tools and scorers enforce, so the instructions cannot drift from the rules.
Three benchmarks form a deliberate gradient in task length. Butter-Bench tests navigating a human-oriented website to buy the right product across 18 scenarios, each with its own generated storefront and a six-step checkout. Drive-Thru tests order handling under adversarial pressure across 188 scripted customers in 11 adversarial categories. Open-Vend tests running a vending machine business for 28 simulated days, with vendors played by a second model, demand that responds to price, and machine breakdowns that must be repaired in time. Each benchmark has hard pass floors, so partial progress never turns a failure into a pass. A minimal overseer steps in only when a model stops calling tools, and its prompts give no hints about the task.
In the cloud, an authorized request to the run API starts one AWS Fargate task per benchmark. Each task writes its results and a compressed event trace to Amazon S3 and writes a completion manifest last. The arrival of that manifest starts an AWS Lambda function that validates the artifacts, writes curated Parquet tables, registers partitions in the AWS Glue Data Catalog, and rebuilds the reporting tables. Amazon Athena queries those tables, Amazon QuickSight presents them as dashboards, and a link from the dashboard opens a trace and replay viewer served through Amazon CloudFront behind Amazon Cognito login. Amazon CloudWatch dashboards and alarms track pipeline health, failures, and account spend, and the AWS resources are defined as infrastructure as code.
Benchmark engine with three benchmarks: Butter-Bench (18 shopping scenarios), Drive-Thru (188 scripted adversarial customers), and Open-Vend (a 28-day business simulation)
Model integration through Amazon Bedrock for five models, with identical, fixture-generated prompts for every model
Scoring with hard pass floors per benchmark and an overseer that handles stalled runs without giving hints
Run-control API on Amazon API Gateway, AWS Lambda, and Amazon DynamoDB, protected by AWS WAF and Amazon Cognito login, that starts benchmark runs as AWS Fargate tasks
Automated data pipeline that converts raw run artifacts into 13 curated and 21 reporting tables, queryable through the AWS Glue Data Catalog and Amazon Athena
Amazon QuickSight analytics with 34 datasets, 5 dashboard sheets, and 29 visuals
Trace and replay viewer that lets a reader step through any run turn by turn
Amazon CloudWatch dashboard and alarms for pipeline health, failures, and spend
Technical paper, campaign reports, and a result dataset from which every reported number can be recomputed
System architecture and knowledge transfer document, runbooks, and a documented source repository for handover
The platform produced a reproducible baseline of how standard models behave as task length grows. Across five models, the benchmark campaign pooled 270 Butter-Bench scenario runs, 2,820 Drive-Thru cases, and 16 Open-Vend runs, and every figure can be recomputed from the delivered result files.
Average pass rates were 75% on Butter-Bench and 78% on Drive-Thru, but only 25% on the 28-day Open-Vend business, where 4 of 16 runs passed.
No model’s short-task rank carried over to the long task. The highest Butter-Bench performer (85%) never turned an Open-Vend profit across its three runs, and the model that led Open-Vend — the only one with multiple wins there — still failed 1 of its 3 runs and wasn’t the top scorer on either short-horizon benchmark.
Pass rate overstated quality on Drive-Thru. Re-scoring by how many model calls each success took lowered the best model from 86% to 72% and one model from 79% to 50%.
Wrong decisions, not missed detections, drove Drive-Thru failures. Closing a correctable order or submitting an irreconcilable one accounted for 60% to 95% of every model’s failures.
Long-horizon results are shaped by harness design as well as by the model itself — but not all forms of help matter equally. A revised wording of Drive-Thru’s rejection guidance moved pass rates by only 1 to 3 percentage points. A more structural change — prompting a model to act again once it stalls into text-only output — had a far larger effect: one model’s completion rate on the 28-day task went from 0 of 4 runs to 3 of 3, using 2.9 to 6.8 million input tokens per run instead of 30 to 42 million.
Let’s explore your high-impact AI opportunities together in a complimentary session
