Every AI vendor has a benchmark score to wave at you. The problem is that almost none of those scores tell you anything about whether the tool will survive contact with your actual company. On July 9 we published Enterprise-Bench, an open, vendor-neutral benchmark built to answer the question the others skip: can an AI agent do real work across a real enterprise?
I sit with enterprise IT and support leaders every week, and the pattern is always the same. A model tops some public leaderboard, the team runs a proof of concept, and it falls apart the moment it has to find the right answer inside fifteen years of fragmented data spread across systems that were never designed to talk to each other. The benchmark said one thing. Production said another.
Why generic benchmarks fall short
Most AI benchmarks test reasoning, coding, and tool use at single-user scale. That is a useful measure of raw model capability. It is a terrible predictor of enterprise performance, because enterprises pose a completely different class of problem:
- Data is sprawling and siloed. The right answer is buried in a haystack that gets bigger every quarter.
- Permissions and safety are non-negotiable. An agent that returns a correct answer it was not allowed to see has failed, not succeeded.
- Real tasks cross systems. Answering one question often means synthesizing data from several sources, with an auditable trail back to each one.
None of that shows up on a single-user coding benchmark. So the industry has been optimizing for the wrong test.
Enterprise-Bench adds a way to measure organizational complexity, not just task complexity.
That framing comes from our CTO, Ahmed Bashir, and it is the whole point. The hard part of enterprise AI is not the individual task. It is the environment the task lives in.
How it works
We built Enterprise-Bench with the Laude Institute, with researchers affiliated with UC Berkeley and Stanford, and it was validated by Professor Alexandros Dimakis of UC Berkeley. Jeff Smith in our Office of the CTO led the effort. It is fully open: the dataset, evaluation harness, queries, and judging criteria are all published on Harbor Hub so any vendor, customer, or researcher can run it and submit results.
Two design choices make it different.
Answer-preserving data scaling
The correct answer stays fixed while the surrounding irrelevant data expands up to 256-fold. At the smallest scale, roughly 40% of the data is relevant. At the largest, only 0.16% is. This deliberately isolates a system's retrieval architecture from raw model capability, because in a real enterprise the noise-to-signal ratio only gets worse over time.
Three axes, scored simultaneously
- Precision. The correct answer, with a verifiable source and an auditable path to it.
- Efficiency. Whether cost stays sane as complexity and data volume grow.
- Safety. Permission boundaries respected and every action traceable.
Tasks are graded on autonomy levels borrowed from self-driving, L1 through L4. The current L1–L2 benchmark covers factual retrieval and complex multi-source queries. L3 and L4 arrive later in 2026. Every result is verified by independent LLM judges against published criteria, with full traces submitted alongside the scores. No black boxes.
What it revealed
Here is the part that matters. We ran DevRev's Computer head-to-head against Claude Code on identical L1–L2 tasks, using the same underlying Opus 4.8 model. Same brain. The only variable was the architecture around it.
Read that again with the model held constant. A 48% relative improvement in accuracy and a 4.4x efficiency gap did not come from a smarter model. They came from the system wrapped around it, and the gap widened precisely on the enterprise-level work that required synthesizing data across systems.
Retrieval architecture predicts production performance better than the model does.
If two systems run the same foundation model and one is 48% more accurate and 4.4x cheaper, the model was never the differentiator. This is why a demo that dazzles on a clean dataset can quietly fail in your environment, and why "which model do you use" is the wrong first question.
Why an open standard, and why now
According to McKinsey's 2025 State of AI survey, nearly two-thirds of organizations still have not begun scaling AI enterprise-wide. Part of that hesitation is the absence of a neutral yardstick. Buyers are being asked to make eight-figure bets on the strength of marketing claims and staged demos.
We have seen this movie before. In the 1990s the database industry was a fog of competing performance claims until the Transaction Processing Performance Council established common standards. Overnight, the conversation shifted from marketing to evidence. Enterprise AI needs the same move, and no single vendor's internal scorecard can provide it. That is why Enterprise-Bench is open and why we published our own traces alongside everyone else's ability to challenge them.
DevRev's Computer currently tops the public leaderboard. I would rather it be tested and beaten in the open than protected behind a slide.
What to do with this
If you are evaluating AI for your service desk, support operation, or any workflow that touches real company data, the practical takeaway is simple: stop grading the model and start grading the system around it. Ask any vendor to show you results on your kind of complexity, with sources, permissions, and cost at scale, not accuracy on a toy dataset.
The dataset and harness are open, so you can run the comparison yourself. If you want to talk through what a real evaluation looks like for your environment, or where your current tooling would land on these three axes, my inbox is open.
Figures in this piece are drawn from DevRev's July 9, 2026 announcement of Enterprise-Bench and the accompanying results published on Harbor Hub. The benchmark is open for any vendor, customer, or researcher to run and submit results. If you see it differently, I would genuinely like to hear where you land.