🔍 Read the full analysis: How CUGA, Hermes, And Deep Agents Shape The Same AI Model on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
In an IBM-reported AppWorld experiment, CUGA completed 73 of 90 tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The results suggest agent-harness design can affect performance on this benchmark, but they do not establish a general ranking or show how the systems perform on real enterprise workloads.
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents running the same model. The comparison suggests that the software coordinating an AI model can materially affect its results on a particular multi-step workload, but it does not establish that CUGA is generally better across tasks or deployments.
The experiment used one model across all three systems: GLM-5.3-flash. IBM says each harness was run on the same 90 tasks across 30 scenarios using the Harbor evaluation framework, with one attempt per task. The reported task completion rates were 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents.
IBM also compared results by scenario, counting a scenario as passed only when all three variants within it passed. CUGA passed 20 of 30 scenarios, Hermes passed eight and Deep Agents passed two. The systems did not use identical configurations: CUGA had task decomposition, app-focused tools, checks before execution and automatic reflection. Hermes used Harbor’s default setup with AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP alongside LangChain middleware for tool selection, context summarization, task lists, retries and call limits. IBM says it made two fixes to Deep Agents so malformed model replies would not end runs.
The experiment setup is publicly available in harbor-eval, IBM’s Harbor fork, according to the source material. AppWorld tests multi-step tasks using nine apps and 457 APIs, including email, payments and file storage. IBM describes the benchmark as a proxy for some abilities relevant to enterprise automation, such as using API documentation and changing information across apps. It uses consumer applications, however, and does not directly test performance on real enterprise systems.
Harness Design Shifts Benchmark Results
The reported gap is a reminder that a model score may reflect more than the model itself. An agent harness determines how work is broken into steps, which tools are available and whether actions or outputs are checked. Those choices can shape whether a model finishes a task that requires several tool calls, even when the underlying model is held constant.
For teams evaluating agents, the result supports testing the model, harness and workload together, rather than treating a model benchmark as a complete forecast of real-world performance. The findings are most relevant to workflows resembling AppWorld’s multi-app API tasks. They do not show that CUGA would lead in personal assistance, research or other work for which Hermes and Deep Agents are designed.
The result also does not measure whether an agent follows organizational rules. IBM describes CUGA as offering features for applying rules through its harness, but policy enforcement was not tested in this experiment. Performance on the task benchmark should not be read as evidence that an agent is safe or compliant in deployment.
Popular right now
Vitamix 5200 Professional-Grade Blender, 64 oz. Container, Black
Breville BES870XL Barista Express Espresso Machine, Brushed Stainless Steel
KitchenAid 5.5-Qt Bowl-Lift Stand Mixer, Porcelain White (KSM55SXXX)
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Workflows
The comparison brings together systems with different intended uses. The source describes Hermes, from Nous Research, as a harness for always-on personal assistants, and Deep Agents, from LangChain, as suited to longer tasks such as research. CUGA, short for ConfigUrable Generalist Agent, is IBM’s open-source harness aimed at work across enterprise systems such as customer relationship management and payments.
That distinction matters when interpreting the scores: this was a comparison on a shared benchmark, not a test of each product across every task it was built to handle. The source also points to community directories that track nearly 200 agent harnesses, indicating that the software layer around models is a growing area of development. It cites a separate directory analysis in which GLM-5.2’s SWE-bench Pro result changed from 23% to 52% with a different harness. That is a separate reported comparison, not evidence from the AppWorld experiment.
“This is not a verdict on Hermes or Deep Agents.”
— IBM
Limits of the AppWorld Evidence
The comparison covers one model, one benchmark and 90 tasks, with one attempt per task. IBM says the harness configurations differed, so the results do not isolate every implementation difference. The supplied material does not show whether the ordering would persist with other models, more tasks, repeated runs or workloads drawn from production environments.
It is also unclear how the systems compare on cost, latency, reliability at scale or deployment outcomes. The test did not include policy enforcement, access controls, approval steps or data boundaries. The source provides no independent replication or timeline for a broader comparative study, so the reported results remain a narrow benchmark finding.
Testing Against Real Workflows
The public harbor-eval setup gives researchers and teams a way to inspect the reported test, according to the source. For organizations considering these systems, the practical next step is to test the intended model and harness against representative internal workflows, using the tools and configurations planned for deployment.
Those evaluations could track completion across repeated runs and varied tasks, alongside requirements the AppWorld experiment did not measure, such as permissions, human approvals and data boundaries. The supplied material does not identify a date for new results or an independent replication. Until broader evidence is available, the reported scores should be treated as a signal about harness effects on AppWorld—not as a forecast of performance across enterprise work.
Key Questions
What did the IBM-reported AppWorld test find?
CUGA completed 73 of 90 tasks with GLM-5.3-flash. Hermes completed 42 and Deep Agents completed 27 using the same model.
Does the test prove CUGA is the best agent harness?
No. It reports results on one benchmark, one model and 90 tasks. It does not establish a general ranking across other workloads or deployments.
What is AppWorld testing?
AppWorld tests multi-step tasks across nine apps and 457 APIs, including examples involving email, payments and file storage. The benchmark uses consumer apps rather than real enterprise systems.
Did the experiment test security or policy compliance?
No. The source says the experiment did not use policies, so its scores do not show whether the systems enforce organizational rules or meet deployment requirements.
Can other teams examine the experiment setup?
The source says IBM made the setup available in harbor-eval, its Harbor fork. It does not provide a timeline for an independent replication or a broader comparison.
Primary source: Hugging Face · via ThorstenMeyerAI.com
