AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: What Are Harness Adapters, And How Do They Adapt Your Favorite Model? on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face says reinforcement-learning LoRA adapters tailored to Pi, OpenCode and Hermes Agent outperformed the base Qwen3.8-27B model on two benchmarks. The supplied account gives one numerical comparison: in a Pi test on Terminal-Bench 2.1, pass@1 rose from 39% to 55%, while average token use fell from 2.9K to 1.5K per task. Independent replication and detailed results for the other harnesses are not provided.

Hugging Face says it trained reinforcement-learning LoRA adapters for the Pi, OpenCode and Hermes Agent coding harnesses, and that each improved Qwen3.8-27B results against the unadapted model on two benchmarks. The account provides detailed figures only for Pi: on Terminal-Bench 2.1, reported pass@1 increased from 39% to 55%, while average token use fell from 2.9K to 1.5K per task.

Pass@1 measures whether the model completes a task successfully on its first attempt. In the Pi evaluation, the team compared the base model with the same model weights frozen and a custom LoRA adapter added. The project attributes the lower token use to less reasoning and fewer tool calls, alongside more first-attempt successes. Those explanations are the team’s account of its results, not a separately verified analysis.

The project says it trained distinct adapters within Pi, OpenCode and Hermes Agent, then evaluated each with its corresponding harness. It reports that all three adapters beat the base model on performance and efficiency across two benchmarks. However, the supplied summary gives no numerical results for OpenCode or Hermes Agent and does not identify both benchmarks, so those reported comparisons cannot be assessed in detail here.

Training and agent rollouts used Hugging Face Hub infrastructure with Jobs, TRL’s AsyncGRPOTrainer, OpenEnv and Harbor. The account says the agents ran in their shipped forms, with a proxy recording tokens, rather than substituting a training framework’s version of each agent loop. The project lists the final training images as Pi 1.5.2, OpenCode 2.2.7 and Hermes Agent v0.22.6.

At a glance
reportWhen: Reported in the supplied project accoun…
The developmentHugging Face has reported that separate reinforcement-learning LoRA adapters tailored to three AI agent harnesses improved Qwen3.8-27B results in the project’s evaluations.
AI Agents · Briefing

Harness adapters tune models to the way agents work

Hugging Face reports that harness-specific LoRA adapters improved Qwen3.8-27B coding results while reducing token use. The detailed comparison provided is for Pi.

55%
Pi pass@1, up from 39%
1.5K
average tokens per task, down from 2.9K
3
harnesses: Pi, OpenCode, Hermes Agent

One adapter for each workflow

The team trained adapters inside the harnesses where agents use tools and complete tasks.

Pi

Detailed result reported

On Terminal-Bench 2.1, pass@1 rose from 39% to 55% with the adapter.

OpenCode

Harness-specific tuning

The project says its adapter beat the base model, but supplied figures do not quantify the result.

Hermes Agent

Same reported approach

A separate adapter was trained for this harness; detailed scores were not provided.

What changed in the Pi test

The team attributes the gains to less reasoning, fewer tool calls and more successful first attempts.

Base model
  • 39% pass@1
  • 2.9K average tokens per task
With LoRA adapter
  • 55% pass@1
  • 1.5K average tokens per task

A recipe others can inspect

The project describes training in the shipped agent environments and says artifacts are available on Hugging Face Hub.

1
Prepare coding tasks

Tasks came from existing sources and generated examples, with validation checks described by the team.

2
Train in each harness

The model weights stayed frozen in the reported Pi comparison while a custom LoRA adapter was added.

3
Share artifacts

Adapters, GGUF files, tasks and a SafeAgents plugin are reported as available for inspection.

What the evidence leaves open

The reported gains are tied to the project’s experiments and evaluation setup.

  • Numerical comparisons for OpenCode and Hermes Agent are not included.
  • The supplied account does not provide uncertainty ranges or independent replication.
  • More results are needed to assess the merged “super adapter” and performance on other task sets.
CaveatLower token use may matter for inference costs, but the material does not report costs or show whether the gains generalize.
The takeaway
Harness-specific tuning is promising: the Pi result pairs higher first-attempt success with lower token use, while broader claims need full results and independent replication.

Why Harness-Specific Training Matters

An AI agent harness governs how a model uses tools and carries out a task. A model’s general coding ability may not translate directly into success within a particular agent workflow, so training in that environment could tune its behavior to the tools and steps it actually encounters. Hugging Face’s results offer a project-level example of that approach: its Pi figures pair a higher reported first-attempt success rate with lower token use.

For teams that run agents repeatedly, fewer tokens and tool calls could reduce some usage costs and make workflows more efficient. But token totals alone do not establish the full cost or practical value of an adaptation. The supplied account does not report monetary costs, evaluation uncertainty or whether the gains carry over to other tasks and deployments. The figures should be treated as results from the project’s experiments, not a general guarantee about harness adapters.

The team also says it merged the three harness-specific LoRAs into one “super adapter” that runs unchanged across Pi, OpenCode and Hermes Agent. If the approach holds up, a shared adapter could simplify distribution. The available account does not describe the merge method or give comparative results showing whether the combined adapter matches the separate ones.

De'Longhi Classic Signature Espresso Machine with Adjustable Milk Frother

De'Longhi Classic Signature Espresso Machine with Adjustable Milk Frother

Sony BRAVIA XR8B 65-Inch OLED 4K HDR Smart Google TV (K-65XR8B)

Sony BRAVIA XR8B 65-Inch OLED 4K HDR Smart Google TV (K-65XR8B)

As an affiliate, we earn on qualifying purchases.

Tasks and Training Setup

The work extends earlier experiments that Hugging Face describes as involving multi-harness reinforcement learning and harness adapters for data analysis and code-building tasks. This iteration applies the approach to Qwen3.8-27B and software-engineering work across three agent harnesses. The team says its aim included publishing a reusable training recipe as well as measuring performance.

For training, the team assembled coding tasks in Harbor format from existing datasets and newly generated examples. The reported sources include SWE-smith, which injects bugs into Python and Go repositories; SWE-Gym tasks packaged in TaskTrove, based on real GitHub issues and merged fixes; and Tasksmith tasks created from merged pull requests. The supplied account says 24 tasks from 30 pull requests passed checks across 10 Python repositories.

The team says it screened tasks with OpenEnv validation and checked their difficulty against the base model. Such checks are relevant because reinforcement learning depends on tasks and verifiers that reward correct solutions. The project account describes steps intended to identify faulty tests, but does not give full validation results or a complete breakdown of the task set.

““Most of this is down to the model reasoning less, calling less tools, and performing better first time.””

— Hugging Face project team

What the Evaluation Leaves Open

The supplied account contains a detailed numerical comparison for one harness and one named benchmark: Pi on Terminal-Bench 2.1. It does not give the corresponding OpenCode or Hermes Agent scores, identify the second benchmark in its summary, or report sample sizes, uncertainty ranges and a full task-level breakdown. Those omissions make it difficult to judge the size and consistency of the reported gains across evaluations.

It is also unclear how the merged adapter compares with the three individual adapters, and whether the results hold outside the selected tests. The account describes generated tasks and verifier checks but includes no independent audit of those checks. It does not establish that other researchers have reproduced the findings. The reported results should therefore be read as claims from the project team, rather than independently confirmed performance gains.

Artifacts Available for Reproduction

Hugging Face says the adapters, GGUF files, reinforcement-learning tasks and a SafeAgents plugin are available through a collection on its Hub. Their availability gives developers and researchers materials to inspect and a route to attempt the training recipe with their own tasks and harnesses.

The next useful evidence would include complete scores for all three harnesses and both benchmarks, alongside task counts, uncertainty estimates and comparisons between the individual adapters and the merged version. Independent attempts to reproduce the Pi result and evaluate the approach on different tasks would help establish how broadly the reported changes apply. The supplied account does not specify a date for further results or independent evaluations.

Key Questions

What is a harness adapter?

In this report, it is a LoRA adapter trained for use with a specific AI agent harness, such as Pi, OpenCode or Hermes Agent. The harness supplies the tools and workflow through which the model completes tasks.

What changed in the reported Pi test?

On Terminal-Bench 2.1, Hugging Face reports that Qwen3.8-27B pass@1 rose from 39% to 55% with the Pi-specific adapter. Average token use declined from 2.9K to 1.5K per task in that reported comparison.

Did all three harness adapters show the same improvement?

The project says adapters for Pi, OpenCode and Hermes Agent each beat the base model on two benchmarks. The supplied account gives numerical results only for Pi, so the size of the other reported changes cannot be compared here.

Have the results been independently confirmed?

The supplied material does not report independent replication. It presents the findings as the project team’s experiments and does not provide enough detail to assess all benchmark results or uncertainty.

Can other developers try the method?

Hugging Face says the adapters, GGUF files, training tasks and SafeAgents plugin are available in a collection on its Hub. The account does not specify when further results or independent evaluations will be published.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

9 Portable SSDs Leading AI Storage Solutions In 2026

Discover the nine leading portable SSDs in 2026 for AI storage, highlighting speed, capacity, durability, and value for professionals and enthusiasts.

Testing 12 Different Zigbee Temperature/humidity Sensors

SmartHomeScene compared 12 battery-powered Zigbee temperature and humidity sensors against a calibrated reference, including a fridge test added in August.

Is Your Desk Ready For AI Automation In 2026?

ThorstenMeyerAI.com outlines a desk checklist for AI development, with advice on choosing a laptop, development board, dock and optional hardware.

7 Best Tablet Stands and Docks for Prime Day Deals in 2026

Explore the best tablet stands and docks on Prime Day 2026, including top picks for stability, comfort, and versatility for various setups.