🔍 Read the full analysis: Claude Edited A Local Model, GPT-6 Astra Checked It, And The Legal Score Stayed At Zero on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment used a Claude-written code workflow to edit the weights of local model Ornith 1.5 35B-A3B, reducing looping and increasing completed benchmark answers from 4 of 15 to 12 of 15 in one comparison. Its legal score remained zero; GPT-6 Astra reviewed the code and claims and identified faults. The result suggests the edit improved completion, not legal knowledge, and the experiment has significant limitations.
A weight edit to local AI model Ornith 1.5 35B-A3B increased the number of completed answers in a small legal benchmark from 4 of 15 to 12 of 15, but its legal score stayed at zero, according to an experiment reported by ThorstenMeyerAI.com on November 5. Claude Opus 5.5 wrote the code used for the edit, while OpenAI’s GPT-6 Astra reviewed the code and claims and identified faults; the result is a single experiment, not proof that AI editing improves legal performance.
The experiment tested whether an edit aimed at a model’s repetitive output could help it answer questions on law. Researchers ran Ornith, recorded its internal signals, calculated a direction associated with looping, and removed that direction at half strength from the output weights of 27 of the model’s 40 layers. The method, directional ablation, changes weights based on observed model behavior. It did not train the model on new legal data, and an AI did not inspect the weights as text.
In the reported comparison using the same setup, the unedited model finished 4 of 15 answers, while the edited version finished 12. But the edited answers met only 7 of 75 checked legal criteria, compared with 3 before the edit; both conditions received a legal score of zero. The 12 edited answers contained 62 wrong or invented citations, an average of 5.2 per answer, according to the report. These figures describe this benchmark run, not a general measure of the model’s legal abilities.
A separate comparison highlights how sensitive the result may be to settings: with the sampling settings recommended in Ornith’s model card and no weight edit, the original model completed 13 of 15 answers, also with a score of zero. The report says changing the local setup alone moved the original model’s completion count from 10 to 4. That makes it difficult to credit the edit alone for the difference in completed answers across runs.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Answers Did Not Mean Better Law
The experiment illustrates a practical distinction for anyone evaluating AI tools: a model that produces a complete response is not necessarily one that produces a correct one. Here, the edit appears to have reduced a specific failure mode—repetitive output—but did not yield reliable legal answers. Completion improved; the legal score did not.
That distinction matters in legal work, where fabricated or inaccurate citations can make a polished answer actively misleading. The report’s tally of 62 wrong or invented citations in the edited responses is a warning against judging a system by whether it finishes a task or sounds confident. The benchmark checks cited laws and court decisions against primary sources, but this experiment remains small and should not be treated as evidence about professional legal use.
The results also point to a simpler evaluation lesson: model settings can affect outcomes substantially. If a recommended sampling configuration lets the unedited model complete more answers than the edited setup, a weight change should be tested against carefully matched settings and repeated runs. Otherwise, it is hard to tell whether the change, the configuration, or ordinary variation explains the outcome.
Popular right now
De'Longhi Rivelia Fully Automatic Espresso Machine with Milk Frother, White
Breville BES870XL Barista Express Espresso Machine, Brushed Stainless Steel
Shark HydroVac MessMaster Wet/Dry Vacuum & Mop Combo (AW261)
As an affiliate, we earn on qualifying purchases.
From MIT’s Weight-Editing Proposal
The test was prompted by a post from four MIT CSAIL researchers—Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—describing what they call “weight-space programming.” Their reported Qwen example involved using GPT-6 Astra to reason directly about another model’s weights. The source article says that model’s maths score rose from 7.9% to 48.1% after an edit and that it stopped looping. Those are results reported by the researchers, not findings from the Ornith test.
The home experiment used a related but different technique. Rather than having GPT-6 Astra reason over weights directly, a Claude Opus 5.5 session wrote code to run Ornith, measure a direction linked to its repetitive behavior, and remove that direction from selected weights. The report describes this as a simpler test of the idea that correcting an obstacle can expose an existing ability, but cannot supply knowledge the model does not have.
Ornith was selected because the report says it showed the relevant looping behavior among the local models considered. Its publisher describes it as a mixture-of-experts model with about 35 billion parameters overall and roughly 3 billion active for each token, and built it for coding and agent work rather than law. The legal benchmark covers 15 questions on German, U.S. and EU law, uses no internet or tools, and checks citations against primary sources. The report says five questions are hidden and that the new results are not shown on the benchmark website.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, describing the experiment
Limits of One Benchmark Run
The reported comparison is limited to one run per setting and a small development set. The loop detector changed during the experiment, and three graders came from the same model family. GPT-6 Astra provided a separate review from another company, but it was one reviewer assessing the work—not a second, independently conducted experiment. The report says that review found real faults, without giving a full account of each fault in the supplied material.
It is also unclear how stable the completion difference would be across repeated runs, different sampling settings, other hardware or other legal questions. The original model’s completion count varied when the local setup changed, and the no-edit result under recommended settings was higher than the edited result in the stated fair comparison. The reported scores and citation counts should therefore be read as observations from this test, not as settled estimates of typical performance.
The source also describes the benchmark’s legal score as zero in both comparison conditions while listing changes in checked criteria from 3 to 7 out of 75. It does not explain in the supplied material how those criteria translate into the overall score. The experiment is explicitly described as not being legal advice, and the results do not establish that editing a model can make it suitable for legal work.
Replications Must Test Accuracy
The immediate next step would be to repeat the comparison across more runs while holding hardware, token limits and sampling settings constant. Such testing could establish whether the directional edit reliably reduces looping, and whether it improves completion beyond what ordinary configuration changes can achieve. The supplied report does not announce a schedule for a follow-up or a larger study.
Further evaluation would also need to focus on answer quality, not just whether a response arrives. That means checking legal conclusions and citations against primary sources, reporting results across the benchmark’s question set, and separating any development questions from hidden evaluation tasks. Until results of that kind are available, the finding is narrow: the edit was associated with more completed answers in one comparison, while the legal score stayed at zero.
Key Questions
What was changed in the local model?
The experiment removed a measured direction associated with looping at half strength from the output weights of 27 of Ornith’s 40 layers. The report says the process used code written in a Claude Opus 5.5 session and did not involve training on new legal data.
Did the edit improve the model’s legal performance?
Not according to the overall legal score reported: it stayed at zero. The edited model met 7 of 75 checked criteria, up from 3, and its answers included 62 wrong or invented citations, the report says.
What did GPT-6 Astra do?
OpenAI’s GPT-6 Astra, running in Codex, reviewed the experiment’s code, numbers and claims. The report says it found real faults. Astra did not conduct a separate competing experiment.
Was the edited model better than the unedited model in every test?
No. In the same-setup comparison, the edited model completed 12 of 15 answers versus 4 of 15. But with the model-card-recommended sampling settings and no edit, the original completed 13 of 15; both results had a legal score of zero.
Can this result be used as legal advice?
No. The report describes a small AI experiment, not legal advice. Its limited runs and the incorrect or invented citations it reports do not support relying on the model for legal decisions.
Source: ThorstenMeyerAI.com
