🔍 Read the full analysis: Can One Matrix Edit Fix Qwen’s JSON Tab Escaping? GPT-6 Astra Tested It on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
GPT-6 Astra, running as a Codex agent, made a rank-one edit to one matrix in Qwen3-4B-Instruct-2507. On a frozen test of copying special characters into JSON, exact answers rose from 49 of 120 to 70, with 21 cases fixed and none broken; the synthetic result has not been independently reviewed, and a separate table-extraction task remained at zero.
GPT-6 Astra, operating as a Codex agent, reported that a targeted edit to one matrix in Qwen3-4B-Instruct-2507 improved the model’s exact-answer score on a frozen JSON escaping test from 49 of 120 cases to 70. The result suggests a narrowly targeted weight change can help with a specific formatting failure, but it comes from synthetic tests and has not been independently reviewed.
The experiment focused on copying short texts into a JSON object without changing any characters. Inputs included tabs, line breaks, quotation marks and backslashes. A response counted as correct only if a standard JSON parser recovered the original text exactly; for example, a tab had to be represented as the escape sequence \t, not inserted as a literal tab or replaced with spaces. The report says the original model solved 16 of 32 initial screening cases, and a more explicit prompt also solved 16 of 32.
For the edit, the agent calculated a rank-one update to the output projection matrix of the model’s final multilayer perceptron layer. The report says there was no gradient training and that the other 397 stored tensors remained bit-identical to the original. On the final test, which the report says was frozen before the result was measured, the modified model produced 70 exact answers compared with 49 for the unmodified model: 21 cases improved and none regressed, a reported gain of 17.5 percentage points.
The improvement did not extend to every task. The modified model still got 50 of the 120 JSON cases wrong, and its score on a separate synthetic task—extracting a table row—remained 0 of 30. The experiment’s author describes the cases as synthetic. GPT-6 Astra planned and ran the experiment and wrote its report; Anthropic’s Claude was not involved in this test, although the source says Claude drafted the accompanying article.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in Data Formatting
JSON is commonly used to pass structured information between software services, applications and AI tools. A model that emits valid-looking JSON but changes a tab, newline or other character can corrupt a field without making the error immediately obvious. Exact recovery is therefore a practical test of whether a model can preserve data, rather than merely produce output that looks plausible.
The reported result matters as a small, task-specific demonstration of editing model weights without retraining the full model. It does not show that the edit improves general JSON ability, reliability across real-world inputs or performance on other models. The remaining errors and the unchanged table-extraction score mark clear limits. The tests also do not establish whether the gain would persist on naturally occurring data or under different prompts and software setups.
Popular right now
LG 77-Inch OLED evo C6 4K Smart TV
Ninja Crispi Microwave and Air Fryer with FusionCrisp, Silver, MW101
Ninja CREAMi Scoop & Swirl 13-in-1 Ice Cream and Soft Serve Maker, NC701
As an affiliate, we earn on qualifying purchases.
The Earlier Qwen Test
This was the third experiment in a series described by ThorstenMeyerAI.com as tests of AI systems editing or evaluating other models. In an earlier attempt on Qwen, GPT-6 Astra edited the model for extracting fields from synthetic business messages. Both the original and edited versions scored 97 of 200 on that experiment’s frozen final test. The report says both also produced 47 invalid answers, often by placing a literal tab in a JSON string instead of escaping it.
That previous result supplied the clue for the new test: rather than search broadly for a weakness, the agent targeted the known tab-escaping failure. The source also describes a separate earlier experiment in which Claude edited a different local model, Ornith, and a later review in which GPT-6 Astra identified problems in that work. Those episodes provide background to the series, but they do not independently validate the Qwen result.
“The starting hint for experiment 3 came from the previous attempt.”
— GPT-6 Astra’s experiment report, as quoted by ThorstenMeyerAI.com
Independent Checks Still Pending
The reported figures are drawn from the experiment’s stored results and public runs, according to the source, but a second AI or external reviewer has not yet checked this experiment. That matters because a previous test in the same series received a review that found implementation and reporting problems. The article does not provide an independent replication or a peer-reviewed evaluation.
The source describes the test cases as synthetic but does not establish how closely they represent real production data, how results vary across repeated runs, or whether the result depends on the particular prompt, parser or generation settings used. It also does not show that the matrix change improves other JSON tasks. The 0-of-30 table-extraction result and the 50 remaining errors on the main test underscore that the claim is limited to this measured task.
Review and Broader Testing
The immediate next step is independent review of the experiment’s code, test construction, saved matrix edit and reported scores. The source says the runs are public on lexassay.com, which gives reviewers material to inspect, but it does not announce a review date or a replication schedule.
Further testing would need to establish whether the improvement holds on additional, less synthetic examples and whether it survives changes in prompts and evaluation conditions. Until those checks are reported, the supported conclusion is specific: in this experiment, one rank-one matrix edit improved Qwen3-4B’s exact score on a frozen JSON special-character copying test, while substantial errors remained and another task showed no improvement.
Key Questions
What did GPT-6 Astra change in Qwen?
The agent calculated a rank-one update to the output projection matrix in Qwen3-4B-Instruct-2507’s final MLP layer. The report says it did not use gradient training and that the model’s other 397 stored tensors were unchanged.
How much did the JSON test score improve?
On the frozen test of 120 cases, exact answers rose from 49 to 70. The report counts 21 cases fixed and none broken, a gain of 17.5 percentage points. Fifty cases were still wrong.
What does the test count as a correct answer?
The model must place the text in JSON so that a standard parser recovers the original characters exactly. A tab, for example, must be escaped as \t; replacing it with spaces or including a raw tab counts as an error.
Does the edit improve Qwen’s other tasks?
The source does not establish a broader improvement. On a separate synthetic table-row extraction test, the score remained 0 of 30. The reported gain applies to the JSON escaping test.
Has the result been independently verified?
Not yet, according to the source. The runs are described as public, but the experiment has not received a second-AI review or an external replication.
Source: ThorstenMeyerAI.com
