🔍 Read the full analysis: GPT-6 Astra’s Review Uncovered Issues Claude’s Green Tests Overlooked on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A review by OpenAI’s GPT-6 Astra found reproducible flaws in a Claude Opus 5.5-led experiment to edit a local language model, including stale results that could be reused and a command-line failure missed by green unit tests. The review also prompted corrections to claims about weight changes and the strength of the experiment’s causal evidence; it did not establish that the edit improved legal accuracy.
An independent review by OpenAI’s GPT-6 Astra found reproducible bugs and unsupported claims in an experiment run by Anthropic’s Claude Opus 5.5 to edit a local language model, according to a report published November 5. The review found that green unit tests did not catch problems including a broken real command-line run and a resume process that could reuse old answers as new results; the model’s reported legal score remained zero.
The experiment targeted Ornith 1.5, a local model, with a weight-editing method inspired by a MIT CSAIL post on weight-space programming. The report says Claude identified and weakened a “loop direction” in the model’s weights without training. The model completed 12 of 15 legal answers after the edit, compared with four before, but its legal score stayed at zero. Completion counts do not establish that the answers were legally correct.
GPT-6 Astra, operating in Codex, was assigned to review the implementation rather than run a competing model-improvement experiment. It was instructed not to change the main code or inspect hidden benchmark tasks, the report says. The Claude session retained responsibility for the experiment, benchmark runs and blind grading by three Claude agents; it made the fixes after receiving the review findings.
The first review, on November 4, reported four reproducible problems. A resumed run could accept answers from a different model or sampling setup without generating new responses; an unknown task ID could produce a nominally successful run with no evaluations; comma-separated sampling settings conflicted with commas used to separate models in the real command line; and converting weights back into a compact format altered them even when the intended edit strength was zero. The report says the team changed the command syntax, made unknown IDs fail and stored edited tensors at higher precision.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Green Tests Missed Real Faults
The review highlights a practical weakness in evaluating AI experiments: passing unit tests are not proof that the full workflow works. In this case, parser tests passed, but a dry run of the actual command failed. A separate resume bug could have made earlier outputs appear to belong to a new setup, undermining the validity of a benchmark comparison.
Weight conversion also complicates claims about what caused a model’s behavior. The report’s first-round check found that a conversion at zero edit strength changed 0.84% of one matrix, with 105,315 of 1,048,576 reconstructed values differing. In a later review round, a separate measurement using a different number format recorded a 0.51% change and 915,360 changed values. These are distinct measurements of conversion effects, not legal-quality scores or error rates. They show why the edit itself must be separated from changes introduced by storage and conversion.
The case is also a limited example of independent AI review. The reviewer reported problems and corrected wording, while the original experiment team implemented the fixes. The report does not establish that AI review is generally reliable or that the underlying model-editing technique works beyond this test.
Popular right now
Breville BES870XL Barista Express Espresso Machine, Brushed Stainless Steel
Ninja Foodi DZ201 8-Quart DualZone 6-in-1 Air Fryer
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Black
As an affiliate, we earn on qualifying purchases.
How the Review Was Set Up
The two sessions began work on the experiment on November 4. Once the participants recognized that they were working on the same project, the roles were divided: Claude continued the experiment, while GPT-6 Astra reviewed its implementation. The report says the reviewer wrote separate checking programs and used small, reproducible examples rather than taking over the main test rig.
The review was intended to check the implementation and its claims, not to provide legal advice or independently grade every answer. According to the report, GPT-6 Astra did not inspect the hidden benchmark prompts, topics, answers or reference solutions. It also did not run a second model-improvement experiment. The report describes the first round’s four bugs and a later round in which the author’s 12 tests passed, while four of the reviewer’s six provenance checks failed. Those counts refer to checks of experiment records and implementation, not model performance.
Limits of the Review’s Evidence
The report does not provide enough information to establish whether the weight edit caused the change in answer completion, whether the result would hold across other tasks or models, or whether any legal responses became more accurate. The stated 12-of-15 completion count is not a legal-quality measure, and the legal score was zero. The report also says the causal claim based on one control was stronger than the evidence supported.
There are limits to the review itself. GPT-6 Astra did not rerun every test during its final documentation step, was not a fourth grader, and measured conversion effects on a CPU rather than the production GPU path, according to the report. The later provenance checks also exposed unresolved failures. The source does not fully detail those six checks or establish whether all resulting issues were corrected. The figures from the two conversion measurements concern different formats or stages and should not be combined into a single estimate.
Further Validation Still Needed
The immediate next step is to establish whether the fixes hold across the complete test workflow, including the production GPU path and a fresh run that records model, sampling and edit settings with each result. The report does not announce a date for that work or say that a further validation has been completed.
Any claim that the method improves legal performance would also require evidence beyond answer completion: repeated, controlled evaluations with the benchmark conditions disclosed as appropriate, independent grading and a clear account of conversion effects. Until then, the report supports a narrower conclusion: GPT-6 Astra identified implementation problems and prompted corrections, while the experiment’s legal-quality result remains unproven.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation of a Claude Opus 5.5-led experiment that edited the weights of the local model Ornith 1.5. It did not run a competing model-improvement experiment.
What problems did the review find?
The first review reported four issues: resumed runs could reuse results from another setup, unknown task IDs could yield empty runs reported as successful, a real command-line invocation failed despite passing parser tests, and weight conversion changed values even at zero edit strength.
Did the edit make the model better at law?
The report says the model completed 12 of 15 legal answers after the edit, up from four, but its legal score remained zero. Completion is not the same as correctness, so the source does not establish improved legal accuracy.
Did GPT-6 Astra inspect the hidden benchmark?
The report says the reviewer did not inspect hidden benchmark prompts, topics, answers or reference solutions. It also says GPT-6 Astra did not rerun every test during its final documentation step.
What remains to be verified?
The report does not show that the fixes have been validated across the full production GPU workflow, nor that the model-editing result is repeatable or improves answer quality. Those points need further testing.
Source: ThorstenMeyerAI.com
