🔍 Read the full analysis: We Tried Editing A Local Model At Home—Here’s How The AI Review Went on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A home experiment used directional ablation to reduce looping in the local Ornith 1.5 35B-A3B model. In the reported comparison, completed answers rose from 4 of 15 to 12 of 15, but the legal score stayed at zero; recommended sampling settings without an edit produced 13 completed answers. An independent GPT-6 Astra review found faults in the experiment, which had several stated limitations.
A home experiment reported November 5 found that a weight edit to the local Ornith 1.5 35B-A3B model helped it finish more legal benchmark answers, but did not raise its score: in the stated comparison, completions increased from 4 of 15 to 12 of 15, while the legal score remained zero. The test was a simplified technique inspired by an MIT CSAIL study, not a repeat of the researchers’ method, and an independent AI reviewer identified faults in the work.
The experiment measured a recurring internal signal associated with looping, then removed that direction at half strength from the output weights in 27 of the model’s 40 layers. The authors say they used no training and supplied no new legal data. Claude Opus 5.5 generated code to run the model and carry out the edit; Claude agents also graded answers. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, results and claims and found what the report describes as real faults.
On the same setup used for the reported comparison, the edited model completed 12 of 15 answers, versus 4 of 15 for the unedited model. But the 12 edited responses contained 62 wrong or invented citations, averaging 5.2 per answer, according to the report. Across 75 checked criteria, the edited answers met seven, compared with three before editing. Both results had a legal score of zero.
The report also gives a comparison that complicates the apparent improvement: using the model card’s recommended sampling settings, the unedited model completed 13 of 15 answers, also scoring zero. The authors say the local setup changed during testing and that this alone shifted the original model’s completion count from 10 to four. The available material does not provide enough detail to independently resolve the effects of that change.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Finished Answers, Same Legal Score
The result illustrates the difference between a model that can finish a response and one that can produce reliable work. Reducing repetitive output may make a system easier to use, but this test did not show that the model gained legal knowledge or became dependable for legal research. Its invented and incorrect citations are a concrete warning: a fluent, complete-looking answer can still fail basic checks against primary sources.
The findings also matter to people experimenting with local models because the claimed benefit depends on the comparison. The edited model beat the unedited run under one stated setup, while the original with recommended sampling settings finished one more answer than the edited model. With one run per setting and a setup change affecting the baseline, the experiment is suggestive rather than a firm estimate of how much the edit helps.
The report explicitly says the experiment is not legal advice. It supports a narrow observation about completion behavior in this test, not a claim that editing model weights makes a local AI suitable for legal work.
Popular right now
Hisense 43-Inch E6SR Series Hi-QLED 4K UHD Smart Roku TV (2026 Model)
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
De'Longhi La Specialista Touch Semi-Automatic Espresso Machine with Grinder
As an affiliate, we earn on qualifying purchases.
From MIT Weight Editing to a Home Test
The experiment followed a report by four MIT CSAIL researchers—Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—on what they called “weight-space programming.” In the case described by the source, GPT-6 Astra directly reasoned over another model’s weights and edited a Qwen model associated with looping. The source reports that the model’s maths score rose from 7.9% to 48.1% after that edit. That result is separate from the home experiment and should not be treated as evidence that the legal test improved.
The home test used directional ablation, a different, established style of intervention: run a model, record internal signals, identify a direction, then remove some of it from weights. The report cites prior research on single-direction edits, including work by Arditi and colleagues in 2024. Its stated prediction was limited: removing a barrier to completing a task might help, but an edit would not supply knowledge the model did not have.
The chosen system, Ornith 1.5 35B-A3B, is described by its publisher as a model for coding and agent tasks, rather than law. The test used a 15-question benchmark covering German, U.S. and EU law, with no internet or tools; cited statutes and court decisions were checked against primary sources. The report says five questions are hidden to prevent tuning against the full test. Its new results are not shown on the benchmark website, which, according to the source, still lists only the original Ornith result.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, author of the experiment report
Limits of the One-Run Comparison
The reported evidence is limited: there was one run per setting, the development set was small, and the loop detector changed during the experiment. The source also says a change in local setup shifted the original model from 10 completed answers to four. It does not give enough information here to determine how much of the observed difference came from the weight edit, sampling configuration or changed setup.
Grading also has a potential conflict: Claude carried out the edit and Claude agents performed blind grading. GPT-6 Astra provided a review from another company and found faults, but the source does not specify in this material what each fault was or whether all were corrected. The grading involved three graders from the same model family, according to the report, which limits how independent those assessments were.
It remains unclear whether the completion difference would persist across repeated runs, other models, or the benchmark’s hidden questions. The available results do not establish that the edit improved legal reasoning; the reported score remained zero. The benchmark site also had not posted these new runs as of the report.
Replication and Benchmark Updates
The immediate next step for evaluating the claim would be repeated tests under a fixed local setup, with the same sampling settings for edited and unedited models and the loop detector held constant. Publishing the code, complete run records and details of the independent review would help readers assess the identified faults and reproduce the comparison.
Readers can also watch for whether the report’s new results are added to the benchmark site; the source says they were not posted there at publication. Until repeatable results are available, the finding should be read narrowly: the edit was associated with more completed answers in one comparison, but the model’s legal score did not improve. The report says the test is an experiment, not legal advice.
Key Questions
What changed after the model edit?
In the reported same-setup comparison, the edited Ornith model completed 12 of 15 answers, up from four. Its legal score stayed at zero.
Did the edit make the model better at law?
The experiment did not show an improved legal score. The edited answers met 7 of 75 checked criteria, versus three before editing, and both runs scored zero. The report also found 62 wrong or invented citations in the 12 edited answers.
Was this the same method used by the MIT researchers?
No. The report says the MIT work involved GPT-6 Astra directly reasoning over and editing model weights. The home test used code written in a Claude Opus 5.5 session to measure a loop-associated direction and remove it from some weights, a technique called directional ablation.
Did the unedited model always complete fewer answers?
No. Under the model card’s recommended sampling settings, the unedited model completed 13 of 15 answers, according to the report. The authors also say a change in local setup shifted the original model’s completion count, making the comparison harder to interpret.
Can readers rely on this as legal advice?
No. The report says the test is not legal advice, and the model’s legal score remained zero. It documents a limited experiment about completion behavior, not a validation of the model for legal work.
Source: ThorstenMeyerAI.com
