🔍 Read the full analysis: A 15-Question Legal Test For 42 AI Setups on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
A test run by Thorsten Meyer AI says it evaluated 42 AI setups on 15 questions about German, US and EU law, with models answering offline and citations checked against official sources. Its operator reports low scores for some legal-focused models and hundreds of invented or misattributed citations, but says the exercise is a spot check, not a strict benchmark.
Thorsten Meyer AI has published results from Lexassay, a test of 42 AI setups answering 15 legal questions across German, US and EU law. The operator says the legal-focused GLM-5.3 FlashX Legal scored 8 out of 100, while its citation review found hundreds of references that were invented or did not support the claims made.
Lexassay’s 15 tasks include deadline calculations, changes in law and court decisions, as well as questions containing false assumptions that a reliable answer should challenge. Ten questions are public and five are hidden; the hidden-set score is published without the answers. Models respond without internet access or tools, and different “effort” settings may be tested separately.
According to the published results, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Meyer says even the leading model fully met every criterion on only eight of the ten public tasks. Mistral Large 4 scored 10, while GLM-5.3 FlashX Legal scored 8 and fully solved none of the 15 tasks. The operator says it produced no answer on two.
Lexassay grades answers against model-specific gold answers with three to six weighted criteria per task. It checks citations against official databases and classifies them as real, misattributed, invented or not findable. Across the published runs, Meyer reports 2,644 confirmed real citations and 804 invented or misattributed citations. Under the scoring rules, each invented or misattributed citation costs a quarter of the task’s points; an answer without a legal-advice disclaimer or referral to a lawyer loses 5%.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Citation Errors Can Change Legal Advice
The test highlights a practical risk for people using chatbots to understand legal options: an answer can sound specific while citing a decision or statute incorrectly. A mistaken appeal deadline or an inaccurate account of a court ruling could affect a person’s next step, even when much of the response is right.
The article illustrates this with two hypothetical German civil cases whose judgments arrive on the same date in Munich and Berlin. The stated appeal period is one month, but the source says Corpus Christi is a public holiday in Bavaria and not Berlin, producing different filing deadlines. Meyer reports that one model placed the Berlin deadline a day late and another gave a deadline for appeal grounds a month late. The example is a test scenario, not a report of actual litigants.
The results do not establish how frequently these errors occur in everyday legal work. They do show why users should treat model answers as a starting point, not a substitute for checking the underlying authority. Meyer advises readers not to rely on a chatbot for deadlines and to consult a lawyer when money, a deadline or contact with an authority is at stake.
Popular right now
Dyson V15 Detect Cordless Vacuum
KitchenAid Artisan 5-Quart Tilt-Head Stand Mixer, Almond Cream
Shark HydroVac MessMaster Wet/Dry Vacuum & Mop Combo (AW261)
As an affiliate, we earn on qualifying purchases.
How Lexassay Tests Legal Models
Lexassay is operated by Thorsten Meyer AI. The test examines what models produce without web search or other tools, rather than their performance when supplied with documents or connected to legal databases. This distinction matters: a lawyer-facing system with source documents available is being tested under different conditions from a plain chatbot asked to answer from its model knowledge.
Meyer describes the results as a spot check, not a benchmark in the strict sense: each model and task receives one answer, so small score differences may reflect variation rather than a stable ranking. The published article also compares Lexassay with Vals AI’s legal-agent benchmark, which supplies documents and ranks some of the same models differently. That comparison underscores that benchmark results depend on what a system is asked to do and what resources it can use.
The operator disclosed a potential conflict of interest: Thorsten Meyer AI runs Lexassay, and several Claude agents graded answers blind, while Claude Opus 5.5 ranks first in the reported results. Meyer says the method page sets out the relationship and that public answers, gold answers and citation checks are available for inspection. The evaluation’s claims and scores come from its operator, not an independent replication described in the supplied material.
“optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution”
— Z.ai, as quoted in its developer documentation
What the Scores Cannot Establish
The findings are operator-reported results; the supplied material does not describe an independent audit or replication. A single answer per model and task cannot show how consistently a system would perform across repeated attempts, different prompts or real client files. The test also excludes internet access and tools, so its ranking should not be read as a ranking of every legal AI product in practical use.
The reported 2,644 real and 804 invented or misattributed citations are totals across published runs. The source does not give a per-model or per-task breakdown for those aggregate figures in the material provided here, so they should not be treated as an error rate for any one model. It is also unclear how the reported results would change with different grading criteria, updated model versions or access to verified legal databases.
Meyer disclosed that Claude agents graded answers and that a Claude model ranked first. Blind grading and publication of answer materials may help readers examine the process, but the disclosed relationship remains relevant when interpreting the rankings. The underlying questions and evaluation rules are described by the operator; the source does not establish that other reviewers reached the same scores.
Review the Methods and Sources
Readers can consult Lexassay’s method page and published answer materials, including public answers, gold answers and citation checks, to examine how the scores were assigned. Five tasks remain hidden, according to the operator, to reduce the chance that future models simply reproduce published answers.
The source does not announce a date for a follow-up test or independent review. Further results would be needed to determine whether model performance is repeatable, how it changes across versions and whether access to legal documents or search tools reduces citation errors. Until then, the results should be read as a limited comparison under Lexassay’s stated conditions, not as a guarantee about a model’s suitability for legal work.
Key Questions
What did Lexassay test?
It tested 42 AI setups on 15 questions involving German, US and EU law. The questions cover topics including deadlines, legal changes, court decisions and false premises.
Which model scored highest?
According to the operator’s results, Claude Opus 5.5 scored 79 out of 100, ahead of GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The test’s operator disclosed that Claude agents graded answers.
How reliable are these rankings?
Meyer calls Lexassay a spot check rather than a strict benchmark, because each model-task combination received one answer. The test also excludes web access and tools, so it does not establish how models perform in every legal workflow.
What did the citation review find?
Across published runs, the operator reports 2,644 real citations and 804 citations that were invented or misattributed. The source material does not provide an error rate for each model in those aggregate totals.
Should I use a chatbot to calculate a legal deadline?
The article advises readers not to rely on a chatbot for deadlines. It recommends checking cited laws and rulings against their official sources and contacting a lawyer when a deadline, money or an authority is involved.
Source: ThorstenMeyerAI.com
