🔍 Read the full analysis: Can AI Handle Legal Questions? We Tested 42 Setups on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Thorsten Meyer AI’s Lexassay test compared 42 AI setups on 15 legal questions involving German, U.S. and EU law. The published results show leading models performed better than a law-focused model, but citation errors remained widespread; the test operator says its findings are a spot check, not a definitive benchmark.
Thorsten Meyer AI published results on November 4 from a Lexassay test of 42 AI setups answering 15 legal questions, reporting large differences in scores and frequent citation problems. The test operator says the results are a spot check, not a definitive benchmark, and discloses that its grading used Claude agents while a Claude model placed first.
Lexassay’s reported scores put Claude Opus 5.5 at 79 out of 100, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The report says the leading model fully met every criterion on only eight of the ten public tasks. The model marketed by Z.ai for legal work, GLM-5.3 FlashX Legal, scored 8 and fully solved none of the 15 tasks; it returned no answer on two. Its base model scored 3, according to the report.
The operator also reports that citation checks across published runs found 2,644 citations that were real and 804 that were invented or did not support the claim made. Lexassay deducts points for invented or misattributed citations. A correct legal reference matters because a response can sound plausible while relying on a case or statute that does not establish the stated rule.
One example in the report concerns German appeal deadlines. For two judgments served on April 27, 2027, it says the one-month appeal deadline falls on May 28 in Munich because May 27 is a public holiday in Bavaria, while the Berlin deadline remains May 27. The written grounds are due June 28 in both cases. Meyer says some tested models gave a deadline a day late or placed the grounds deadline a month late. These are examples from the test, not legal advice for an individual case.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal Errors Can Affect Deadlines
The results highlight a practical risk for people using general-purpose chatbots to understand legal rights: a confident answer can still contain a wrong date or unsupported authority. In a court case, a missed filing deadline can have serious consequences. The example also shows how jurisdiction-specific details, such as a regional public holiday, can change the answer even when the underlying time limit is the same.
The test does not establish how AI performs across all legal work or how a lawyer using research tools would perform. It evaluates model responses made without internet access or external tools. Its findings are most directly relevant to users who ask a standalone chatbot for legal information and may not independently verify the result.
Popular right now
Apple AirPods Max 2 Wireless Over-Ear Headphones – Midnight
Sony BRAVIA XR8B 65-Inch 4K OLED Google TV (K-65XR8B)
De'Longhi Rivelia Fully Automatic Espresso Machine with Milk Frother, White
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay asked each setup the same 15 questions across German, U.S. and EU law: ten public tasks and five hidden ones. The questions covered deadline calculations, legal changes, court decisions and prompts containing false assumptions. According to the method description, the models answered without internet access or tools, and separate effort settings were tested separately.
Answers were graded against reference answers using weighted criteria, and citations were checked against official databases. The operator says each model-task combination received one answer, so small score differences may reflect noise. The hidden-task scores are published, but the answers are not. This structure and the operator’s disclosed role are relevant when interpreting the rankings.
The report compares its findings with a separate Vals AI legal-agent benchmark, which supplies documents to models. Meyer says that benchmark ranks Mistral Large 4 eighth, ahead of Claude Opus 5.5 in 21st and GPT-6 Astra in 24th. Those results are not directly comparable: the tasks and access to source material differ.
““optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.””
— Z.ai, as quoted in the Lexassay report
What the Scores Cannot Establish
The report is produced by Thorsten Meyer AI, which runs Lexassay. It also says Claude agents graded the answers and that a Claude model ranked first. Meyer identifies this as a conflict of interest and says the method page and public answers, reference answers and citation checks allow readers to inspect the grading. The material provided does not include an independent replication or external audit of the scores.
The test’s one-answer-per-task design means it cannot show how consistent a model is across repeated attempts. It also does not test browsing or legal research tools, and the results may not predict performance on a particular jurisdiction, case type or real client matter. The full results for all 42 setups are not detailed in the supplied material, so the reported figures should not be treated as a complete account of every model’s performance.
Checking Results and Legal Advice
Lexassay says its public answers, reference answers and citation checks are available for review, while answers to the five hidden tasks remain unpublished. Readers can compare the stated grading rules with the examples and keep in mind that small ranking differences may be noise. No further test date or independent review is specified in the supplied report.
For a real deadline, contract or letter to a public authority, the operator advises consulting a lawyer rather than relying on a chatbot. If AI is used as a starting point, its guidance should be checked against the relevant legal sources, including the specific law or court ruling it cites.
Key Questions
What did the Lexassay test assess?
It compared 42 AI setups on 15 questions involving German, U.S. and EU law. Models answered without internet access or tools, and answers were graded against reference answers.
Which model scored highest?
The report says Claude Opus 5.5 scored 79 out of 100, ahead of GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The operator discloses that Claude agents graded the answers and that a Claude model ranked first.
What did the test find about legal citations?
Across published runs, Lexassay says it confirmed 2,644 citations as real and found 804 that were invented or did not support the claim made. Those figures are the operator’s reported results.
Does this test prove AI cannot answer legal questions?
No. It is a limited spot check with one answer per model and task, and it does not test internet research or external legal tools. It does identify errors and citation problems in the conditions tested, but it cannot settle how every AI system performs across legal work.
Should I use a chatbot to calculate a legal deadline?
The test operator advises users not to rely on a chatbot for a deadline. Because a missed filing date can affect a case, ask a qualified lawyer about deadlines and other legal decisions specific to your situation.
Source: ThorstenMeyerAI.com
