🔍 Read the full analysis: A Closer Look At Z.ai’s Legal AI Benchmark Results And High Ranking on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s version of Harvey’s Legal Agent Benchmark, ranking seventh among 81 listed models. The score means it completed every requirement on about one in five tasks, although it passed 92.5% of individual rubric criteria; that distinction matters for assessing legal work that may be unusable if even one required detail is wrong.
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria pass rate on Vals AI’s version of Harvey’s Legal Agent Benchmark, ranking seventh among 81 models on the archived leaderboard. The result is a notable showing for Z.ai’s legal-focused model, but the score is not a measure of general accuracy: a task passes only when every required criterion is met.
The leaderboard lists the legal model at $0.15 per test, compared with 6.67% and $0.57 per test for the base GLM-5.3-Flash model. That difference is 12.50 percentage points, matching Z.ai’s stated improvement over its base model. Z.ai describes the legal model as built on GLM-5.3-Flash and developed with legal-domain mid-training and post-training, including reinforcement learning in what the company says were tens of thousands of simulated legal-workflow environments.
The 19.17% score is an all-or-nothing task rate, not the proportion of rubric items answered correctly. Vals AI’s archived listing also reports that GLM Legal met 92.5% of individual criteria. Under this benchmark’s grading rule, a task with even one failed requirement does not count as complete, whether that requirement concerns a citation, deadline, conclusion, or formatting choice.
Z.ai’s developer documentation describes the model as intended for legal research, contract review and drafting, including initial drafts. The company recommends using it as an agent connected to services supplying laws, cases and company information. Z.ai’s October 13 AutoClaw blog post announced a beta available by application; Vals AI lists the model’s release date as October 12. The developer documentation page itself does not carry a date.
What the Score Means for Legal Work
The ranking gives law firms and legal technology buyers an independent benchmark result to compare with Z.ai’s promotional description. It also makes clear why the headline percentage needs context: 19.17% of tasks passed completely, while the model met most individual rubric criteria. Those figures describe different things. Strong partial performance may be useful for drafting or research assistance, but it does not establish that an output is ready for a lawyer or client without review.
That distinction matters in legal work because a missing deadline, incorrect citation or overlooked risk can undermine a document even when the rest is well written. Harvey’s benchmark uses all-pass grading on the basis that a work product can be materially incomplete if it misses a required issue. The score therefore tests end-to-end completion against a defined rubric, not whether a model can produce plausible legal prose or provide partial assistance.
Cost is another point of comparison, not a full procurement verdict. Vals lists GLM Legal at $0.15 per test, lower than the listed figures for higher-ranked entries such as Muse Spark 1.2 at $2.09 and Gemini 4 Argon at $10.55. Those prices are leaderboard test costs; they do not by themselves capture implementation, human review, data governance or the cost of errors. Firms considering deployment still need to examine how client information is handled and who verifies authorities and citations.
Popular right now
Dyson Car+Boat Cordless Handheld Vacuum
TCL 65-Inch QM6K QLED Mini-LED 4K HDR Smart Google TV, 144Hz
iRobot Roomba 105 Robot Vacuum with LiDAR Navigation
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released the Legal Agent Benchmark (LAB) as open source on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas. Each task gives an agent an instruction and client-matter materials, then asks it to produce a work product for review. Harvey says the tasks are graded against more than 75,000 expert-written, binary rubric criteria, covering items such as facts, conclusions, citations, recommendations, deadlines, dollar amounts and formatting.
The all-pass rule is central to interpreting the result. Harvey says a task counts as complete only when every criterion passes; its example is a report that identifies eight of ten risks, which still falls short of a complete work product. Vals AI’s public leaderboard applies Harvey’s grading protocol to a 120-task private holdout set that mirrors the public benchmark’s mix, with internet access disabled. Two AI judges grade each submission, and Vals averages their task pass rates.
In Z.ai’s statement, the model’s “state-of-the-art performance among open-source models” is a company claim. The archived Vals leaderboard establishes the seventh-place ranking among the models it lists, but the source material does not identify a comparison set that would establish the broader open-source claim. The material also reports that the legal model’s weights were not published, so its status as an open-source model cannot be independently confirmed from the cited information.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model.””
— Z.ai, in its model documentation
Limits of the Available Evidence
The leaderboard result does not establish how reliably the model performs across all legal matters or in live use. It reflects a 120-task private test set, a particular version of the benchmark and its grading process. Vals reports that the individual-criterion rate was 92.5%, but that does not reveal which types of requirements were most often missed or how consequential those misses were.
The source material also describes a small, four-task spot check in which the author said the legal model cited rulings that could not be found. That check was not a benchmark and cannot establish a general error rate. In the same spot check, the author reported that two Claude models cited a real ruling for the opposite of its holding, while OpenAI and Claude models cited no nonexistent rulings. These observations should not be treated as a broad comparison among vendors.
It remains unclear what independent evidence supports Z.ai’s “state of the art among open-source models” wording, since the company did not name the comparison set in the supplied material and the legal model’s weights were not found there. The source also does not specify the model’s data-processing arrangements for client information. Z.ai’s pages describe outputs as drafts for legal professionals, but the reviewed pages did not carry a “not legal advice” notice.
What Buyers Should Verify
Z.ai has opened a beta by application, but the supplied material gives no date for general availability or a next benchmark update. Prospective users can look for further results on Vals AI’s leaderboard and for clarification from Z.ai about the model’s release status, the basis for its open-source comparison claim, and how its legal workflow handles data and citations.
For law firms, the immediate next step is evaluation under their own review and data rules rather than treating the leaderboard rank as a deployment decision. That means checking outputs against authoritative legal sources, defining who reviews drafts before they leave the firm, and asking how client data is processed and retained. The benchmark supplies a comparison point; it does not answer those operational questions.
Key Questions
What did Z.ai’s legal model score?
GLM-5.3-FlashX-Legal scored 19.17% on the all-criteria task pass rate in Vals AI’s archived version of Harvey’s Legal Agent Benchmark. Z.ai rounded the result to 19.2% in its documentation.
Why is the 19.17% score lower than the criteria pass rate?
The task score counts a task as complete only if every required rubric criterion passes. Vals separately reports that GLM Legal met 92.5% of individual criteria, but a task with even one failed criterion does not count as a full pass.
Where did the model rank?
The archived Vals AI leaderboard placed GLM Legal seventh among 81 models. That rank applies to the leaderboard snapshot updated October 18, 2026, not necessarily to later versions or results.
Did the model improve on Z.ai’s base model?
Yes, according to the archived leaderboard figures: GLM Legal scored 19.17%, while GLM-5.3-Flash scored 6.67%. The difference is 12.50 percentage points, consistent with Z.ai’s reported improvement.
Does the benchmark show the model is ready to provide legal advice?
No. The benchmark measures performance on a defined set of tasks and does not establish readiness for unsupervised legal work. Z.ai describes outputs as drafts for legal professionals; firms would still need to verify citations, conclusions, data handling and review procedures.
Source: ThorstenMeyerAI.com
