🔍 Read the full analysis: Z.ai’s Legal AI Ranks Near The Top: A Look Beyond The 1-In-5 Pass Rate on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s archived Harvey Legal Agent Benchmark leaderboard, ranking seventh of 81 models. That all-or-nothing task score sits alongside a 92.5% pass rate on individual criteria, but neither result establishes that the model can be used without professional review.
Z.ai’s GLM-5.3-FlashX-Legal placed seventh among 81 models on Vals AI’s archived leaderboard for Harvey’s Legal Agent Benchmark, scoring 19.17% on all-or-nothing task completion. The result means it satisfied every rubric criterion on roughly one in five tested tasks—not that it earned only that share of all available credit: Vals also reported a 92.5% pass rate on individual criteria.
The leaderboard score was produced using Harvey’s grading protocol, according to Vals AI’s archived results. Each task is counted as complete only when every required criterion passes. A missed deadline, citation, or other rubric item can therefore cause a task to fail even if the model gets many other parts right. The score is an average of two AI judges’ task pass rates, Vals says.
The archived Version 1 leaderboard, updated October 18, listed GLM Legal at 19.17% and $0.15 per test. Z.ai’s documentation rounds the result to 19.2% and says it is a 12.5-percentage-point improvement over the base model. The same leaderboard listed the base GLM 5.3 Flash at 6.67%, making the stated point increase consistent with those figures. The listed per-test figure is a benchmark cost, distinct from Z.ai’s API pricing.
Z.ai describes the model as optimized for legal research, document understanding, contract review, drafting and task execution. Its documentation says it underwent legal-domain mid-training and post-training, including reinforcement learning in tens of thousands of task environments modeled on legal workflows. Z.ai recommends using it as an agent connected to services providing laws, cases and company data.
The Gap Between Criteria and Tasks
The contrast between 92.5% of criteria passed and 19.17% of tasks completed shows how sharply an all-or-nothing measure can reduce a score when work products contain many requirements. A model may satisfy most individual checks yet still fail a task because one required element is missing or wrong. For legal teams, the distinction matters: a high rate across isolated criteria does not establish that a full filing, contract review or research memo is ready to use.
The benchmark offers a standardized way to compare legal agents, but it does not establish performance in every firm’s workflow or make outputs reliable without review. Vals listed GLM Legal at a lower test cost than several higher-ranked systems, including $0.15 per test compared with $2.09 for the top-listed model. That is a benchmark cost comparison, not a full estimate of deployment expenses, which can also include data access, integrations and human review.
For buyers, the practical questions extend beyond rank: how the system handles confidential client material, what data is retained, which laws and case sources it consults, and who checks citations and conclusions. The supplied material does not establish answers to those procurement and governance questions.
Popular right now
LG 85-Inch QNED84B Mini LED 4K Smart TV (2026)
Ninja Foodi Air Fryer, 8 Qt DualZone, 6-in-1, Grey, DZ201
Dyson W1 PencilWash Cordless Wet and Dry Floor Cleaner with Dock
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark on May 6, 2026. The open-source benchmark contains 1,250 tasks across 24 legal practice areas, with more than 75,000 expert-written rubric criteria, according to Harvey. Tasks ask an agent to produce work product using materials from a client matter. Criteria can cover facts, conclusions, citations, recommendations, deadlines, amounts and formatting.
Harvey calls its approach all-pass grading: every criterion must pass for a task to count as complete. Vals says its public leaderboard uses a private holdout set of 120 tasks that mirrors the public benchmark’s mix, with internet access disabled. Two AI judges assess each submission, and Vals averages their task pass rates. Harvey’s earlier baseline, published May 26, reported that the best overall all-pass rate among the frontier models it tested was 7.1%. That was a separate run and should not be treated as a directly comparable result.
Z.ai’s AutoClaw blog announcement, dated October 13, 2026, opened a beta by application and named Vals as the evaluator. Vals lists the model release as October 12. The supplied source says Z.ai’s documentation page itself carries no date. Z.ai’s pricing page lists API rates of $0.37 per million input tokens and $1.25 per million output tokens.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark grading rule
What the Score Cannot Establish
The result is a benchmark measurement, not proof of reliable performance across real legal matters. The source material does not provide the full underlying submissions or a breakdown showing which criteria GLM Legal missed on each task. It also does not establish whether the private holdout reflects the mix of work handled by a particular firm or jurisdiction.
Z.ai claims state-of-the-art performance among open-source models in this evaluation, but the source material says the company did not identify its comparison set and that published weights for the legal model could not be found. The claim should therefore remain attributed to Z.ai; the leaderboard ranking alone does not establish that the model is open source or leads every relevant comparison group.
The source author reports a four-task spot check in which the legal model cited rulings the author could not find. The author also says OpenAI and Claude models in that small check did not invent rulings, while two Claude models cited a real ruling for the opposite of its holding. This is an informal four-task check, not a controlled evaluation, and cannot establish broad comparative rates of citation error. The supplied material also does not answer how client data is processed or what legal and operational safeguards apply.
Beta Access and Firm Review
Z.ai’s October 13 AutoClaw post said access to the beta would be available by application. The supplied material does not give an application timeline, the number of participants, or a date for a wider release. The company’s documentation recommends connecting the model to external services for legal materials and company data, making the quality and permissions of those sources relevant to any deployment.
For legal organizations considering the system, the next useful evidence would include more detailed results by practice area and task type, citation checks on larger samples, and clear documentation about data handling. Until those details are available, the leaderboard supports a narrow conclusion: GLM Legal performed strongly enough to rank near the top of this particular list, while still failing at least one required criterion on most benchmark tasks.
Key Questions
What does GLM Legal’s 19.17% benchmark score mean?
It is the share of benchmark tasks on which the model passed every required rubric criterion, averaged across two AI judges, according to Vals AI’s description. A task with even one missed criterion does not count as complete.
How can the model pass 92.5% of criteria but complete only 19.17% of tasks?
The two figures use different units. The 92.5% figure counts individual criteria, while 19.17% counts whole tasks that passed every criterion. Missing one requirement can fail an entire task under Harvey’s grading rule.
Where did GLM Legal rank?
Vals AI’s archived Version 1 leaderboard, updated October 18, 2026, listed it seventh of 81 models at 19.17%. The ranking applies to that leaderboard and version, not every legal AI evaluation.
Does this result show the model is ready to give legal advice?
No. The score is a benchmark result and does not establish that outputs are safe or suitable for a real matter without professional review. The source material describes Z.ai’s output as drafts for legal professionals and reports concerns from a small, informal citation spot check.
Is GLM Legal open source?
Z.ai claims state-of-the-art performance among open-source models in this evaluation, but the supplied source says no published weights for the legal model were found and that Z.ai did not name its comparison set. The available information does not independently establish the model’s open-source status.
Source: ThorstenMeyerAI.com
