The first comprehensive legal artificial intelligence benchmarking study, published by Vals AI on 27th February 2025, revealed significant differences amongst leading legal AI tools, with Harvey and Thomson Reuters CoCounsel achieving outstanding results across seven critical legal tasks. The study compared four AI tools—Harvey, CoCounsel, Vincent AI (vLex) and Oliver (Vecflow)—with a control group of solicitors, whilst LexisNexis withdrew prior to publication.
The Vals AI study evaluated in detail the performance of legal AI tools across various task areas. (Not all AI tools participated in every test; for example, Harvey Assistant was involved in six tasks, while CoCounsel participated in four tasks.) The Harvey Assistant tool was the most successful: it performed best in five tested task areas. It particularly excelled in two —when answering questions related to documents (94.8% accuracy) and arranging legal events chronologically (80.2% accuracy). The second most successful tool was Thomson Reuters CoCounsel, which participated in four task areas and performed with an average accuracy of 79.5%. CoCounsel also performed exceptionally well in answering document-related questions (89.6% accuracy) and summarising documents (77.2% accuracy). The difference in speed between AI tools and solicitors was remarkable. According to the report, the fastest AI responded up to 80 times faster than solicitors, whilst even the slowest AI operated six times faster. Solicitors outperformed AI tools in only two specific areas: researching in the EDGAR database (with 70.1% accuracy) and proofreading contracts (with 79.7% accuracy).
The significance of the Vals Legal AI Report is further enhanced by its creation in collaboration with ten leading American and British law firms, using real legal questions and tasks for testing. This milestone report provides an objective and transparent framework for legal professionals to evaluate legal AI tools, as the tests demonstrated that in specific tasks—particularly in document analysis, information retrieval, and data extraction—AI tools already surpass human performance whilst offering significantly faster response times.
Sources:
1.
artificiallawyer
2.
Bob Ambrogi
