JudgeGPT Experiment Boosts Case Closures in Pakistan Courts

Judges around the world have made headlines for illicitly using generative AI in their work. But in Pakistan, a large-scale trial of a specially designed AI tool for judges found the technology—together with appropriate training–boosted the number of cases resolved by 6.3 percent with no obvious drop in the quality of judgments.

With a backlog of 2.26 million cases and fewer than two judges per 100,000 people—compared to 22 in the EU and eight in Brazil—Pakistan’s judiciary was in sore need of help. So, in consultation with the judiciary, economist Sultan Mehmood, of the New Economic School in Moscow, and collaborators tested whether AI could ease the burden.

They built a custom tool combining OpenAI’s GPT-4 large language model (LLM) with a knowledge base of nearly 130,000 Pakistani judicial opinions and statutes, to help judges with legal research and drafting judgments. They began offering the tool in 2024 to 1,559 trial judges—roughly half the country’s justices.

“We do find an increase in cases resolved, and we don’t find any corresponding decrease in decision quality,” Mehmood says.

First of its kind

“It’s pretty amazing that he’s able to pull this off,” says David Autor, an economics professor at MIT. “It’s not easy to do large-scale field experiments in civil service, but especially where the stakes are so high.” The 6.3 percent productivity boost is not overwhelming, he says, but it’s credible and likely to improve as the tool is more widely used.

AI tools for judges are already being rolled out in Brazil and India, and prominent U.S. law professor Eric Posner has compared LLM judgments to human judgments in a single case study. But until now, there has been no major independent assessment of ongoing judicial use of AI. The new study focused on Pakistan’s trial courts; Mehmood says judges there were enthusiastic from the start.

“They were more techno-optimist than we were,” he says. “The delays are so huge, this is something which they thought was worth trying anyway to reduce people’s suffering.”

Some judges were also already using AI chatbots, Mehmood says, but commercial offerings performed poorly on Pakistani legal queries, frequently hallucinating case law. So the team built a tool tailored to the Pakistani context, called JudgeGPT.

They used retrieval-augmented generation (RAG), which allowed the model to query a database of 128,292 Pakistani judicial opinions and 943 statutes. Responses included footnotes linking to cases and laws.

“It turns out that actually the way to fix [hallucinations] isn’t just more intelligent models,” says study coauthor Elliott Ash, an associate professor of law, economics, and data science at ETH Zurich in Switzerland. “It’s to attach the models to a tool that can do a search and verify the sources.” However, the researchers do not report hallucination rates.

The team also put 1,197 judges through six 90-minute Zoom training sessions where Ash…