How LLMs are Reshaping the Legal Industry

Aviv Gelfand

January 1, 2025

The launch of ChatGPT’s conversational interface in November 2022 marked a turning point in AI history.

While this AI interface, known as a large language model (LLM), had already gained some attention by data scientists with earlier versions of GPT, such as GPT-2 in 2019 and GPT-3 in 2020, 2022 was the first time ChatGPT made this technology accessible to the general public. This launch also catalyzed a shift, with tech companies starting to integrate AI into daily operations on a large scale.

For data scientists like me, these models have increased the scale of tasks that can now be performed accurately and automatically. This transformation allows my wildest dreams of automating complex inference to come true through my keyboard.

AI in the legal industry

AI is transforming the legal industry, allowing attorneys and legal professionals to streamline processes and handle larger volumes of work more efficiently and at an accelerated pace.

In my current role at Darrow, I work in research and development (R&D), creating tailored software for our Legal Data (LD) team. To give you some context, the LD team consists entirely of experienced attorneys who also excel as data intelligence experts (Interestingly, many of them previously served in elite technology and military intelligence units, bringing a unique skill set to this role that bridges technology and the legal domain).

Our team of data scientists work closely with our LD team to deliver an incredibly powerful legal intelligence platform that we use to dive deep into the web to uncover legal risks early. The evidence we gather forms a solid foundation for legal action towards fixing wrongdoings.

To do so, we harness a sophisticated array of AI technologies and advanced algorithms. From complex similarity search through high dimensional clustering methods to custom-built language models, we employ solutions that push the boundaries of what's possible with textual information from a wide range of sources.

But in this article, I'll focus on one particularly elegant solution: our strategic use of advanced LLMs from industry-leading providers to efficiently process data with high reasoning capabilities.

How do LLMs work?

Think of LLMs as incredibly sophisticated pattern recognition systems that have been trained on vast amounts of information. During training, a model learns not just individual words, but the intricate relationships between words, concepts, and context, much like how a human learns language, but at a massive scale.

The key difference from earlier AI systems is that LLMs understand context through what's called attention mechanisms, meaning that they can focus on the most relevant parts of the input text and draw connections between them.

Image 1: This image visualizes how attention mechanisms in LLMs work. It highlights the connections and relationships between words in a sentence, allowing the model to focus on the most relevant parts of the text to understand context and meaning. These mechanisms enable LLMs to process and generate coherent, context-aware responses (Source: *3Blue1Brown*)

For example, consider how you analyze a legal document. You don't just scan for specific terms, but understand the context, identify relevant sections, draw connections to precedents, and form conclusions based on their expertise.

When reading "The court dismissed the case because the statute of limitations expired," your brain automatically connects "dismissed" with "statute of limitations" to understand the causation. LLMs use this same logic to make these same connections, weighing the relevance of each word to every other word in the text.

Image 2: Word embeddings in action: This visualization demonstrates how large LLMs represent relationships between words in a vector space. For example, the difference between 'daughter' and 'son' is analogous to the difference between 'woman' and 'man,' showcasing how LLMs capture semantic relationships and context through mathematical representations (Source: *3Blue1Brown*)

But here’s the difference: LLMs can process enormous volumes of data at incredible speed.

The large, in large language model, refers to both the amount of training data (hundreds of billions of words) and the model's complexity (hundreds of billions of parameters or decision points). This scale allows LLMs to recognize subtle patterns and generate coherent, contextually appropriate responses to almost any prompt.

Image 3: LLMs have billions of parameters, making it hard to understand what each is responsible for. However, some research done in this field distinguishes parameters or groups of parameters that can be accountable for certain semantic relationships, as seen in the table (Source: Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). *Efficient estimation of word representations in vector space*)

When you input a question or task into an LLM, the model analyzes the prompt against its vast training data to understand what you’re asking. It then generates a response by predicting the most appropriate sequence based on its understanding of language patterns and the context provided. This process happens in milliseconds, making LLMs incredibly efficient tools for processing and analyzing information at scale.

Using LLMs to detect legal violations at scale

What makes modern LLMs particularly powerful for legal applications is their ability to understand domain-specific language and concepts. When properly prompted, they can identify legal issues, analyze documents for relevant information, and even help draft preliminary legal documents, all while maintaining the precise language and logical reasoning that legal work demands.

At Darrow, we use pre-trained LLMs to detect egregious legal violations by analyzing and processing vast amounts of long documents quickly and at scale. Our technology allows the team to process massive amounts of information, identify patterns, cluster related data, and draw actionable conclusions.

Here’s how we use LLMs for some of the simpler tasks in our workflows:

We carefully craft queries with the appropriate context so the models we use can accurately identify patterns, assess evidence, and prioritize potential cases with the strongest legal foundations.

The LLM evaluates the data and ranks potential legal risks, prioritizing those with the most substantial evidence for building bulletproof cases. This allows us to maximize efficiency while achieving a significant scale of coverage (we data geeks call it recall), ensuring we focus our resources to uncover potential legal violations with speed and precision.

We send structured questions tailored to the specific information we need, and the model answers each piece of data we analyze. This approach significantly accelerates the detection of legal risk and evidence collection.

Each LLM provider also has several models, each offering its own advantages and pricing for usage. So when we write queries, we need to specify which LLM model we want to use.

At Darrow, our favorite LLMs include but are not limited to:

GPT: Created by OpenAI, the leading player and disruptor in the LLM arena
Claude: Created by Anthropic, the second-leading LLM
Open Source LLMs like Llama and Mixtral

But do LLMs pose risks?

While LLMs are powerful and incredibly useful tools and have already begun transforming how legal professionals operate, it’s also crucial to be aware of the associated risks. An overreliance on AI can be detrimental, and human oversight in the practice of law is essential to maintaining accuracy, upholding ethical standards, and exercising sound judgment in decision-making.

As Katherine Crowley, Legal Director, PDL at Womble Bond Dickinson, says:

“AI is clearly set to disrupt the legal world, but remember that you still need to shop for your humans.”

Incorporating LLMs into legal work can pose possible data privacy and confidentiality issues, as inputting case-specific details into AI tools can expose sensitive client information to unauthorized access or breaches. Of course, all firms should have cybersecurity measures like encryption, secure access protocols, and data monitoring in place, but it’s important to select LLMs with strong reputations and transparency in their decision-making processes.

One of our primary concerns in the R&D world is that AI-generated outputs can produce inaccuracies and hallucinations, which is when a model generates information or responses that are false, misleading, or entirely fabricated, despite sounding plausible or accurate. This can lead to unreliable legal analyses or arguments if not thoroughly vetted.

Here’s a noteworthy example:

In June 2023, attorneys Steven A. Schwartz and Peter LoDuca were representing a plaintiff in a personal injury case. They relied on ChatGPT for legal research and the tool provided fabricated case citations, which were then submitted in court filings.

Judge Kevin Castel of the Southern District of New York imposed a $5,000 fine on the attorneys after the errors were flagged by opposing counsel. The attorneys admitted they had not verified the citations, highlighting a critical failure of due diligence. Judge Castel called the incident “an unprecedented circumstance” and stressed the importance of human oversight in using AI tools.

Alignment methodologies are the techniques used to ensure LLMs adhere to human values, legal standards, and factual accuracy. For all of us working in the legal domain, following these practices is essential. This case serves as a cautionary tale for both attorneys and data scientists about the dangers of unvetted reliance on AI-generated content in legal proceedings.

We take these risks seriously at Darrow

Our team ensures the safe and responsible use of LLMs by maintaining strict protocols for data handling and output validation. Additionally, we do not input sensitive client-specific information into our models at all, mitigating risks of data breaches or misuse altogether.

Human oversight is central. Our LD team, comprised of experienced attorneys and data intelligence experts, only investigates publicly available sources when analyzing and identifying potential legal violations.

While our AI tools assist in sorting, filtering, and analyzing this data, the LD team carefully reviews and interprets all AI outputs. This ensures that any insights or conclusions drawn by our models are accurate and that all cases are built on sound evidence and reliable legal foundations.

‍

_______________________

This might interest you:

‍

Partner with Darrow to grow your practice