Beyond the Hype: the reality and potential of AI in law

Introduction

Artificial Intelligence (AI) has changed the way lawyers work. The future is here. So, does it live up to the promise? As former corporate lawyers and Founders of Deeligence, an AI-powered due diligence tool, we’re uniquely positioned to offer our thoughts on the topic. We’ll explore current trends in law firm AI adoption, the transformative potential of AI in our industry, its challenges and limitations, and the broader implications for jobs.  

As process-experts, for us the starting point for thinking about AI must always be – what is the problem to be solved? What is the cost or revenue impact on a business? And how often is it faced? Assuming the answer is high impact and business critical, it is then time to assess process and tech solutions and see if AI is the best tool for the job. For vendors like Deeligence, we only considered layering AI into our tool where it made sense, is highly accuracy and directly responded to our customers’ feedback.

State of play  

Who doesn’t want to believe in the AI promise? Increased effectiveness of operations, enhanced accuracy and decision-making capabilities. Sounds like a winner – and lawyers are understandably enthusiastic about this technology. A recent LexisNexis survey found that nearly half of lawyers interviewed in Australia and New Zealand have already used GenAI to perform daily tasks. Nearly all surveyed believed that it would change the future of legal work.  

We speak with firms regularly and there’s a broad split between the larger firms investing heavily today and mid-sized to smaller-sized firms taking more of a ‘wait and see’ approach. But it’s not a neat divide. For larger firms with big innovation budgets there are AI steering committees, working groups, hubs, innovation leads and external management consultants. All broadly seeking the same thing: how to transition lawyers to a new age of practice.  

Most of these innovators have taken a wide lens exploratory approach, rather than a use-case driven one. They’re enabling exploration with in-house safe versions of ChatGPT for lawyers to play with, like Allens Airlie. For those that may be jealous of these tools, these are basically ChatGPT in a data-secure environment. It’s worth remembering that every bit of custom tech built requires an army of change managers, marketers, product managers and software developers enabling it, taking it out of the reach of many. So understandably few firms have gone down this path.

In a similar vein, Microsoft Copilot (powered by the OpenAI technology underlying ChatGPT) has rolled-out at firms such as Lander & Rogers and MinterEllison. While Copilot is evolving fast, for now, results have been mixed. It is strong at summarising meetings and querying them. But not much else (yet). That is some big enterprise-grade prices for a tool that essentially keeps notes.  

It’s better thought of as a general business tool, rather than a legal tool. To truly solve lawyers’ problems, you need deep industry knowledge. This is where vendors like us help firms bridge the time, knowledge and investment gap to take a generic tool to a legal-specific one.

As for in house lawyers, “most can see the benefits of their administrative workload being improved by AI, but these same tools are yet to deliver on the actual legal work,” shared Katrina Gowans, Australian Co-Chair of the Corporate Legal Operations Consortium. Many are cautious of using GenAI to complete legal research but these tools do point “lawyers in the right direction, drawing their attention to the right section of a piece of legislation.” 

GenAI and legal research

Firms like King & Wood Mallesons are using AI to conduct legal research. They require lawyers to undergo mandatory training to understand the limitations of these tools before using them. Training ensures safety and high governance standards, as well as helping lawyers prompt better, which then gets better answers out. With all generative AI, to get good quality legal-type outputs it’s often necessary to have prompts of 1,000+ words, which follow well-worn patterns and have been heavily tested. A lawyer writing an on-the-spot query will always struggle to achieve this without adequate support and training.  

For this reason, prompting is now considered a foundational skill for KWM lawyers, says Chief Executive Partner Michelle Mahoney. The firm offers training including “a prompt library ensuring stability of output from AI responses which has resulted in lots of engagement from lawyers.” These prompts are varied, and lawyers can pick which serves their purposes best – lifting the floor on response quality.  

Firms including Clayton Utz, Maddocks and others are trialling the AI-fuelled offerings from the Lexis / Thomson Reuters oligopoly. These vendors are betting that their legal data business helps them easily surface reliable and relevant information through traditional AI-powered legal search, helping achieve higher quality outputs from the same LLMs while reducing inaccuracies. However, in our experience even with good data the huge variation in legal tasks and audience means prompting issues still make it hard to achieve extremely high accuracy for these general-purpose tools. Some inaccuracies in these tools on over 30% of queries have been exposed by a recent Stanford study, though the companies dispute this assessment.  

For this reason, the way forward is having a heavily tested version of an AI-powered tool that completes specific tasks. Instead of trying to do everything and doing nothing well, diving deep on specific tasks means accuracy and quality can be extremely high.

Realistic expectations of what is achievable for our profession  

Now, shall we pour some cool water on some of the AI hype?

These forward-thinking firms are some of the first to get their hands on the technology so that they can understand and harness its potential. I get that. I love hearing partners get excited and delighted about the changing face of legaltech.  

However, it would be remiss not to mention – that the best way to have long term, sustained innovation in a firm is to deeply understand a problem for lawyers and fix it. It’s that simple. There is such appetite to be first, or not be left ‘behind’ that these larger tools, even the ones by legal information purveyors, do not and cannot meet a lot of lawyers’ needs.  

It’s also essential to have realistic expectations. AI is not and will not be for the foreseeable future a magic solution that can replace human judgment. LLMs are ‘probabilistic’ models, not ‘deterministic’ ones, which means the same prompt asked 10 times will produce 10 different results (sometimes imperceptibly so). It’s worth keeping in mind that so would any lawyer.

Practitioners (for professional obligations as well as common sense ones) must check all AI work, like how they review work of a junior lawyer to ensure it is client ready. AI should be seen as a tool that can augment human capabilities. Not replace them. And by understanding the limitations of AI, lawyers can better integrate it into their practices without falling for the hype.

It’s important that firms don’t use AI as a solution to an unknown problem, but rather determine a specific problem they want to solve and how to leverage AI to best help solve it. Gilbert + Tobin understand this, which is why they famously offered a $20,000 bounty to their staff who submitted great ideas as to how AI could be integrated into everyday work. Gilbert + Tobin Head of Innovation Caryn Sandler knows that only by targeting a repeatable problem with a predictable solution can you achieve true gains with today’s tech. “The best use cases for AI come from our people. We used the AI bounty as a way of encouraging our people to experiment with AI and share their ideas for use in their everyday work,” making it one of the largest collaborative projects at the firm and helping inform their AI strategy and deliverables.  

But do you know what else can also help the way lawyers work? Good process. Great precedents. Diving deep on lawyers’ specific tasks enables simple automation of routine work. This is decidedly less sexy than AI but often just as, if not more effective.  

Issues with US-Centric Models  

A limitation of using established providers eg. Microsoft, ChatGPT is that they’re trained on US-centric data sets and rarely include Australian cases, legislation and contracts. The outputs of these LLMs are inherently constrained as they are more broadly trained on generic internet data like Amazon reviews and Reddit posts (sometimes a bit of a dumpster fire). An LLM can’t know what it hasn’t seen. CoPilot even recently released an info sheet for “Attorneys” (tell me you’re not trained on Australian data without telling me you’re not trained on Australian data…).  

Some folks like Kelvin Legal’s KL3M LLM are trying to solve by using only high quality, legal data sources focusing on case law rather than internet message boards. But this will still be constrained by its US-centric data diet as these models produce what they know best. They know little about Australia. Allens conducted an internal benchmarking study that showed that an out of the box LLM cannot advise on the nuances of Australian law. This should have been wholly unsurprising to everyone.

Unfortunately for Australian lawyers, this problem probably won’t go away. Our legal market doesn’t have the scale to justify spending the tens of millions to train a model like GPT-4 as the expenditure could never be recouped, unlike in Europe and the US. Though this cost to train a foundational model will reduce over time, it won’t be worth doing any time soon.  

You can also ‘fine-tune’ these larger models by showing them lots of relevant Australian legal content. Though this can be useful to give the LLM domain specific expertise, there are limitations here too. Fine-tuning ties you to the state of the art of your chosen model as it is today. Given the transformational leap in quality between generations of LLMs, a fine-tuned model quickly stales. For example, the gulf in quality between OpenAI’s GPT-3 and GPT-4 rendered most fine-tuned specialist models obsolete on the day of launch. This is true for other specialist models like medicine, not just law.  

This jurisdictional bias has been a well-known issue with traditional machine learning (ML) tools over the last decade too, such as those that sped up contract review. Although these tools are powerful, when they are trained on US and UK legal documents they are essentially not fit for purpose in Australia. Most firms we speak to in Australia no longer use these at all.  

In our market, law firms and Australian vendors must rely on ‘traditional’ ML techniques like classifier models, extraction models and retrieval augmented generation (RAG). Don’t let these word-salad names spook you, they’re just fancy ways of saying that we can surface the right piece of context, be that legislation, case or text to ensure that the LLM gives us an Australian-sensible result. For example, Deeligence is achieving 90%+ accuracy in Australian contract review because we’re focussed on the Australian market and keen to ensure we’re actually useful for Aussie lawyers doing legal due diligence.  

Data privacy

It is also worth keeping in mind that ChatGPT is not suitable for client data. It requires transferring and storing data in the US, and the company has default settings enabled that allow it to improve the model based on your inputs. So, firms should be concerned that their client’s sensitive data may become part of how these AI tools produce text for other users.  

There are solutions for this now. Microsoft, AWS and Google Cloud all have data-respectful ways of hosting LLMs. But it’s critical lawyers actually know what’s happening to their client data.  

What are good legaltech vendors doing and what comes next?

The best legal tech vendors take the guess-work out of collaborating with AI, with exceptional product design to make the tech work in a way where lawyer’s just ‘get it.’ That’s why it’s vital to for us to have top-shelf User Experience (UX) designers in house to make legal tech as easy as using your smartphone. Lawyers are time poor and often stereotyped as change resistant. They will not tolerate a bad UX when they’ve grown accustomed to using Apple products in their personal life. The AI must be reliable, and in a way that easily slots into their day-to-day practice.  

Those with deep expertise are spending tens of thousands of hours ensuring that AI-powered tools are producing extremely accurate outputs. US based law firm, Wilson Sonsini’s Cloud Services Agreement LLM collaborated with an ML vendor to use its high-quality firm data and legal expertise to achieve over 92% accuracy when marking up a Cloud Services Agreement. This shows how deep use-case by use-case expertise unlocks the true potential of this technology.

The newest frontier is the development of multimodal models (e.g. GPT-4o) that can synthesise text, pictures and audio at the same time! This matters because, unlike the assumption of most LLMs of today, not all data is easily reduced to text. To illustrate this power, let’s consider a high-volume motor vehicle insurance claims practice. The next generation of multimodal AI-powered tools could integrate the written case documentation, the video and sound of a car accident and pictures of the resulting damage to generate holistic advice on whether to settle a claim. That this underlying technology is available to anyone with a login and credit card was unthinkable even 12 months ago.

Impact on Jobs

The integration of AI into the legal profession will undoubtedly impact jobs. While some routine tasks may become automated, this does not necessarily mean a reduction in jobs. Instead, the nature of legal work will evolve. Lawyers will need to develop new skills to work effectively alongside AI technologies.

Do you remember being a junior lawyer and being asked to keep notes at a meeting? Now, AI tools can do this task, often to a better quality. Katie Higgins of Law Squared, a firm with a tech-focused approach, queries whether this means juniors should wave goodbye to “mundane tasks and can instead focus on the more strategic parts of a meeting”, or whether this creates a gap in their learning and a risk they are missing core developmental skills. She’s experimenting with a ringfenced version of GPT trained on LawSquared’s own precedents and some legislation to help draft legal advice. But knows that clients expect any output to be validated by their lawyers first.  

“The transformational changes that AI brings shouldn’t be feared”, says Lander & Rogers’ Chief Executive Partner, Genevieve Collins. Collins predicts a shift in workforce roles, and that “firms that don’t embrace AI will struggle to stay relevant and competitive…including rethinking pricing”. Firms won’t be able to recover fees for their intellectual input in the same way where it could easily have been done using technology.  

We will continue to see growing demand for professionals who can manage and interpret AI outputs, ensuring they are used correctly and ethically. Indeed, AI could become a new battleground for the legal talent war. If you’ve avoided doing a mind-numbing monotonous task by using AI before at your old firm, we’re guessing you’re probably not going to tolerate doing it old-school at your new employer.

Conclusion  

AI presents exciting opportunities for the legal profession. By embracing these technologies while remaining mindful of their limitations, lawyers can enhance their practice, offer better services to their clients, and navigate the changing landscape of the legal industry.  

‍