Chat GPT is putting the future of grad lawyers under the microscope

ChatGPT, the chat bot that is a social media sensation given its ability to do all manner of human-esque things like explain complex topics, write essays, come up with jokes, and even pen catchy songs, has been touted as a threat to many white-collar jobs, including lawyers.

That’s all the more so post the latest iteration of the bot, GPT-4, developed this year by Microsoft-backed OpenAI, which is the most sophisticated version of the tool to date.

It has generated much chatter, with some pundits going so far as to suggest it may signal a future where AI has ascendancy over not just lawyers, but all human content producers.

For lawyers, the threat from the bot, at present, appears most keen at the junior end of the industry, given the tool’s ability to process huge amounts of data, including legal data.

GPT-4 builds its text-generating models by using machine-learning algorithms to process massive loads of information from not just basic web sources like books, news articles, Wikipedia pages and websites, but also specialist legal sources.

The result, in layman’s terms, it what the University of Sydney has called “in general, genuinely informative and helpful explanations on a broad range of topics”.

Legal graduates, perhaps predictably, along with the likes of paralegals and legal researchers are some of the profession’s workers likely to be in the firing line of the fast-emerging chat bot technology, according to business futurist and strategist, Morris Misel.

“Law graduates have traditionally been brought in at the lower end to do the grunt work, the exploratory work, read through contracts and find errors and (do) legal research. But that to some degree will be done by this type of technology,” says Melbourne-based Misel, who assists large companies of the likes of IBM, Allianz and NAB to resolve issues that may impact their workforces of the future.

What’s happening is not new, he suggests.

“There has been technology that has been encroaching on the legal industry forever, and in the last 25 years,” he says, pointing to the loss of many stenographers and receptionists as “technology has taken up their role”.

It is easy to see why some young lawyers may see their futures looking bleak due to a tool like ChatGPT.

For instance, since its release in November 2022, the software has shown it can pass law school exams, with law professors at the United States’ University of Minnesota deploying it to sit law exams, then grading them blindly alongside actual students’ tests.

The result, according to the researchers, is that the digital tool “consistently achieved low but passing grades” across 95 multiple choice questions and 12 essay questions, which, if replicated across an entire degree, would be good enough to graduate.

Even more impressive, perhaps, is the bot’s recent effort on the US bar exam, the tough two-day test aspiring lawyers must pass to practise law in that country.

In an experiment conducted by two law professors and two staff of legal tech company Casetext, GPT-4 scored 297 on the bar exam which, according to the researchers, places it in the 90th percentile of human test takers, a good enough result to be admitted to practise law in most US states.

“These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society,” the researchers wrote in the paper, published in March in the electronic journal Social Science Research Network.

Despite its strong showing on some legal tests, Lyria Bennett Moses, director of UNSW’s Allens Hub for Technology, Law and Innovation, says Chat GPT is unlikely to replace grad lawyers – in the short term. Her view is that, right now, its responses are “too ridden with errors including fake case references to be used directly in legal advice”.

“While it might prove to be a useful aid in legal research in some contexts, this supports humans in doing their work rather than replacing them,” Bennett Moses tells LSJ, adding that “it is hard to conceive of this technology having similar reliability to graduate lawyers”.

“Historically despite technology increasing law firm efficiency, [such as] word processing rather than typing on carbon paper, research online rather than via citators, changes in the number of legal jobs is more closely correlated with broader market conditions than with technological change,” the law professor says.

Wisdom vs knowledge

Bennett Moses’ opinion aligns with law firm Allens which, in a note published in February, pointed to potential benefits for legal practitioners using the tool, including lowering costs by automating and cutting the need for human input, especially for initial tasks on matters.

ChatGPT, the firm argues, also appears to have capacity to boost productivity and efficiency by rapidly generating responses to simple legal queries and quickly preparing preliminary drafts of contracts and briefs. What’s more, it can swiftly locate relevant information in voluminous sources like case law and statutes, helping to speed up resolution of simple legal queries.

But “being a good lawyer is more than text generated by a clever predictive algorithm,” Bennett Moses says.

“’Knowing stuff’ is often less important than being able to think critically and creatively about a client’s problem, and communicate the advice clearly and empathetically, [and] being able to think strategically through options in litigation or in a transaction or conceiving new business models for delivery of legal services, inter alia.”

Misel concurs. He predicts law grads will shift in their roles from, say, doing legal research, to overseeing the work carried out by a future version of something like ChatGPT.

With that in mind, he urges law students, and others wanting to work in law, to remember: “the reality is that there is a legal industry moving forward, that’s the bottom line”.

According to the futurist, the key skill young legal players must develop, when looking forward to an increasingly AI-tech heavy legal profession, is “wisdom” instead of “knowledge”.

Technology like ChatGPT is fast mastering both mass information gathering, and also turning that information into knowledge, which can then be put into the hands of humans. Where it cannot beat actual lawyers at the moment, he contends, is in the realm of softer skills like nuanced communications and negotiation between competing parties.

“What graduates, and legal professionals, and every professional, will sell from now on is their wisdom,” he says.

“You can put together a whole set of facts, which are brought together by ChatGPT, and that still on paper may seem to be the most sensible thing, but a human looks at it [and] says it’s nonsense and it’s not going to work (in reality).

“So, there’s still this great need for humanity to be involved. When it comes to negotiation in legal its often about wisdom. It is often about ‘how do I move elegantly to get from where I am to where I need to be on behalf of a client?’ Artificial intelligence still can’t dance like that, and won’t be dancing like that for a long time.”

A new reality

Even so, with Harvard Law School predicting that AI may not eliminate the need for lawyers but does “portend the end of lawyering as we know it”, young lawyers entering the workforce will need to ready themselves for the new reality.

As Harvard’s Centre on the Legal Profession points out, law schools face several challenges stemming from AI tools like Chat GPT, including how to accurately assess students now they have easy access to AI help, as well as how to embed the tools into department curricula.

“For example, first-year legal writing classes and clinical programs may need to teach AI document drafting so that future lawyers understand how to use the technology in practice,” the US-based legal centre suggests.

Legal grads, as it also notes, will benefit by becoming familiar with the latest AI technology in the sector given “clients will not want stand-alone lawyers who eschew AI”.

“Conversely, clients with challenging legal matters are unlikely to rely on technology by itself. The future, at least for complex legal issues, will require the use of tech-enhanced lawyers,” the centre says.

Hence, for John Flood, Professor of Law and Society at Griffith University in Brisbane, there’s no use in graduates ignoring what’s already here in AI. The challenge is how to deal with it.

Flood, echoing Bennett Moses, says there’s likely to be a role for young lawyers – up to the junior associate level – in overseeing the work carried out by programs like ChatGPT. Grads, he says, will have to keep in mind that this AI is like a “sophisticated Google or Bing”.

“It will put things into context for you (and) whereas Google will tell you a whole list of things, it won’t tell you a lot about them, whereas chat GPT will tell you about those things and give you a context in that way,” the academic tells LSJ.

“That’s useful as a research tool as long as you realise ‘rubbish in rubbish out’ – you’ve always got to check it.”

Along with replacing lots of “standard and routine” work commonly done by grads, AI will also probably reshape the structure of firms, leading to different opportunities for juniors, he suggests, predicting a “tech division” to be a key part of legal companies in years ahead.

Widespread impact

Additionally, some areas, like family law, will be more taken up by AI, than others. Flood points to AI-backed legal software already available to couples going through divorces, which is already reducing the need for some work traditionally done by juniors.

“You go in and AI asks you questions about your assets and your children and works out a division based on what you might want, and what the court may accept or not accept. That takes a several week process down to 10 minutes,” he explains.

“You can still have lawyers involved to check over things but as a process it’s open, transparent, clear to clients (and) it does it a lot quicker and more efficiently than a human.”

Amid all the hype about the chat bot replacing lawyers, both young and old, Bennett Moses urges a look back in time, arguing that “the same thing has been said about earlier tech.”

“For example, the use of computer systems in document review was going to have the same effect. No-one can see the future, but I’ll believe it when it actually happens,” she says.

Misel, too, urges legal grads to remain upbeat. After all, they are not the only ones in line to feel the effects of generative AI systems, like ChatGPT.

As global consulting firm McKinsey has said, it may fundamentally alter business not just in the legal profession, but also in areas like marketing, sales, IT, HR, and customer service.

“There will be whole slew of law graduates that will invent this new space,” Misel says.

“They’re not going into a predetermined landscape where it was obvious, they’d have to work all of God’s hours before they got up to the next rung.”