Artificial intelligence (AI) is the simulation of human intelligence processes by machines and has been having a transformative effect on the global legal services market. A landmark 2018 study pit 20 highly experienced lawyers against an AI system. The study found the AI system averaged accuracy to 94% while the lawyers averaged 85%. So is it game over for lawyers?
Overview of the AI vs Lawyer study
It is important to first understand the parameters of the study before drawing conclusions. This study was conducted by LawGeex who primarily perform automated contract review. 5 non disclosure agreements (NDA’s) were used from the Enron data set which are considered industry standard. The lawyers were then asked to identify and highlight issues as they appeared in the contracts.
Only highly experienced US trained lawyers participated in the study, sourced through freelance hiring sites and the research team’s . LawGeex AI uses custom built machine learning and read tens of thousands of non-disclosure agreements (NDAs) as preparation. This level of technology for analysing legal documents has only been possible with advances in computing over the last five years.
The accuracy of all participants was measured through 3 final metrics.
- Recall. This measured the number of issues accurately identified. As a standalone measurement recall is insufficient as it allows a maximum score through guesswork.
- Precision. Measures the number of correct answers made against the number of total answers given.
- F-Measure. The final accuracy score is the harmonic mean between Precision and Recall.
Results
The key statistical takeaway is that the lawyers had an average accuracy of 85% while the AI averaged 94%. Further to this, the lawyers averaged 92 minutes to complete all 5 NDA‘s while the AI averaged 26 seconds. Even when looking at the highest performing lawyer the highest individual score on a single contract was 97%. The AI engine achieved 100% accuracy in one of the contracts.
Clearly within the realm of this study the AI is leaps and bounds ahead of the lawyers. So does this mean AI is going to replace lawyers in the future?
What this means for the future of AI and lawyers?
On the surface it may appear that artificial intelligence will completely replace lawyers. However while AI will continue to be a necessary tool for lawyers, undue weight should not be placed on the technology alone. Lawyers will continue to be necessary in strategic legal work. Although this study did not go beyond the identification of legal issues. Responding appropriately to each issue would be too challenging for the current technology. The AI is currently unable to handle more complex legal tasks such as case strategy and the day to day duty of relationship management.
Subsequently, there will be numerous uses for AI in the future of the legal industry. However, this technology is not a standalone tool. Similarly to the way pilots use auto pilot, AI in the future will be utilised symbiotically. Lawyers will experience a reduced workload in addition to an increase in accuracy and efficiency.
Conclusion
This study represents a major landmark in the history of legal technology. Lawyers who fear artificial intelligence overrunning the legal industry should instead recognise the advantages of embracing AI. In the bigger picture in addition to reducing the workload for lawyers, clients will also receive more efficient and accurate services.
