The Model Hallucinates, The Partner Signs

South African firms have adopted AI for contract review and due diligence without resolving who carries the liability when it fabricates.

Sign up for my Substack daily AI newsletter here.

See my AI Training course portfolio for corporate Business Leaders here.

Article link: https://open.substack.com/pub/johanosteyn/p/the-model-hallucinates-the-partner?r=73gqa&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Follow me on LinkedIn: https://www.linkedin.com/in/johanosteyn/

A machine cannot be struck from the roll, referred to a regulator, or sued for negligence. The professional who signs off on its output can be all three. That asymmetry sits at the centre of how South African law firms, audit practices and advisory businesses are deploying artificial intelligence, and it is the part of the bargain that almost no one priced in when the tools arrived.

CONTEXT AND BACKGROUND

Professional services firms have spent much of the past decade warning clients about the dangers of unverified technology, which made it all the more uncomfortable when one of their own became the cautionary tale. In October 2025, KPMG published a flagship report on agentic AI in which only five of its forty-five citations pointed to real, intact sources, prompting the firm to withdraw the report and review how it had been published. South African firms are adopting the same class of tools for contract review, due diligence and regulatory monitoring, driven by real and substantial time savings. The efficiency case is settled. The question of who answers for the output when it is wrong is not.

INSIGHT AND ANALYSIS

South African courts have already given a preview of where accountability lands. In Mavundla, the Pietermaritzburg High Court found that of nine authorities a legal team had cited, only two existed, with the remainder appearing to be AI fabrications, and in Northbound Processing the Gauteng High Court held that neither urgency nor a sincere apology excused placing non-existent cases before it . The junior counsel in Northbound had used a tool marketed as trained solely on South African legal sources, and the senior counsel conceded he had done only a sense-check rather than verifying the citations, yet the court referred the practitioners to the Legal Practice Council and affirmed their duty to check AI output against authoritative sources. The regulatory point is often misunderstood.

Since 2018 the Legal Practice Council, not the Law Society of South Africa, has regulated the conduct of attorneys and advocates, while the Law Society continues as a voluntary representative body. The principle is blunt. Liability for a fabricated clause attaches to the human who signed, whether framed as a breach of mandate or as negligence measured against the standard of care a reasonable practitioner owes.

IMPLICATIONS

The exposure does not stop at the regulator. Professional indemnity cover has long assumed a competent human stands behind the work, and insurers are now moving to clarify, condition or exclude AI-related claims rather than absorb them silently, which leaves firms vulnerable where they cannot show that a person verified what the machine produced. A managing partner who assumes existing cover answers for an AI error may learn otherwise at the worst possible moment. There is a quieter risk beneath the visible one. Every contract and due diligence pack fed into an AI tool is personal information processing under POPIA, section 71 restricts decisions taken solely on automated processing, and a prudent control is to de-identify client data before it reaches any external model. The hallucinated clause is the failure the firm can see. The data that left the building through an unvetted tool is the exposure the Information Regulator may act on, depending on what was shared and whether it was lawfully justified.

CLOSING TAKEAWAY

None of this is an argument against the tools. It is an argument for treating their output as a draft from an unverified source, never as a finished product. That means a senior professional checking every AI-assisted citation, figure and clause against the primary source before it leaves the firm, disclosing the use of AI in engagement letters, and keeping client data out of public models. The board-level point is simple. Adopting AI does not delegate the duty of care that comes with professional work, it concentrates it. When the model hallucinates, the partner signs, and in South African law and practice the signature is what counts.

Johan Steyn is a prominent AI thought leader, speaker, and author with a deep understanding of artificial intelligence’s impact on business and society. He is passionate about ethical AI development and its role in shaping a better future. Find out more about Johan’s work at https://www.aiforbusiness.net