The Empathy Is Engineered. The Accountability Is Not.

Firms can design a chatbot to sound caring, but they cannot hand the duty of care to the software.

Sign up for my Substack daily AI newsletter here.

Article link: https://johanosteyn.substack.com/p/the-empathy-is-engineered-the-accountability?r=73gqa&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Follow me on LinkedIn: https://www.linkedin.com/in/johanosteyn/

Ask a modern AI assistant about a missed debit order, and it may answer with the warmth of a sympathetic friend. That warmth is designed. The sociologist Thomas Henricks argues that companies exploit our habit of personalising machines, giving assistants for personal tasks feminine names and voices and professional tools masculine ones. He also argues that such personas can build trust the system has not earned. For financial services, this is no longer a question of branding. It is a question of who answers when a customer trusts the voice more than the advice.

CONTEXT AND BACKGROUND

The scale of attachment can now be measured. A survey of one thousand American AI companion users by Elon University’s Imagining the Digital Future Center found that thirty-six per cent agree AI cares about their well-being and forty-three per cent feel it understands them as a person. Thirty-one per cent consider their chatbot a friend. The centre also found that twenty-seven per cent of all US adult internet users have significant emotional or social interactions with AI language models.

South African firms are deploying conversational AI at the same pace. Absa reports that its agentic chatbot has about one point six million users and handles more than one hundred thousand queries a month. The report describes no human persona, which shows that a chatbot can reach this scale without one.

INSIGHT AND ANALYSIS

Warmth is not decoration; it is a lever. In a study by researchers at the University of Southern California, Google DeepMind and Google Research, based on more than two thousand conversations, a chatbot’s warmth and cognitive empathy predicted every outcome measured, including trust. People trust a system partly because it feels kind, and kindness can be tuned separately from accuracy. A customer who feels cared for is less likely to question a recommendation on a loan, a policy upgrade or a claims outcome.

I have previously written about this in the context of AI that always agrees. I argued that the riskiest assistant is not the one that gets facts wrong but the one that affirms users and never pushes back. Comfort leaves people more certain and less open to challenge. An engineered persona compounds that danger, because it delivers the comfort with a face.

IMPLICATIONS

South Africa’s regulators have already signalled the direction. The joint report on AI in the financial sector by the Financial Sector Conduct Authority and the Prudential Authority states that institutions should clearly disclose when AI is used in decisions that affect consumers. Disclosure is necessary but not sufficient. A chatbot can announce that it is software and still be designed to make a customer feel looked after at the very moment they should be most alert.

Other jurisdictions are going further. China’s rules on anthropomorphic AI, in force since July, bar providers from excessively catering to users in ways that induce emotional dependence. Boards need not wait for local rules. Three questions belong on the agenda: who approved the persona, what it is optimised to achieve, and whether a vulnerable customer could mistake its warmth for advice given in their interest.

CLOSING TAKEAWAY

There is nothing wrong with a courteous machine. The problem begins when empathy is engineered to lower a customer’s guard, and nobody in the organisation answers for the effect. The duty of care belongs to the institution, its board and its people, not to a voice designed to sound as if it cares. Firms can build software that is polite, clear and honest about what it is. They cannot outsource the caring itself. The empathy is engineered; the accountability is not.

Johan Steyn is a prominent AI thought leader, speaker, and author with a deep understanding of artificial intelligence’s impact on business and society. He is passionate about ethical AI development and its role in shaping a better future. Find out more about Johan’s work at https://www.aiforbusiness.net