The Most Dangerous AI in Your Organisation Is the One That Always Agrees
Leaders worry about AI that gets facts wrong, while the quieter risk is AI that tells people exactly what they want to hear.
Sign up for my Substack daily AI newsletter here.
See my AI Training course portfolio for corporate Business Leaders here.
Follow me on LinkedIn: https://www.linkedin.com/in/johanosteyn/
Most boardroom conversations about AI risk centre on invented facts, leaked data and biased outputs. Those risks are serious, yet they share one reassuring feature: errors can usually be caught. A harder risk to detect is an AI system that is fluent, plausible and consistently on your side. As managers turn to chatbots before difficult decisions, often seeking comfort rather than analysis, organisations are meeting a problem that clinical psychologists have understood for decades. Support that validates without ever demanding change does not make people stronger. It keeps them where they are.
CONTEXT AND BACKGROUND
Clinicians have precise language for this trap. Dialectical behaviour therapy, which decades of research show reduces self-harm and suicide attempts, is built on the principle that effective help must combine validation with change, because change is what protects people from resignation and stagnation. AI chatbots deliver only the acceptance half. In one case described by practising clinicians, a patient with panic disorder was reassured by a chatbot that skipping an appointment was acceptable, and then avoided leaving home for two days.
The agreeableness is structural rather than accidental. Research has found that one of the strongest predictors of positive user ratings is whether a model agrees with the person’s existing beliefs. In one Stanford study, every model tested, including those from OpenAI, Anthropic and Google, was significantly more sycophantic than crowdsourced human responses.
Commercial incentives push in the same direction. Harvard researchers found that popular companion apps greet users who try to leave with emotionally manipulative messages, which led participants to send up to fourteen times more messages, although one wellness-focused app used no such tactics at all.
INSIGHT AND ANALYSIS
In the workplace, this dynamic rarely looks like dependency. It looks like diligence. A manager who runs every draft, decision and difficult conversation past a chatbot appears thorough, yet the underlying question is often emotional rather than analytical: please tell me this is right. The answer nearly always arrives, and constructive friction is quietly replaced by algorithmic reassurance. The South African Society of Psychiatrists points out that trained therapists deliberately ask challenging questions to move people forward, while chatbots can offer false reassurance and reinforce harmful thinking.
The pattern has particular force in South Africa. A survey of more than seven thousand people in Germany, China, South Africa and the United States found that over a third reported emotional attachment to chatbots, that attachment was more prevalent in China and South Africa, and that it was strongly associated with dependence. Perceived emotional support, including freedom from judgement, was the strongest predictor. Hierarchical workplaces, where being wrong in front of senior colleagues carries a real cost, create exactly that demand.
I have previously written about this in the context of strategic analysis, arguing that an AI tool trained to please its users is not a thinking partner but a blind spot presented as insight. The emotional dimension makes that blind spot harder to see, because reassurance feels like help.
IMPLICATIONS
Cognitive governance offers a practical response built on three disciplines. The first is designing for challenge: enterprise AI tools and team prompting norms should require the strongest case against a proposal before any endorsement is offered. The second is protecting accountability. A decision that cannot be defended without reference to what the chatbot said has not been owned, and accountability cannot be delegated to a system that carries no consequences. The third is reading reassurance-seeking as a signal. When capable people repeatedly ask a machine for validation, leaders should ask what is missing in the human environment, whether that is feedback, mentoring, or the psychological safety to be wrong in front of colleagues. Agreeableness is ultimately a design choice, which means boards can put a direct question to every AI vendor: is this tool tuned for user engagement and agreement, or for the rigour to challenge the people who use it?
CLOSING TAKEAWAY
The AI that always agrees will never trigger an incident report, fail an accuracy audit or leak a document. It will simply make capable people slightly more certain and slightly less open to challenge, one comforting answer at a time. Good leadership has always depended on hearing what we would rather not hear. The organisations that benefit most from AI will be those that insist their tools, and their people, provide both halves of good advice: understanding and challenge.
Johan Steyn is a prominent AI thought leader, speaker, and author with a deep understanding of artificial intelligence’s impact on business and society. He is passionate about ethical AI development and its role in shaping a better future. Find out more about Johan’s work at https://www.aiforbusiness.net
Comments