Consciousness is not a feature your AI vendor can ship
Leading consciousness scientists in Nature and at UCT argue that current large language models are not, and probably cannot be, conscious.
Sign up for my Substack daily AI newsletter: https://johanosteyn.substack.com/
Follow me on LinkedIn: https://www.linkedin.com/in/johanosteyn/
Boards are buying “AI employees”, “digital colleagues” and “AI teammates” from the same vendors who ship them software. The brand on the org chart is not neutral. In late October 2026, three of the most prominent consciousness scientists wrote that current large language models are not conscious and probably cannot be, and a Boston University experiment measured the cost of ignoring the finding: label the same work “AI employee” instead of “AI tool” and manager error-catching falls, review requests rise, and accountability drifts toward a system that cannot hold it. The argument is reaching South African boards at the exact moment the Draft National AI Policy, POPIA and King V all tighten the demand for human accountability. The branding that looked like adoption marketing may turn out to be a quiet liability leak.
CONTEXT AND BACKGROUND
Writing in late October 2026, David Weinberger laid out an old philosophical argument for a new audience. Large language models cannot be conscious, he argued, because the concept of consciousness does not apply to systems without bodies. He routed through John Searle’s Chinese Room, Martin Heidegger’s Being and Time and Iris Marion Young’s Throwing Like a Girl, concluding that embodied care is where meaning lives.
The position does not sit alone. In a Nature comment on 19 October 2026, Anil Seth of the University of Sussex, Giulio Tononi of Wisconsin and philosopher Ned Block of New York University argued that intelligence and consciousness are different things, that substrate matters, and that deep-learning architectures are wired in the wrong direction for subjective experience. Mark Solms at the University of Cape Town argues consciousness is grounded in brainstem affective systems that register what matters to an organism before any cortex gets involved, and that machines cannot host it. David Chalmers at NYU holds that the question remains empirically open.
INSIGHT AND ANALYSIS
Here the philosophy stops being philosophy. In randomised experiments with roughly 1,261 managers, Emma Wiles of Boston University and her co-authors measured what happens when identical work is credited to an AI employee rather than an AI tool. Where organisations already list AI agents on the org chart, error-catching fell by 18%, requests for additional review rose by 44%, and perceived accountability shifted from the manager toward the AI system. The label changes how the human behaves. The system does not change at all.
I have previously written about this in We Are Not Machines, But We Are Becoming One, where I argued that workplace AI policies rarely address what AI does to how people think about their work. The Wiles research puts a number on the drift. A functionalist retort is that the metaphysics is business-irrelevant; the finding is that the measured accountability cost does not care.
Analysts at the University of New South Wales argue that an AI agent occupies a hybrid position fitting neither tool nor employee, and accountability leaks when organisations fail to say which of three modes the system is operating in: drafting, advising or executing.
IMPLICATIONS
For South African boards, this is a King V question in new clothes. The Draft National AI Policy and POPIA both assume human duty-bearers. The courts assume natural persons. If current AI cannot host subjective experience, and if African relational-personhood holds that personhood is achieved through moral community rather than ascribed by label, calling an AI agent a colleague on an org chart constructs an illicit liability shield. In the essay Towards an AI for Africa, Fainos Mangena argues Ubuntu ethics treats personhood as something that develops through relationships of care and mutual recognition, which no software agent can enter.
The practical response is not to abandon AI agents. It is to stop paying vendors to blur the line between a tool and a teammate. Decide whether a deployment is drafting, advising or executing, name the human who owns the outcome at each level, and keep the audit trail visible. Treat product language as governance policy.
CLOSING TAKEAWAY
Consciousness is not a feature your AI vendor can ship. No naming convention and no org chart entry closes the gap between a tool that processes symbols and a person who can be held responsible. The convergence of late 2026 philosophy and science has changed the terms of the debate; King V, POPIA and the Draft National AI Policy are changing the terms of the law. For South African boards, the question is not whether AI is conscious. It is whether the way you talk about it makes you a less effective steward of the duties the law places in your hands.
Johan Steyn is a prominent AI thought leader, speaker, and author with a deep understanding of artificial intelligence’s impact on business and society. He is passionate about ethical AI development and its role in shaping a better future. Find out more about Johan’s work at https://www.aiforbusiness.net
Comments