We built it, and we do not yet understand it
A Yale-led paper is the latest reminder that even the researchers who ship frontier AI models are reverse-engineering their own creations.
Sign up for my Substack daily AI newsletter here.
Follow me on LinkedIn: https://www.linkedin.com/in/johanosteyn/
The most striking line in the frontier AI research of the last two years has not come from a sceptic. It has come from the laboratories that build the models. Earlier this month, a Yale-led team working with researchers from Microsoft and New York University published new evidence that large language models organise information internally in ways their builders are only now beginning to map. The paper is a careful, hopeful piece of work. The implicit admission inside it is not a comfortable one for the C-suite that has signed off on dozens of AI deployments this year. The people who ship these systems do not yet fully understand how they produce the outputs they produce. The governance consequences of that gap are large, and they land on the board, not on the lab.
CONTEXT AND BACKGROUND
The Yale study, led by linguistics professor Tom McCoy, examined how large language models produce answers in domains that look like reasoning: language, arithmetic, logic and code. Across seven large models, the team found that the long numeric vectors making up a model’s internal state are organised in a way equivalent to symbolic structure, and that specific features can be isolated and intervened on. The pointillist painting is McCoy’s analogy. Up close, individual dots are meaningless numeric values. Step back, and recognisable structure emerges. Interpretability is moving from aspiration to method. The gap between where it is now and where the deployment pace demands it to be remains wide, and the C-suite is sitting inside that gap.
INSIGHT AND ANALYSIS
The Yale team’s contribution is the mathematical demonstration that these models carry symbolic structure inside their numeric vectors. Anthropic’s complementary work, published in May 2024, is the empirical mapping of specific features inside a live production model. Its researchers mapped millions of features inside Claude 3 Sonnet, found that features for abstract concepts cluster together, and showed that amplifying or dampening those features changes the model’s behaviour. The research is candid about its own limits. Anthropic acknowledged it had mapped only a small subset of the model’s concepts and had not yet shown the features could be used to improve safety.
The gap between research and deployment matters. A frontier AI vendor can truthfully say its model is working. The same vendor cannot yet say which features drove a specific output or which circuits were activated. Americans for Responsible Innovation, writing a policy guide for the US Congress, concluded that interpretability is a prerequisite for meaningful oversight. The same conclusion lands harder in South Africa, where POPIA Section 71 already requires a responsible party to explain the logic of an automated decision.
The commercial consequence is that the vendor’s assurance is not yet strong enough to carry a board resolution. Safe is a marketing word. Audited for these specific features is a governance word. The research field is only just starting to permit the second kind of statement.
IMPLICATIONS
The explainability gap has already crossed from research into enterprise policy. Raconteur reported this year that explainability has become a procurement requirement for enterprise AI buyers in regulated sectors, and the vendors able to answer it are a shrinking subset of the market. The Centre on Regulation in Europe reached a similar legal conclusion: when the law controls AI, the law needs the black box opened. South Africa’s Draft National AI Policy has not yet absorbed what the Yale and Anthropic work now permits.
For a South African board, the question is a fiduciary one. Section 76 of the Companies Act 71 of 2008 requires each director to act with the care, skill and diligence a reasonable person in the role would exercise. A board that signs off on an AI deployment on a vendor’s unverified assurance, when interpretability research is now beginning to permit specific feature-level questions, is operating below that standard. The next deployment review should ask which features the vendor’s own interpretability team has mapped, and which outputs can be traced to those features in writing.
CLOSING TAKEAWAY
Cognitive governance rests on a simple principle. Accountability for an automated decision stays with the human who delegated it. Yale, Microsoft, NYU and Anthropic are now showing that the field can begin to meet that principle on its own ground. The researchers who build frontier models are starting to open them. The vendors who sell them have not caught up. The South African board that signs off a deployment without noticing the gap is operating on trust the research now permits them to question. The next conversation is not whether the model works, but what the vendor can prove it is doing.
Author Bio
Johan Steyn is a prominent AI thought leader, speaker, and author with a deep understanding of artificial intelligence’s impact on business and society. He is passionate about ethical AI development and its role in shaping a better future. Find out more about Johan’s work at https://www.aiforbusiness.net
Comments