What Happens The Next Time, With A Less Careful Team?

The restrictions applied to AI-designed viruses were voluntary, well judged and entirely dependent on the judgement of the people applying them.

Sign up for my Substack daily AI newsletter here.

See my AI Training course portfolio for corporate Business Leaders here.

Article link: https://johanosteyn.substack.com/p/what-happens-the-next-time-with-a?r=73gqa&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Follow me on LinkedIn: https://www.linkedin.com/in/johanosteyn/

A research team in California did something recently that has never been done before, and did it carefully. They restricted what their model could design, chose a target that cannot infect people, and worked in contained conditions. Not one of those decisions was required of them by anybody.

CONTEXT AND BACKGROUND

Researchers at Stanford University and the Arc Institute reported in Science on 6 August that a generative model called Evo 2 had written complete viral genomes from scratch. Given only a fragment of the bacteriophage known as ΦX174 as a starting point, the model produced novel sequences, of which the team synthesised nearly 300 and found 16 that killed E. coli exceptionally well. Their stated purpose is medical. Bacterial resistance defeats single-agent treatments over time, and a mixture of genetically distinct phages is far harder for bacteria to overcome, with the team showing that their 16 rapidly defeated E. coli strains already immune to the natural phage. Similar approaches might eventually target tuberculosis, MRSA and Pseudomonas aeruginosa, a leading cause of drug-resistant hospital infections.

The researchers have made Evo 2 openly and freely available, and Brian Hie, who leads the work, argues that open availability is essential to expediting research, that existing natural pathogens present a greater risk than potential AI designs because they are easier to access, and that safety checks can be built into AI tools in a way they cannot be built into natural evolution.

INSIGHT AND ANALYSIS

Writing in the same issue of Science, Thomas Inglesby and Moritz Hanke of the Johns Hopkins Center for Health Security praised the team for taking precautions and then stated the problem in a single line: the ability to compose viral genomes using generative AI now exists, and the governance to safely steer it does not. That is the sentence to sit with. Every safeguard applied here was chosen by the researchers themselves. No statute compelled the training data restrictions. No regulator reviewed the choice of target organism. No external body verified the containment. The controls held because the people involved decided they should, and by every account they decided well. A control that depends on the character of the person applying it is not a control at the level of an industry, it is professional ethics, and professional ethics is what every other safety-critical field supplemented with external oversight decades ago. Nobody asks a nuclear operator to self-certify. The wider regulatory picture is a patchwork too, one in which biotechnology breakthroughs routinely outpace the agencies meant to oversee them, and where a recent policy on high-risk life sciences research did not address AI-driven work at all.

IMPLICATIONS

This is not a distant concern for South African readers, and it is not primarily a biology story. It is a governance story with a familiar shape. The MIT FutureTech expert panel identified dangerous capabilities and AI-enabled weapons among the five risk domains still carrying at least a ten percent probability of catastrophic outcomes even under pragmatic mitigation, and noted that competitive development leaves organisations with limited incentive to slow down or invest sufficiently in safety, which is why laws, treaties and other collective-action mechanisms may become necessary.

CLOSING TAKEAWAY

They may well be right that openness accelerates the medical benefit, and the benefit is real. Antimicrobial resistance kills, and it kills disproportionately in places with weaker health systems, which includes ours. The argument here is not that this work should not have happened. It is that the restraint which made it safe was supplied by individuals rather than by institutions, and that this arrangement works exactly as long as everyone with the capability happens to be similarly careful. Inglesby and Hanke observed that this team engaged with these questions more deliberately than most developers of powerful biological AI models. That is a compliment to the team and a warning about everyone else.

Johan Steyn is a prominent AI thought leader, speaker, and author with a deep understanding of artificial intelligence’s impact on business and society. He is passionate about ethical AI development and its role in shaping a better future. Find out more about Johan’s work at https://www.aiforbusiness.net