The AI detector arms race is breaking trust between students and educators

If institutions rely on imperfect detection, they risk punishing honesty and rewarding confidence.

Article link: https://open.substack.com/pub/johanosteyn/p/the-ai-detector-arms-race-is-breaking?r=73gqa&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Audio summary: https://youtu.be/yDUoDKq5xzg

Follow me on LinkedIn: https://www.linkedin.com/in/johanosteyn/

We are entering a new, uncomfortable phase of education: not simply “students using AI”, but institutions being flooded with AI-generated text and responding with detection tools that cannot carry the weight we’re placing on them. The result is an arms race that schools and universities are not equipped to win, and the collateral damage is trust. When a black-box score can question a student’s integrity, classrooms become adversarial.

Educators feel they are losing control, students feel presumed guilty, and honest work becomes harder to defend than dishonest work is to produce. This matters deeply in South Africa, where inequalities already shape outcomes and where second-language learners can be disproportionately exposed to suspicion. We need a calmer, fairer way forward.

CONTEXT AND BACKGROUND

A recent piece in The Conversation describes how AI-generated text is overwhelming institutions, from education to courts and media, and how this is triggering a no-win “arms race” with AI detectors.

Bruce Schneier, writing from a security and governance perspective, echoes the same warning: arms races create perverse incentives, and society suffers when systems are clogged, and trust erodes.

Education is where the tension becomes personal. Unlike spam filters or content moderation, the consequences are human: reputations, scholarships, graduations, and a student’s relationship with learning. Yet many institutions are still treating detection scores as if they were evidence rather than a probability.

INSIGHT AND ANALYSIS

The fairness problem is straightforward: AI detectors make mistakes, and those mistakes are not evenly distributed. A cautious student who writes in simple sentences might be flagged; a highly fluent second-language student might be flagged; a student who uses mainstream grammar tools might be flagged.

Meanwhile, someone who knows how to prompt well, paraphrase, or lightly edit can reduce detection signals. The result is a system that can punish honesty and reward confidence.

Even detection vendors acknowledge this is a moving target. Turnitin’s own guidance notes ongoing updates to its AI writing detection model and positions it as a tool to support interpretation, not a final verdict. The problem is what happens in the real world: when staff are overloaded, policies are unclear, and the temptation is to treat a score as proof.

Research and teaching guidance increasingly emphasise the damage caused by false flags: wrongly accused students, lengthy investigations, and a breakdown of trust between educators and learners. A practical example is the University of Sydney’s discussion of false positives and “broken trust”, which warns that confident detection is often misplaced.

This is already showing up in lived experience. Reporting on universities navigating AI rules highlights the confusion and fear students feel about being wrongly accused, especially when policies vary from class to class and tools are used inconsistently.

IMPLICATIONS

First, schools and universities should stop treating detector outputs as decisive evidence in high-stakes misconduct cases. If a student can be punished, there must be due process: clear policies, transparent criteria, the right to respond, and a requirement for corroborating indicators beyond an algorithmic score.

Second, institutions need to redesign assessment so that authentic thinking is harder to outsource. That does not mean abandoning writing. It means shifting to process-based work: drafts, reflections, oral defences, in-class components, and short viva-style questioning that tests understanding. Detection can play a limited, supportive role, but it cannot be the foundation.

Third, educators need a clear, shared language for acceptable AI use. News organisations are already building transparent AI usage policies to protect trust; education should do the same. Students need to know what is allowed, what must be disclosed, and what counts as misrepresentation.

CLOSING TAKEAWAY

The central risk of the AI detector arms race is not that some students will cheat. That problem has always existed. The bigger risk is that we normalise suspicion and outsource judgment to tools that cannot be fair arbiters of integrity. In South Africa, where language, access, and confidence differ wildly across schools, the cost of false accusations will fall hardest on those who already have the least margin for error. The way out is not better policing. It is a better design: clearer rules, stronger due process, and assessments that make learning visible in ways that no detector ever will.

Author Bio: Johan Steyn is a prominent AI thought leader, speaker, and author with a deep understanding of artificial intelligence’s impact on business and society. He is passionate about ethical AI development and its role in shaping a better future. Find out more about Johan’s work at https://www.aiforbusiness.net