The Algorithm Doesn't See Race, But Your Postal Code Does

South African retail personalisation engines can reproduce the exclusions of apartheid geography while appearing perfectly neutral.

Sign up for my Substack daily AI newsletter here.

See my AI Training course portfolio for corporate Business Leaders here.

Article link: https://open.substack.com/pub/johanosteyn/p/the-algorithm-doesnt-see-race-but?r=73gqa&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Follow me on LinkedIn: https://www.linkedin.com/in/johanosteyn/

An algorithm is never given a field for race. It does not need one. It can see where a customer lives, what they buy, when they shop and how they pay, and from those signals it can price, target and lend. In South Africa, where a person lives still says a great deal about who they are.

CONTEXT AND BACKGROUND

South African retailers have built personalisation engines that reach well beyond points and vouchers. Woolworths has consolidated loyalty, online and financial services data into a single customer profile under its MyDifference programme, and describes loyalty data as a strategic asset that now informs merchandising, pricing and planning. These are not marketing tools any longer. They are decision engines that shape which offer a customer sees, what they may be charged, and whether they qualify for credit. The commercial case is settled. The fairness of those decisions is the question most boards have not asked.

INSIGHT AND ANALYSIS

The danger is not that an engine is instructed to discriminate. It is that it discriminates without being instructed to. A model trained on shopping behaviour and location can produce outcomes that track race, even though race is never an input, because the proxies it does use are correlated with it. This is not hypothetical in South Africa, where residential segregation imposed under apartheid has barely shifted: research on Gauteng found that 80% of residents in areas built after 1990 live in neighbourhoods with under 10% racial mixing.

Where a person lives remains a close proxy for race. South African law saw this long before AI arrived. In Pretoria City Council v Walker, the Constitutional Court held that charging customers different rates by geographical area, though neutral on its face, amounted to indirect discrimination on the grounds of race, because apartheid had made race and geography inseparable. An algorithm that prices or lends by postal code risks reproducing the very discrimination Walker condemned, depending on its effect. POPIA sharpens this: section 71 restricts decisions based solely on automated processing that carry legal or substantial effects, including a person’s creditworthiness. A token human sign-off on an otherwise automated decision is unlikely to insulate a firm from a disparate-impact challenge.

IMPLICATIONS

The exposure is sharper than most boards assume, because intent is not the test. A discriminatory outcome can be unlawful even where no one intended it, which means “the model calculated the risk” is not a defence. The Information Regulator has signalled where this is heading, announcing in 2025 a focus on the ethical use of AI and automated decision-making, naming customer profiling specifically, against a backdrop of tougher enforcement and rising fines. A proxy such as location can be operationally useful, but it must be tested for disparate impact before deployment, not after a complaint. A retailer that cannot show how its engine reaches a price or a credit decision, and cannot test that decision for discriminatory effect, is carrying constitutional, statutory and reputational risk at once.

CLOSING TAKEAWAY

The answer is not to abandon personalisation. It is to govern it as the decision engine it has become. A board should be able to answer three questions before the next campaign launches: what data feeds the engine and where it flows, whether the outputs have been tested for disparate impact across protected groups, and whether any decision with real consequences for a customer has a human who can explain it. An algorithm that cannot see race is not the same as one that does not act on it. In South Africa, the difference is often measured by postal code.

Johan Steyn is a prominent AI thought leader, speaker, and author with a deep understanding of artificial intelligence’s impact on business and society. He is passionate about ethical AI development and its role in shaping a better future. Find out more about Johan’s work at https://www.aiforbusiness.net