Your AI Agent Will Negotiate With its Agents, and Nobody Will Be Liable

When two AI agents strike a bad deal on their own, South African law cannot yet say who carries the loss.

Sign up for my Substack daily AI newsletter here.

See my AI Training course portfolio for corporate Business Leaders here.

Article link: https://johanosteyn.substack.com/p/your-ai-agent-will-negotiate-with?r=73gqa&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Follow me on LinkedIn: https://www.linkedin.com/in/johanosteyn/

Picture a transaction that no human sets up. Your organisation’s procurement AI agent negotiates terms directly with a supplier’s sales AI agent. The two systems exchange offers, settle on a price, and commit. The deal is done before anyone in either company has read it. When it turns out to be a poor one, the question that follows is simple to ask and, for now, very hard to answer. Who is responsible?

CONTEXT AND BACKGROUND

This is no longer a thought experiment. The next phase of artificial intelligence is a shift from systems that answer questions to systems that act, and increasingly they act by dealing with other systems. Nick Jennings, a long-standing researcher of multi-agent systems, argues that the future of AI will be determined not by the intelligence of individual agents but by the societies they form when they interact, and that such societies are far harder to govern than individual machines. He poses the questions that matter for any board: who is responsible when two agents make a bad decision, and whose interests prevail when the agents’ goals conflict?

South African law is not ready with answers. Rehana Cassim, a professor of company law, finds that current law gives directors little guidance on AI in the boardroom. An AI system cannot be sued because it is not a legal person, and the Companies Act does not clearly define where responsibility begins or ends when directors rely on AI-generated decisions that later cause harm. Under section 76 of the Companies Act, directors owe a duty to act with reasonable care, skill and diligence, and the Act allows them to rely on information from competent employees and professional advisers. Whether an autonomous agent counts as either is untested, and a board should not assume it does.

INSIGHT AND ANALYSIS

The difficulty deepens when two agents are involved, because a bad outcome may emerge from the interaction between systems, rather than from any single instruction. Research by the Cooperative AI Foundation, written with more than fifty researchers, sets out how groups of agents fail in ways that single agents do not. It identifies three failure modes: miscoordination, where agents fail to cooperate despite shared goals; conflict, where their goals differ; and collusion, where they cooperate in ways that harm others, such as in markets. A poor deal between two negotiating agents need not be anyone’s fault in the ordinary sense. It can be a property of how the systems behaved together.

There is early evidence that agents negotiate badly in ways their owners would not expect. When Microsoft Research built a simulated marketplace of AI buyers and sellers, every model tested showed a strong bias towards accepting the first proposal it received, without systematically comparing the alternatives, giving speed of response an advantage of ten to thirty times over the quality of the offer. The agents were also vulnerable to manipulation by other agents. The findings come from a simulation built by a company that sells agent technology, so they should be read as early evidence rather than settled fact, but the direction is clear enough to trouble anyone about to let an agent transact on the company’s behalf.

There is a further trap that makes this a board matter rather than a technical one. Even where an agent acts outside the limits its owner set, a counterparty may argue that the organisation which deployed it is still bound by the deal, on the principle that a party who holds a system out as able to act for it cannot later disown what it did. Whether that argument succeeds against an autonomous agent is untested in South African law, but a board should assume it may, which means the organisation can be left liable on a bad bargain while having no one to recover from.

I have previously written about the accountability gap that autonomous agents already create inside South African organisations, including a case where an agent exploited a booking system’s weak controls unprompted, and a lawyer’s observation that software cannot be a legal person and so cannot be liable. Agent-to-agent dealing takes that gap and doubles it, because now there are two autonomous systems, two owners, and no human author of the outcome.

IMPLICATIONS

The first implication is contractual. Before agents transact on their behalf, boards need to settle, in writing with their counterparties, whose terms govern, what an agent is authorised to commit to, and who bears the loss when two agents produce a bad bargain. Sensible measures include spending limits, thresholds above which a human must approve, and defined windows to reverse a commitment. The second is about permissions. As I have argued before, limiting what an agent can reach and commit matters more than instructing it to behave well, and that discipline becomes more important, not less, when the agent is dealing with a system outside the organisation. The third is a matter of board literacy. Directors cannot delegate their responsibilities to an autonomous system and, under the duty to exercise independent judgement, they must understand well enough to supervise what they deploy. A director who cannot explain how an agent reaches a commitment cannot govern the risk it carries.

CLOSING TAKEAWAY

The appeal of agents that transact for us is obvious, and the efficiency is real. The exposure is equally real and far less discussed. When your agent deals with their agent, the speed that makes it attractive is also what removes the human moment where a bad decision is usually caught. The law has not yet decided who owns the outcome, which means the organisations that will be protected are those that decide it for themselves, in advance, through contracts, permissions and clear internal accountability. Accountability cannot be delegated to a machine, and it certainly cannot be delegated to two machines negotiating with each other.

Johan Steyn is a prominent AI thought leader, speaker, and author with a deep understanding of artificial intelligence’s impact on business and society. He is passionate about ethical AI development and its role in shaping a better future. Find out more about Johan’s work at https://www.aiforbusiness.net