Companies across the world are currently creating and deploying computer programs called software agents, intended to undertake commercial and operational tasks automatically. Many such agents access or embody large language models (LLMs) and other advanced AI systems in order to learn about their world, develop plans of action within that world and execute those plans, all without close human supervision.
There are many risks associated with the widespread use of software agents. What if they exceed their budgets or go beyond specified constraints on their actions? What if they misunderstand or misinterpret their instructions, or act illegally or unethically? For example, in Australia, a software agent from a major technology vendor, undertaking a routine research task, managed to circumvent a government portal’s refusal and was able to access non-public medical data.
Different risks of agentic AI should concern anyone responsible for financial control of an organisation
If software agents are instructed by malicious human actors or teams, one can easily imagine the serious misconduct they may be able to engage in. But even if all humans and all agents were well intentioned and well behaved, we would still need to be concerned. This is because agents, as with any software, may have coding bugs that result in aberrant behaviours or may contain vulnerabilities allowing other programs to exploit them.
Moreover, when multiple agents are running on the same platform or in the same digital environment, the possibility exists of interaction effects that are unforeseen or even unforeseeable.
Unanticipated interactions
For example, a recent legal case in Singapore (Quoine Pte Ltd v B2C2 Ltd [2020] SGCA(I) 2) related to unanticipated interactions between an algorithmic trading program operating on a digital assets trading platform and the automated programs running the platform. As in most of the digital economy, the various programs interacting with each other were designed by different human teams at different times, with no one anticipating the complex interactions that would arise between the different software agents.
Recognise the whole AI ecosystem in designing assurance and standards
These different risks and possibilities for serious consequences of agentic AI should concern anyone responsible for management or financial control of an organisation. Facing such risks, 23 British computer scientists, including me, recently wrote an open letter to the UK government, calling on it to consider legal, technical and commercial actions that might eliminate or mitigate such risks from agentic AI.
Government action
The initiative, led by professors from the universities of Loughborough, Oxford and Southampton, recommended that the government take seven actions: three immediate steps and four longer term.
The immediate steps would use only existing government powers:
- Clarify legal responsibility. Working with the Ministry of Justice, ask the Law Commission to build on its 2025 discussion paper and examine who may owe duties, bear liability and provide redress when an agent acts for someone across multiple services.
- Learn from serious incidents. Building on the new agentic AI incident response capability, pilot a common route for reporting significant agent failures and near misses, with clear timescales for notifying affected organisations and UK authorities, using existing AI incident frameworks where possible. Ask an independent body to investigate selected cases and publish lessons, drawing on aviation’s approach to accident investigation.
- Use public procurement to set practical requirements. Require public-sector agent contracts to identify an accountable deployer; record consequential actions and the authority for them; limit access, spending and data use; and provide a tested way for a person to pause, override or take back control.
The four longer-term recommendations seek to develop standards and undertake testing and research to promote wider adoption, which may require new laws or regulations:
- Extend agent evaluation into deployment settings. Commission the AI Security Institute and the National Physical Laboratory, with industry regulators and independent assessors, to develop standard methods for testing agents with real tools, permissions and multi-agent workflows, and make test facilities accessible to smaller suppliers.
- Develop interoperable identity and delegation standards. Work with international standards bodies and partners on ways for an agent to identify itself, prove whom it acts for, and show the limits of its authority across services and borders. Standards should allow permissions to expire or be revoked, and should build on existing identity and provenance work rather than creating a separate regime.
- Fund open-evaluation research and shared facilities. Support technical and social research on long-running agents, multi-agent interaction, human handover, resource use and the effectiveness of controls, including shared testbeds and public challenges whose results can be checked independently.
- Recognise the whole AI ecosystem. Include smaller developers, open-source communities, deployers, users, civil society and sector specialists alongside major model providers in designing assurance and standards, and assess whether requirements are proportionate to risk or would create avoidable barriers to entry.
Agentic AI can offer huge benefits across civil society and industry. But without clear and strong government input, the risks will vastly outstrip the benefits, with serious consequences.
