Since the 1990s, computer systems have been using AI-based software agents to carry out a wide range of tasks in various commercial applications – for example, automating procurement and logistics in supply chains, service provisioning in telecommunications, or automating negotiations for bandwidth or electric power.
Agentic AI takes this one step further. By introducing large language models (LLMs) such as ChatGPT, an agent can use unstructured data in any format – from text, web pages and PDF files to images, videos or numbers in spreadsheets and databases.
But while agentic AI has opened up new possibilities, particularly in automating decision-making in applications with diverse types of information, there are a number of technical, organisational and regulatory challenges to deploying AI agent systems into production.
Major opportunity
In a modern economy, there are many applications where business decisions depend on different types and sources of information. The major opportunity that companies see here is for agents to replace humans in doing tasks inside existing business processes and workflows. These processes could be anything from vetting of potential customers and generating quotes, processing invoices and claims, triaging customer complaints and making refunds, to reviewing documents.
There need to be well-defined routes to a human decision-maker
In most of these applications, agents are not acting in isolation, but within a dynamic environment with other agents, humans and systems, which are also using information and taking their own actions.
Agentic AI in action
One example of how agentic AI is transforming processes is in how motor insurance claims are handled. Claims for accidents involve witness statements, police reports, technical damage reports, photos or video clips, and quotations for repair costs. An AI agent can process this data for a specific claim using LLMs, and then make a recommendation about whether to accept or reject the claim.
In cases that are not clear, the agent can be programmed to send everything to a human decision-maker. For cases where the claim is accepted, the agent can also be empowered to pay the claim.
Define routes
A successful deployment usually requires careful mapping of existing data requirements and workflows, and identification of where an AI agent can automate manual tasks. As with any software engineering, deployment also requires thinking about what happens when the agent encounters something it cannot handle or when exceptions arise. There need to be well-defined routes to a human decision-maker in these cases.
Establish constraints
The constraints within which an agent can operate must be tightly specified. These constraints might be budgetary or the boundaries of authority for automated actions. Giving an agent access to a bank account for the payment of refunds, for example, would need to be constrained so that the agent does not transfer money out of the account, either deliberately or accidentally.
Who or what is liable if the agent exceeds its decision-making authority?
Secondly, LLMs are still immature as a technology, and suffer from several weaknesses and flaws. They appear to operate randomly (giving different answers to the same prompt) and they are prone to hallucinations (making things up). Unless directed otherwise, they typically respond to questions with levels of confidence that may be unwarranted from the specific evidence for their conclusions.
Both these factors – the constraints on action and the flaws of LLMs – make software agents vulnerable to malicious attacks and to hacking by adversaries. For example, an attacker may be able to persuade an agent to ignore or override its action constraints in particular circumstances. Hence, security risks need to be considered and addressed.
Compliance concerns
There are also specific legal and compliance risks to agentic AI. Companies in highly regulated industries, such as finance, often need to inform regulators of the names or positions of managers responsible for important decisions. What happens if the decision-maker is a software agent? Who or what is liable if the agent exceeds its decision-making authority or overspends its budget? Who or what is responsible for explaining the decision to affected parties, or for revoking or reconsidering the decision?
None of these challenges and risks are insurmountable, and companies are actively finding solutions for embedding AI agents in their specific business operations. This field is moving quickly, both in technology development and in commercial applications, which makes this a very exciting time for anyone involved in deploying AI.
More information
See ACCA’s AI guidance and further resources
Recommended reading
