Imagine that a colleague has presented you with a business plan. Your AI recommends that you approve it but your judgment is not to. Do you trust the algorithm or yourself?
Of course, some AI systems are more capable than others. However, new research suggests that successful AI application may depend less on its capability and more on the personality traits of the professionals that use it.
In a recent laboratory experiment, University of Duisburg-Essen researchers Alisa Küper and Nicole Krämer gave 250 participants a task with objectively right or wrong answers. The participants received advice purportedly from an AI that was often correct but deliberately incorrect at times.
Enthusiasm and skill are entirely different things
The researchers found that participants’ personalities sometimes led to dangerous over-reliance on AI. For example, participants with a high propensity to trust technology, or who reported enjoying using technology, were far more likely to follow the AI’s recommendations even when its advice was wrong. In other words, willingness to engage with AI is not the same as ability to use it well.
Confidence over skill
This is consistent with other psychological evidence. In 1999, Cornell University psychologists David Dunning and Justin Kruger demonstrated that people’s confidence in their own abilities rarely relates to their actual skill level. Those with the most confidence are often the least skilled; in contrast, those who are more self-doubting often outperform their more self-assured peers. Applied to AI, this suggests that team members who most zealously adopt new tools might also be most at risk of being misled when those tools turn out to be wrong.
Research into professionals’ use of AI is just starting to grow. However, it already suggests important implications for leaders keen to use AI well.
Be cautious
Increasingly, organisations are setting targets for AI token use or measuring adoption by volume of interactions. However, such metrics can easily backfire. Using more AI does not guarantee better outcomes if individuals cannot discriminate accurately between correct and incorrect advice.
A recommendation that is feasible from a technical point of view may be unworkable in practice
At the team level, savvy leaders gently assess each person’s relationship with and appropriate use of technology. Just because certain individuals enjoy learning about new technologies and spend considerable time using AI does not mean that their recommendations will necessarily be more accurate. High confidence does not guarantee high competence.
Think critically
While AI can analyse data and make seemingly plausible recommendations, its processes are rarely transparent. Teams could gainfully be coached to evaluate AI outputs: checking the AI’s sources, questioning assumptions, testing the logic and considering the wider context. For example, a recommendation that is feasible from a technical point of view may be unworkable in practice; AI models can rarely integrate factors such as interdepartmental politics, organisational traditions and levels of resistance among different stakeholders.
An analyst cannot credibly say a spreadsheet told them to do something
Research in safety-critical industries such as aviation and nuclear operations has long documented that, over the course of many months and years, professionals’ reliance on reliable automation tends to grow while their skill at evaluating it critically tends to decline. The same pattern may emerge around the use of AI. Just because AI seems to be doing well does not mean that it cannot make future, possibly catastrophic, mistakes. Emphasising the need for ongoing oversight and critical thinking will be a continuing necessity for leaders.
Make ownership explicit
Where organisations succeed is when leaders communicate a clear rule: AI makes recommendations, but humans own decisions. Professionals must not be allowed to explain away errors by saying ‘The AI told me to do this.’ An analyst cannot credibly say a spreadsheet told them to do something; if the output was wrong, then the individual who relied on it is accountable. The same principle applies to AI. Establishing – and reinforcing – this rule is one of the most important things leaders can do.
The broader message for leaders is this: the question of how well your organisation uses AI is not primarily about technology, it is about humans. Developing your people’s skill at using AI outputs appropriately could be the difference between competitive advantage and serious harm.
More information
Find out more about the use of AI in the workplace at ACCA’s Accounting for the Future conference, including the session ‘Evolving roles in a data-driven age’. Register to watch live on 24-26 November or on demand.
Visit ACCA’s AI knowledge hub with resources and guidance.
