Generative AI such as ChatGPT offers plenty of opportunities for accountants to automate certain tasks, but it’s important to understand its issues and limitations before putting the technology to use.

Stuart Cobbe, Principal Consultant at The Analytical Accountant, talked to ICAEW about testing the abilities and limits of ChatGPT, having worked with artificial intelligence (AI) and machine learning for eight years. He understood that the oncept of benchmarking for AIs that usually involves some kind of measure against human performance.

Knowing that ChatGPT needs to work with text-based questions, he opted to put it through the ICAEW assurance assessment paper. The version in place at the time, dubbed GPT-3.5, failed with a score of 42%. While this seems like a fairly respectable score for AI, it uncovered some major issues with the bot, most notably hallucination problems.

However, when ChatGPT-4 was released, Cobbe tried the assessment again. This time, it passed with 78% – a significant improvement on last time, with better understanding and reduced rambling.

This demonstrates how quickly generative AI models such as ChatGPT are improving, but also the issues that are still present within them. The accountancy sector is rapidly adopting AI and machine learning within its various firms and functions, and available options are increasing – Google has just announced its generative AI, Bard.

With reports from members about recommendations for nonexistent books and Excel formulae conjured from thin air, it’s clear that accountants need to be more aware of generative AI’s uses and limitations. Part of the issue is the confidence in which AIs can present incorrect data, which can lead people to trust what it’s giving them.

Any use of generative AI should be approached with caution and critical thinking. Members should engage their professional scepticism when using AIs.

It’s important to pay extra attention to any topics that are specific to accounting, audit, standards and regulations, particularly where the UK regulatory landscape differs from other major markets. While it appears to have accounting content within its training corpus, it has been fine-tuned for specific accountancy-use cases.

Appropriate prompting and structuring is also required to get usable answers. Questions might have a number of contexts, so it’s important to be specific. It is also important to ook at various generative AIs when considering what to use, as they are better at some tasks than others.

For the full article visit the ICAEW page.