Researchers and consultants sometimes make the headlines with their use of artificial intelligence because they’ve been dumb and lazy. PwC published reports on AI marred by AI hallucinations, screams the Financial Times in the UK, as some overworked consultants published a report produced in part (or in whole?) by AI with plenty of problems included. While this makes for good copy and induces healthy levels of schadenfreude, it doesn’t really say very much about the usefulness of AI as a tool to create analysis and advice.
My interest in AI as a consultant and researcher has never been in how AI could replace me in doing my job. Frankly, the more that I use it, the more secure that I feel. In the past, I have likened it to a very quick and clever junior colleague, but one who needs constant guidance to steer answers that it gives. It’s good at making connections, but ultimately, just doesn’t quite comprehend what it writes. And of course, it doesn’t, because AI does not properly replicate reasoning. It brings materials together and it makes connections, but it often pulls things together incorrectly.
And don’t get me started on all of that apologising that the tool does in that anthropomorphising way. Please, stop telling me that I’m right and then why the latest answer you’re giving me is the right one.
But treating an AI like a junior employee has also led me down the wrong path in how to use it as a tool, I think. It means that I give it whole tasks in the way that I would for a junior, allowing it to draft materials and then providing critiques until I would get what I wanted. Yes, I’d end up rewriting bits, but in the same way that I would for a junior, I’d leave its voice intact.
I’ve found that this way of working with AI creates a number of problems:
- AI has a tendency to over-complicate problems and write lots and lots of words. Perhaps this is how a lot of consultants write, but it becomes easy to identify when an AI has written something. Have one good word? Why not use both THIS word and THAT word to describe it. And every idea needs qualification after qualification, each nested in the next one–and also at different points in the text, leading to repetitions. Perhaps with a lot of prompts that could coach the AI in how to produce text, it might work better, but I’ve not found the magic solution yet.
- Once an AI has given me a structured argument, I find myself focused on correcting missing or incorrect bits. This can be quite time consuming, as corrections need re-correcting, as one change touches on other parts of the logic in an argument. So every time a correction would be implemented, it would just make the foundation of an entire piece more wobbly. It was like working with Jenga-AI. The structure of an argument never fell down, but it sure wasn’t completely right either.
Recently, I was constructing a slide deck for a series of AI applications that I’ve been developing to help with procurement processes. I spent an entire day debating and arguing with AI, trying to weave its first draft into a complete piece. Having arrived at something that I could agree with, I sent it to a colleague, who immediately identified it as AI slop and just a bundle of confusion (despite AI being happy with the result that it produced).
Starting again from scratch, within an hour, I had produced a new document, which the same colleague described as ‘a breath of fresh air’. Did the day of work help in producing that new document? Probably a bit, but certainly not the day that it cost me.
One further insight that my colleague brought to my attention in working with AI with research and consultancy tasks is that, ultimately, the AI has been designed to drive engagement. The more I use it, the more tokens I use, and the more I can get charged. And indeed, I got lost in interacting with the AI, guiding it to a proper answer, when writing it up myself would simply have been a better way to go.
This isn’t to say that I’ve completely given up on AI as a useful tool in my work. When designing proof-of-concept applications, it does seem to have some value (though perhaps after more experience, I’ll change my mind here). And when working with limited tasks and questions, it can be a supplement to the work that I do. But I’m more and more aware of how it changes my own way of working, and it’s not always for the better.