AI may improve individual performance, but it doesn’t beat the wisdom of crowds

Microsoft’s 2024 Work Trend Index shows that up to 75% of knowledge workers are using generative AI, even though more than half are reluctant to admit that they do so. The vast majority even use their own technology:

Power users – those who use AI extensively – say that using AI tools:
– makes their workload more manageable (92%)
– boosts their creativity (92%)
– helps them focus on the most important work (93%)
– makes them feel more motivated (91%)
– helps them enjoy work more (91%).

But does AI help us make better decisions?

According to the paper “Will Humans-in-the-Loop Become Borgs? Merits and Pitfalls of Working with AI”, co-authored by Andreas Fügener, Jörn Grahl, Alok Gupta and Wolfgang Ketter, AI helps us make more accurate decisions, but less unique ones.

The authors conducted a series of tests and experiments. In one of those experiments, the authors asked participants to classify a set of images. Some participants worked independently; others with an AI tool that would suggest a classification plus a degree of confidence for its own classification. Moreover, some participants worked alone and others in groups.

The authors found that (p. 1549):

“The effect of AI advice on group performance changes quickly as the group becomes larger. The accuracy for humans with AI advice does not increase after reaching a certain group size: for AI advice without AI certainty, the accuracy reaches a level of .84 for group sizes above three, while it reaches a level of .86 for group sizes above four if AI certainty was added. 

[However], humans without AI advice keep improving over the entire spectrum. Starting with a group size of seven, human groups without AI advice outperform human groups with AI suggestions only. If AI suggestions are complemented with AI certainty, it takes groups of more than eleven humans without AI advice to outperform the groups with AI advice.”

The authors go on to argue that (pp. 1549-1550):

“The value of correct advice decreases by increasing group size, as groups are likely to select the correct decision without advice as well. 

Incorrect advice harms groups in two ways. Not only do humans who would have otherwise chosen the correct answer select the incorrect AI suggestion, but many humans who do not know the correct answer tend to select the AI suggestion, making it the modal choice.

Overall, we can conclude that while AI advice helps individual performance, the reduction in unique human knowledge severely harms crowd performance. When very small in size, pure human groups are outperformed by crowds of humans that work with AI. However, as the group size increases, even at modest sized groups (a computable threshold), the performance of human groups without AI assistance starts dominating performance of those with AI assistance. Finally, interestingly, note that a group of four humans provides more accuracy than the AI.“

That is, while AI advice improves the performance of individuals and smaller groups, it significantly harms the performance of larger groups. So, by all means, do use AI. But remember that AI may be incorrect, and do not underestimate the value of the wisdom of crowds.