The Effect of Use of Generative AI on Group Problem Solving
Description
Our study investigates how adding an AI chatbot as a third member to a group of two people affects their ability to work together and learn from each other while solving problems. We set up an experiment where different groups solve a murder mystery game - some groups work with AI help, while others have restricted AI access or no AI at all, to see how this changes the way people interact and come up with different solutions. The goal is to understand whether using AI in group work might reduce how much people learn from each other and limit their creativity in finding diverse solutions, ultimately aiming to suggest better ways to use AI in workplace teams.
Documentation
As AI chatbot adoption reaches 70% in workplaces (Singla et al., 2024), its impact on collaborative problem-solving is becoming increasingly significant. Platforms such as Slack and TeamAI, which incorporate advanced models like Gemini, GPT-4, and LLaMA, are specifically designed to enhance teamwork through AI support. However, concerns have emerged that individuals may begin to rely too heavily on AI tools, potentially diminishing meaningful peer interaction during key tasks like information gathering and idea generation. This study explores the tension between utilizing AI chatbots and engaging with human collaborators by examining the effects of integrating an AI chatbot as a third member in triadic groups. Through analyzing group dynamics in these settings, we offer insights into the broader implications of AI integration for effective collaboration in organizations, online communities, and society at large.
What is the effect of AI chatbots on problem-solving? Research reveals a tension between findings at the individual level and those at the group level. On one hand, AI chatbots have been shown to outperform humans in creative tasks (Bohren et al., 2024) , and their use at the individual level has been associated with more accurate decision-making (Gajos & Mamykina, 2022). Additionally, human experts can complement AI prediction models when the algorithm's input lacks contextual information, especially when the duration is long and uncertainty is low (Revilla et al., 2023).
On the other hand, the findings from the group level have been somewhat mixed. Replacing human decision-making with machine learning algorithms can significantly reduce the diversity that arises from involving multiple human perspectives. Algorithms often lack the nuanced understanding of rich contextual information, which can lead to learning myopia and make organizations more susceptible to overlooking environmental changes
(Balasubramanian et al., 2022). Additionally, the adoption of AI tools may unintentionally reinforce existing biases if their limitations are not critically evaluated (Kellogg et al., 2020). For instance, when AI is used to filter job candidates, recruiters may lose the chance to collaboratively assess which applicants are the best fit for the role (Cohen & Mahabadi, 2022). Furthermore, people over-rely on the AI and disregard their own knowledge (Fügener et al., 2022). These findings underscore the potential risks of AI integration in group settings, particularly when it diminishes human input and critical reflection.
While prior research presents contrasting views on human–AI collaboration, it remains unclear what is happening during the collaboration in human—AI group. Thus, we ask the question: how does the level of integration of AI chatbot into the group affect collaborative problem-solving? We extend existing literature in two ways. First, we focus on collaboration tasks that require active interpersonal communication(Shore et al., 2015), rather than passive or individual engagement. Second, we examine collaborative learning as a main outcome, moving beyond performance-based metrics. Drawing on March’s exploitation— exploration framework and the current human—AI collaboration literature, we propose a framework for understanding how AI chatbot shapes group dynamics and test our hypotheses through a series of controlled experiments.
In group problem solving, the groups’ performance depends on the diversity of individuals’ belief (March, 1991). Prior research suggests that inefficient or slower learning processes can help preserve this diversity, ultimately leading to superior problem-solving outcomes (Hong & Page, 2004). Studies have further shown that imperfect or somewhat inefficient communication can enhance information sharing by delaying premature convergence on a single solution. This occurs because reduced direct information copying fosters the exploration of multiple ideas and promotes broader solution generation (Mason & Watts, 2012; Shore et al., 2015).
Building on this framework, we investigate how varying levels of AI chatbot integration influence collaborative group dynamics. To do so, we conceptualize three distinct levels of integration in the context of collaborative problem-solving. At the lowest level, there is no AI involvement—only human collaborators work together, independently seeking information and communicating via online platforms. In the partial integration condition, individuals use an AI chatbot independently to support their own tasks, while still engaging in group collaboration with their human teammates. At the highest level, the AI chatbot is fully integrated into the group as an active participant in the problem-solving process, treated as a peer and interacting with human members throughout. Based on these levels of integration, we develop three hypotheses to test how AI presence shapes group collaboration and learning outcomes.
