Virginia Tech® home

The Effect of Use of Generative AI on Group Problem Solving

Description

Our study investigates how adding an AI chatbot as a third member to a group of two people affects their ability to work together and learn from each other while solving problems. We set up an experiment where different groups solve a murder mystery game - some groups work with AI help, while others have restricted AI access or no AI at all, to see how this changes the way people interact and come up with different solutions. The goal is to understand whether using AI in group work might reduce how much people learn from each other and limit their creativity in finding diverse solutions, ultimately aiming to suggest better ways to use AI in workplace teams.

Documentation

As AI chatbot adoption reaches 70% in workplaces (Singla et al., 2024), its impact on  collaborative problem-solving is becoming increasingly significant. Platforms such as Slack  and TeamAI, which incorporate advanced models like Gemini, GPT-4, and LLaMA, are  specifically designed to enhance teamwork through AI support. However, concerns have  emerged that individuals may begin to rely too heavily on AI tools, potentially diminishing  meaningful peer interaction during key tasks like information gathering and idea generation.  This study explores the tension between utilizing AI chatbots and engaging with human  collaborators by examining the effects of integrating an AI chatbot as a third member in  triadic groups. Through analyzing group dynamics in these settings, we offer insights into the  broader implications of AI integration for effective collaboration in organizations, online  communities, and society at large.  

What is the effect of AI chatbots on problem-solving? Research reveals a tension between  findings at the individual level and those at the group level. On one hand, AI chatbots have  been shown to outperform humans in creative tasks (Bohren et al., 2024) , and their use at  the individual level has been associated with more accurate decision-making (Gajos &  Mamykina, 2022). Additionally, human experts can complement AI prediction models when  the algorithm's input lacks contextual information, especially when the duration is long and  uncertainty is low (Revilla et al., 2023). 

On the other hand, the findings from the group level have been somewhat mixed. Replacing  human decision-making with machine learning algorithms can significantly reduce the  diversity that arises from involving multiple human perspectives. Algorithms often lack the  nuanced understanding of rich contextual information, which can lead to learning myopia  and make organizations more susceptible to overlooking environmental changes 

(Balasubramanian et al., 2022). Additionally, the adoption of AI tools may unintentionally  reinforce existing biases if their limitations are not critically evaluated (Kellogg et al., 2020).  For instance, when AI is used to filter job candidates, recruiters may lose the chance to  collaboratively assess which applicants are the best fit for the role (Cohen & Mahabadi,  2022). Furthermore, people over-rely on the AI and disregard their own knowledge (Fügener  et al., 2022). These findings underscore the potential risks of AI integration in group settings,  particularly when it diminishes human input and critical reflection.

While prior research presents contrasting views on human–AI collaboration, it remains  unclear what is happening during the collaboration in human—AI group. Thus, we ask the  question: how does the level of integration of AI chatbot into the group affect collaborative  problem-solving? We extend existing literature in two ways. First, we focus on collaboration  tasks that require active interpersonal communication(Shore et al., 2015), rather than  passive or individual engagement. Second, we examine collaborative learning as a main  outcome, moving beyond performance-based metrics. Drawing on March’s exploitation— exploration framework and the current human—AI collaboration literature, we propose a  framework for understanding how AI chatbot shapes group dynamics and test our  hypotheses through a series of controlled experiments. 

In group problem solving, the groups’ performance depends on the diversity of individuals’  belief (March, 1991). Prior research suggests that inefficient or slower learning processes  can help preserve this diversity, ultimately leading to superior problem-solving outcomes (Hong & Page, 2004). Studies have further shown that imperfect or somewhat inefficient  communication can enhance information sharing by delaying premature convergence on a  single solution. This occurs because reduced direct information copying fosters the  exploration of multiple ideas and promotes broader solution generation (Mason & Watts,  2012; Shore et al., 2015). 

Building on this framework, we investigate how varying levels of AI chatbot integration  influence collaborative group dynamics. To do so, we conceptualize three distinct levels of  integration in the context of collaborative problem-solving. At the lowest level, there is no AI  involvement—only human collaborators work together, independently seeking information  and communicating via online platforms. In the partial integration condition, individuals use  an AI chatbot independently to support their own tasks, while still engaging in group  collaboration with their human teammates. At the highest level, the AI chatbot is fully  integrated into the group as an active participant in the problem-solving process, treated as  a peer and interacting with human members throughout. Based on these levels of integration,  we develop three hypotheses to test how AI presence shapes group collaboration and  learning outcomes.

Figure 1: Experiment condition: collective problem solving with a human partner + AI chatbot (ChatGPT – 4o)