How Does Conversation Length Impact User’s Satisfaction? A Case Study of Length-Controlled Conversations with LLM-Powered Chatbots
Video Engineering 2025 Graduate ExhibitionPresentation by Shih-Hong Huang
Exhibition Number 512
Abstract
Users can discuss a wide range of topics with large language models (LLMs), but they do not always prefer solving problems or getting information through lengthy conversations. This raises an intriguing HCI question: How does instructing LLMs to engage in longer or shorter conversations affect conversation quality? In this paper, we developed two Slack chatbots using GPT-4 with the ability to vary conversation lengths and conducted a user study. Participants asked the chatbots both highly and less conversable questions, engaging in dialogues with 0, 3, 5, and 7 conversational turns. We found that the conversation quality does not differ drastically across different conditions, while participants had mixed reactions. Our study demonstrates LLMs’ ability to change conversation length and the potential benefits for users resulting from such changes, but we caution that changes in text form may not necessarily imply changes in quality or content.
Importance
People can discuss various topics with AI chatbots, but they don’t always prefer long conversations to solve problems or find information. This led us to explore whether adjusting a chatbot’s response length affects how users perceive the conversation. We developed Slack chatbots powered by ChatGPT with the ability to modify conversation length and conducted a user study to gather feedback. Our findings suggest that chatbots capable of adjusting their response style based on the situation can enhance user experience. We further argue that future conversational assistants should be designed to recognize user needs and adapt their responses and conversation strategy accordingly. As a next step, we aim to develop chatbots that can detect user situation in real time and adjust conversations base on situation.