Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI
N = 153 Graduate admissionThis study in the framework
Held constant, but two different figures apply. The domain-specific model was a multi-category linear regression, discretised into four decision labels then binarised for scoring, trained on a 70% split of a synthesised 100-profile dataset. It reached 80% accuracy on the held-out 30% test set. The four cases actually used in the study were not drawn at random from that test set. They were selected from a pilot as cases where human and AI opinions conflicted, typically ambiguous profiles near the admission borderline, and on these cases both the humans and the model performed between 50% and 60%. The effective AI accuracy experienced by participants was therefore close to chance, not 80%. The authors are explicit that this reflects deliberate selection of conflict cases rather than an unrealistically weak model, and they note that this is precisely the regime where conventional XAI failed to help and could even hurt.
- RQ1: How will Human-AI Deliberation affect task performance and humans' reliance on AI suggestions?
- RQ2: How will Human-AI Deliberation affect humans' perceptions of the AI partner and their user experience?
- RQ3: How will humans perceive the effectiveness of the proposed Human-AI Deliberation and what can be improved for future Human-AI Deliberation design?
"participants agreed significantly less with AI's suggestions in Deliberative AI than in XAI, and switched significantly less to AI's predictions"
results, RQ1
"our tool mitigates over-reliance not merely by increasing cognitive effort but by fostering meaningful deliberation"
results, RQ1
"the traditional XAI might lead to humans' illusionary confidence, which could prevent humans from making optimal decisions"
results, RQ2
Experimental design
A between-subjects mixed-methods study with three conditions, run on Prolific. 174 participants recruited, 153 valid after attention checks.
System. The paper proposes Deliberative AI, an architecture that couples a domain-specific model with a large language model so that human and AI can externalise, compare and argue about their reasoning rather than exchange a recommendation. It has three layers: a communication layer that translates between user and system, a control layer managing querying and opinion updating, and a knowledge layer holding the domain model. Human and AI thoughts are put in a common representation, weight of evidence per profile dimension, with the AI side computed by SHAP so that contributions aggregate additively to the model's actual prediction.
Task. Graduate admission prediction: deciding whether to admit or reject applicants to a US university. The dataset is a synthesised set of 100 applicant profiles built from public aggregate statistics, with attributes including GRE Verbal, GRE Quant, GRE Writing, GPA, statement of purpose strength, diversity statement strength, country, major, undergraduate institution rank and recommendation letter strength. Original four-level labels (strong reject, weak reject, weak accept, strong accept) were binarised into reject and accept for scoring. The domain model was a multi-category linear regression discretised into the four labels, trained on a 70% split and reaching 80% accuracy on the held-out 30%.
Stimulus selection. Five cases were used, one for the tutorial and four for the main task, chosen from a pilot specifically as cases where human and AI opinions conflicted, typically ambiguous profiles near the admission borderline. On these cases both humans and the AI performed between 50% and 60%. The authors are explicit that this reflects deliberate selection of conflict cases rather than a weak model.
Conditions (three, between-subjects): Deliberative AI (N = 48). The participant states their own weight of evidence on each dimension, then sees the AI's per-dimension thoughts, then can open a dialogue with the AI about any conflicting dimension, with both sides able to update their opinions. Explainable AI (N = 51). The participant predicts, then receives the AI recommendation with feature-contribution explanations, then makes a final judgement. Human Alone (N = 54). No AI assistance.
Procedure. Consent, background questionnaire covering demographics and AI expertise, introduction to the task and the AI's functions including its ability to update its opinions but not the mechanism behind it, an interactive tutorial with one practice case and per-attribute distribution statistics, then qualification questions that had to be answered entirely correctly to proceed. Four main cases in random order, then an exit survey.
Measures. Decision accuracy. Reliance as agreement fraction and switch fraction, with appropriateness as over-reliance ratio and under-reliance ratio. Perceptions of AI as 7-point items for helpfulness, trustworthiness and understanding. User experience as decision confidence, mental demand, effort, perceived complexity and satisfaction, all on 7-point scales. Open-ended feedback and full conversation logs.
Full findings
Deliberation improved accuracy by making people rely on the AI less, not more. Decision accuracy was significantly higher with Deliberative AI (M = 0.598, SD = 0.169) than with conventional XAI (M = 0.524, SD = 0.16, p < .05). The comparison that matters more is with the unassisted baseline: XAI accuracy fell below Human Alone, so on these conflict-selected cases the conventional explanation interface actively hurt.
Both reliance measures fell sharply under deliberation. Agreement fraction dropped from 0.68 (SD = 0.27) with XAI to 0.57 (SD = 0.24) with Deliberative AI (p < .05), and switch fraction from 0.51 (SD = 0.41) to 0.23 (SD = 0.35, p < .001). Decomposed by appropriateness, over-reliance fell from 0.65 (SD = 0.33) to 0.47 (SD = 0.31, p < .001) while under-reliance did not differ (0.29 against 0.32, n.s.). This is the paper's cleanest result and its most useful one for the base: deliberation cut over-reliance without the compensating rise in under-reliance that cognitive forcing functions produce. The authors read this as evidence that the mechanism is meaningful engagement with the AI's reasoning rather than raised cognitive effort per se, and support it with the workload data, which showed no significant differences in mental demand, effort or perceived complexity across the three conditions.
Trust moved with behaviour, downward. Participants reported significantly less trust in Deliberative AI (M = 4.47, SD = 1.68) than in XAI (M = 5.52, SD = 1.27, p < .01), while perceived helpfulness and perceived understanding did not differ. The conversation logs explain why: deliberating exposed the model's flaws in a way that reading its explanation did not. Of the 48 participants in the Deliberative AI condition, 43 said deliberation helped them make more informed decisions, and 21 specifically said it let them identify the AI's limitations.
XAI participants reported significantly higher decision confidence (M = 6, SD = 1.10) than Human Alone participants (M = 5.59, SD = 1.12, p < .05) while being less accurate than them. Conventional explanations produced confidence that ran opposite to performance.
Deliberation cost user experience. Satisfaction was significantly lower with Deliberative AI than with Human Alone, which the authors attribute to the system surfacing conflicts the participant then has to resolve, quoting one participant complaining about having to find evidence to defend their own view. They frame this as a trade-off between deliberative engagement and experience, consistent with the cognitive forcing literature.