Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant

A between-subjects study with a single independent variable, the XAI interface, over five conditions. N = 306 after exclusions, recruited on Prolific.

Task. Loan approval using the public Kaggle loan prediction dataset. Participants judged whether an applicant was Credit Worthy or Not Credit Worthy from eleven features including loan amount, income and presence or absence of credit history, presented both as a table and as a textual description. Ten tasks per participant, order randomised. The authors chose the domain because it carries a clear risk and benefit when adopting AI advice, because laypeople are familiar with it, and because it has precedent in behavioural economics and human-AI collaboration research.

Protocol. Two-stage decision making, described by the authors as similar to the update condition in Green and Chen. The participant decided unaided, then saw the AI's prediction and, depending on condition, its explanation, then had a second chance to revise. At each stage the participant also named the three features most important to their decision and rated their confidence.

Conditions (five, between-subjects): Control, no XAI interface. Dashboard, an interactive dashboard exposing five XAI methods one at a time through a navigation tab. CXAI, a rule-based conversational XAI interface over the same five methods, driven by custom buttons, dropdowns and feature inputs rather than free text. ECXAI, the same conversational interface with evaluative steering: the assistant adapts its guiding messages to the top three features the participant named at stage one, prompting them to compare their own criteria against the model's. The authors relate this to evaluative AI and to cognitive forcing functions. LLM Agent, the conversational interface reimplemented with autogen and GPT-4, accepting free-text input, selecting XAI methods on demand and elaborating the outputs into coherent prose. Otherwise identical in procedure.

XAI methods (identical across all four XAI conditions). Five methods mapped to six information needs from the XAI question bank: PDP for global feature effect (how), SHAP for feature importance on the current prediction (why), MACE for the minimum profile change that flips the prediction (why not, how to be that), WhatIf for model output on a user-modified profile (what if, how to still be this), and a decision-tree path for the sequence of steps leading to the advice. All implemented with the OmniXAI library. Participants in conversational conditions had to view at least two different XAI responses per task, enforced only when they tried to proceed without engaging.

Stimulus selection. Ten cases drawn from a 4:1 train-test split, evenly divided between Credit Worthy and Not Credit Worthy ground truth and chosen by a matrix of prediction correctness and model confidence: for each class, one high-confidence correct, one low-confidence correct, one random-confidence correct and one high-confidence wrong case, plus one further low-confidence wrong case for Credit Worthy and one further random-confidence correct case for Not Credit Worthy, so as to fix AI accuracy at exactly 70%.

Procedure. Consent and machine-learning background, pre-task affinity-for-technology-interaction questionnaire, onboarding tutorial and practice example tailored to condition, ten tasks, then post-task questionnaires covering understanding, explanation utility, trust and engagement.

Measures. Understanding was captured objectively as the nDCG of the participant's top-three features against the SHAP importance ranking, and subjectively across perceived feature understanding, learning effect across tasks and understanding of the system. Explanation utility covered completeness, coherence, clarity and usefulness. Trust used three validated subscales of the Trust in Automation questionnaire: reliability/competence, understanding/predictability, and trust in automation. Reliance used agreement fraction and switch fraction; performance used overall accuracy and accuracy on trials of initial disagreement; appropriate reliance used relative positive AI reliance (RAIR) and relative positive self-reliance (RSR). Engagement used UES-SF. Analysis was ANCOVA with propensity to trust, familiarity, affinity for technology interaction and machine-learning background as covariates, plus Kruskal-Wallis H-tests with Mann-Whitney post-hoc for non-normal measures, and estimation plots alongside p-values.

Every interactive XAI interface increased reliance on the AI, and almost none of that increase was appropriate. Both reliance measures rose sharply against the control: agreement fraction H = 33.66, p < .001, from 0.74 in control to 0.85 to 0.89 in all four XAI conditions; switch fraction H = 19.14, p = .001, from 0.31 to 0.57 or 0.58. Team accuracy did not follow: overall accuracy was not significantly different across conditions (H = 9.09, p = .059, means 0.62 to 0.67) and neither was accuracy on trials of initial disagreement (H = 5.06, p = .281). Participants averaged 64.5% accuracy against the AI's 70%.

Splitting appropriate reliance in two shows where the added reliance went. Relative positive self-reliance collapsed in every XAI condition (H = 38.26, p < .001): 0.57 in control against 0.29 dashboard, 0.23 CXAI, 0.26 ECXAI and 0.11 LLM Agent, with the LLM Agent condition significantly worse than both control and dashboard. Participants stopped standing by their own correct judgements. Relative positive AI reliance improved only marginally and only in one arm (H = 11.01, p = .026, above the Bonferroni-corrected threshold of 0.0125), with control 0.35 below CXAI 0.60 the sole post-hoc difference. H3 was not supported.

The conversational interface did not beat the dashboard on understanding. ANCOVA across Dashboard, CXAI and ECXAI found no significant effect of interface on any dimension of self-reported understanding. On the objective measure the effect ran the wrong way: Kruskal-Wallis found a significant difference (H = 16.19, p = .001) with the LLM Agent condition scoring significantly worse on objective feature understanding than Dashboard, CXAI and ECXAI. H1 was not supported.

The mechanism the authors propose is the illusion of explanatory depth. Interactive XAI interfaces function as persuasive technology: they raise perceived understanding, perceived explanation utility and trust, and their exploratory analysis finds all three positively correlated with over-reliance. Making the conversation more fluent amplifies the illusion rather than correcting it, which is why the GPT-4 arm produced the worst self-reliance and the worst objective understanding simultaneously. Perceived explanation utility itself did not differ significantly across interfaces on one-way ANOVA, so the effect is not that the LLM explanations were rated better; it is that greater plausibility displaced the participant's own judgement.