Guided Reflection in AI-Assisted Decision-Making: Effects on AI Overreliance and Decision Accuracy

Between-subjects experiment with three conditions, run over a two-stage prediction task.

Task. Diabetes prediction from the Pima Indians Diabetes dataset, which holds diagnostic records for 769 individuals across eight predictors. Four test-set cases were used, the same four for everyone in random order, chosen because a pilot indicated four gave an appropriate workload once participants also had to articulate their reasoning, and because prior studies with comparable designs used four. Only five features were displayed: fasting plasma glucose, insulin, BMI, diabetes pedigree function and age.

AI system. A random forest, selected over AdaBoost and XGBoost on performance, trained on a random 70% split and achieving precision 81% and recall 87%. Its predictions on the held-out cases were shown to participants verbatim.

Explanations. SHAP values rendered as bar plots, with features pushing towards a diabetes prediction coloured red and features pushing away coloured green, each bar labelled with its numeric contribution. Participants in the two XAI conditions received training on reading SHAP plots and had to pass a manipulation check; those who failed were excluded.

The reflection intervention. Four questions in three steps, derived from Pennycook's three-stage dual-process model, in which conflict detection is the early mechanism of Type 2 processing and cognitive decoupling the late one. Step 1 (Q1) asked whether the participant perceived any conflict between the AI's recommendation and explanation and their own initial thinking. Step 2 (Q2-Q3) asked them to write down what they had judged correctly and incorrectly, and which parts of the AI's reasoning they considered accurate or flawed. Step 3 (Q4) asked them to choose between their initial decision and the AI's recommendation and to justify that choice in writing. The steps were iterative, so a participant could return to Step 1 after Step 2.

Conditions. AI: after the independent Stage I judgement, only the AI recommendation is shown in Stage II. XAI: recommendation plus SHAP plot. XAI+Reflection: recommendation plus SHAP plot, followed by the four-question reflection before the final decision.

Procedure. Consent, random assignment, a pre-questionnaire covering demographics, AI background and diabetes health literacy. Stage I: for each of the four cases, a binary diagnosis made without AI, then beliefs about each feature's relationship to diabetes reported on sliders from -1 to 1, then confidence on a 7-point scale. Stage II: the same four cases revisited under the assigned condition, with the final decision and the belief sliders repeated. A post-questionnaire followed.

Measures. Decision accuracy against ground truth. Over-reliance as the percentage of cases where the participant agreed with the AI when it was wrong; under-reliance as the percentage where they rejected it when it was right. Need for cognition on six items adapted from Gabriel et al.; perceived understanding of AI on two items adapted from Wang and Yin; trust in AI on a single item; mental demand and system complexity adapted from Bucinca et al.; satisfaction adapted from Ma et al. All subjective measures on 7-point scales. Belief sliders were collected at both stages, giving a per-feature record of belief change.

Reflection worked, but the comparison that carries the most weight is one the authors report and then step past: explanations on their own nearly doubled over-reliance relative to an opaque AI, and the reflection prompts brought it back to roughly where the opaque AI had been.

Over-reliance was 0.58 (SD = 0.06) in the XAI condition, 0.33 (SD = 0.06) in the plain AI condition and 0.34 (SD = 0.06) in XAI+Reflection. The reflection-versus-XAI contrast is significant at p < 0.05, supporting H1a; the AI-versus-XAI contrast is significant at p < 0.01. Read together, the intervention did not push over-reliance below the no-explanation baseline, it repaired damage the explanations had done. The authors' framing, that reflection mitigates blind agreement with AI predictions, is true but understates what the AI arm shows about SHAP plots in this task.

On accuracy the intervention does beat both baselines, which is the stronger claim. Stage II accuracy was 0.881 (SD = 0.020) with reflection against 0.817 (SD = 0.018) for XAI and 0.810 (SD = 0.024) for plain AI, both contrasts at p < 0.05, supporting H1b. A regression on accuracy change controlling for Stage I accuracy and covariates reproduced the effect. Stage I accuracy did not differ across conditions (p = 0.627), so randomisation held and the Stage II differences are attributable to the manipulation. Under-reliance did not move, 0.05 (SD = 0.17) with reflection against 0.04 (SD = 0.16) for XAI, which matters because it rules out the cheapest explanation of a fall in over-reliance, namely blanket disengagement from the AI.

Both moderators behaved as predicted, and both are individual capacities rather than states. Among high need-for-cognition participants, reflection raised accuracy from 0.83 to 0.90 (p = 0.040) and cut over-reliance from 0.58 to 0.34 (p = 0.040); among low need-for-cognition participants neither contrast approached significance (p = 0.351 and p = 0.195). Among those reporting high perceived understanding of AI, reflection raised accuracy from 0.82 to 0.90 (p = 0.022) and cut over-reliance from 0.55 to 0.31 (p = 0.028); among those reporting low understanding, nothing (p = 0.202 and p = 0.147). H2a and H2b are supported. The practical implication is uncomfortable: a prompt designed to make people think harder helps those already disposed and equipped to think hard, and leaves the rest where they were.