Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making

Two randomised controlled experiments, one per decision task, each between-subjects with five treatments. Conducted on Amazon Mechanical Turk, US workers only, each worker permitted to participate once.

Tasks. Recidivism prediction: participants review a criminal defendant profile of seven features, namely race, sex, age, number of non-juvenile prior crimes, name of the current charge, degree of the current charge and days in custody for the current charge, and predict whether the defendant will reoffend within two years. Profiles come from the COMPAS dataset of 7,214 defendants in Broward County, Florida, 2013 to 2014. Forest cover prediction: participants review a geological profile of a 30 m by 30 m wilderness cell with eight features, namely elevation, aspect, slope, hillshade index, horizontal and vertical distance to surface water, horizontal distance to roadway and horizontal distance to wildfire ignition points, and predict whether the area is primarily covered by spruce-fir forest. Profiles come from the UCI cover type dataset of 581,012 areas in the Roosevelt National Forest, Colorado, with the original six-way species label reduced to a binary judgement.

Operationalising perceived domain expertise. The two tasks were chosen because laypeople feel differently knowledgeable about them, and the difference was validated rather than assumed. In the pilot, 82.6% of 98 workers reported being more knowledgeable on recidivism prediction, 63.3% believed they personally would predict it more accurately, 71.4% believed a normal person would, and 71.4% expected to feel more confident on it.

Treatments. Five arms per task, varying whether and how the model's prediction was explained. No explanation, the control, shows the prediction alone. Feature importance shows a bar chart of permutation feature importance, computed as the increase in prediction error after permuting a feature; this is a global explanation. Feature contribution shows a bar chart of each feature's log-odds contribution to this prediction plus the base rate; local. Nearest neighbors shows, alongside the current profile, two correctly predicted training profiles, the most similar one carrying the same prediction and the most similar one carrying the opposite prediction; local and example-based. Counterfactuals shows, for each feature, the smallest change that would flip the prediction, or a statement that changing it would not.

AI system. A logistic regression model per task, trained on a subset of the respective dataset. Accuracy on 1,000-instance hold-out test sets was 69.1% for recidivism and 69.5% for forest cover.

Task instances. All treatments within a task saw the same 32 instances with the same model predictions. Instances were selected rather than sampled: model confidence was dichotomised at a 0.7 probability estimate, confirmed to align with correctness likelihood, and 16 low-confidence and 16 high-confidence instances were chosen as prototypes covering the centres of the data distribution for each confidence stratum. Model confidence was never displayed to participants; the design requires them to infer it from the explanation.

Procedure. Background survey covering demographics, technical literacy and machine-learning expertise; interactive tutorial with worked examples of the assigned explanation type and qualification questions; for forest cover only, an additional introduction to spruce-fir characteristics and 10 unassisted training tasks with immediate correctness feedback. Then 32 tasks in randomised order, each following a three-step protocol: make an independent prediction, see the model's prediction and explanation, make a final prediction. No accuracy feedback was given on any of the 32 tasks. An exit survey followed with nine multiple-choice questions on model behaviour and two 7-point self-report items on perceived understanding.

Incentives. Base payment $1.80 for recidivism and $2.00 for forest cover, the latter higher because of the added training tasks. Performance-contingent bonuses: $0.03 per correct final prediction conditional on overall final accuracy of at least 60%, and $0.10 per correct exit-survey question about model behaviour. Maximum total bonus $2.26.

Measures. Objective understanding as the number of the nine multiple-choice questions answered correctly, spanning comparison of feature importance, a feature's marginal effect, counterfactual thinking, simulation of model behaviour and error detection. Subjective understanding as the mean of two 7-point items. Reliance as agreement fraction, the proportion of tasks where the final prediction matched the model's. Appropriate trust as the proportion of tasks where the participant used a correct model prediction and did not use an incorrect one, which the authors note is effectively final decision accuracy; overtrust as the proportion of tasks using a wrong model prediction; undertrust as the proportion not using a correct one.

Domain expertise gates whether explanations do anything at all. On the task where laypeople feel knowledgeable, several explanations satisfied several desiderata; on the task where they do not, none of the four reliably satisfied any of the three. This is the paper's central result and it holds across all three research questions.

Understanding (RQ1). On recidivism prediction, feature importance and counterfactual explanations raised objective understanding (Cohen's d = 0.26, 95% CI [0.03, 0.48] and d = 0.27 [0.04, 0.48]), and all four types raised subjective understanding (d = 0.28 [0.05, 0.49] aggregated). On forest cover, only feature importance raised objective understanding (d = 0.33 [0.06, 0.59]) and only feature contribution raised subjective understanding (d = 0.28 [0.01, 0.55]). The mixed-effects regressions reproduced this pattern. Two incidental covariate effects recur across both tasks and are worth recording: women scored higher on objective understanding than men, and participants reporting higher education scored lower.

Uncertainty awareness (RQ2). Model confidence was never shown, so any sensitivity to it had to be inferred from the explanation. On recidivism, participants in the control arm relied on high-confidence and low-confidence predictions about equally, while every explanation type pushed them toward relying more on high-confidence predictions (d = 0.20, 95% CI [-0.02, 0.42] aggregated; interaction coefficients between confidence and explanation type all reliably positive, feature contribution largest at beta = 0.27 [0.12, 0.42]). On forest cover, none of the four interaction coefficients could be distinguished from zero. Without domain knowledge, participants could not read model uncertainty off the explanation at all.

Trust calibration (RQ3). On recidivism, feature importance and feature contribution both slightly raised appropriate trust (d = 0.19 [-0.05, 0.41] and 0.19 [-0.03, 0.40]) and lowered undertrust, but feature importance bought this partly with increased overtrust (d = 0.15 [-0.08, 0.36]), specifically on the model's low-confidence predictions (d = 0.19 [-0.03, 0.42]). In the regression, only feature contribution raised appropriate trust without a matching rise in overtrust or undertrust (undertrust beta = -0.03 [-0.05, -0.01]). Nearest neighbors and counterfactuals were inconclusive. On forest cover, none of the four helped, at any level of model confidence.

The counterfactual result is the sharpest negative. Counterfactual explanations are the ones usually held to resemble how humans explain decisions, and they did raise understanding on the familiar task, but that understanding did not convert into better trust calibration. Satisfying one desideratum does not imply satisfying another, which is the authors' warning against treating decision accuracy as evidence that explanations are working.

Feature contribution emerges as the best of the four, but only where participants had some domain expertise. The only effect consistent across both tasks is narrow: feature importance raises objective understanding and feature contribution raises subjective understanding.

The authors offer a cognitive-load mechanism for the expertise gate. Without domain knowledge the explanation must be processed entirely in working memory, and participants in the forest cover task had to learn what the features even meant before they could use them. They also point out that model uncertainty was in principle recoverable from the explanations without any domain knowledge, by summing feature contributions toward the base rate or by counting counterfactuals and their magnitudes, and that participants did not do this. The information was present and went unused.