Does Explainable AI Improve Human Decision-Making?
N = 300 Recidivism risk assessmentIncome predictionThis study in the framework
AI Accuracy was reported and allowed to vary naturally, rather than held constant or assigned. Two classifiers were selected after comparing logistic regression, a two-layer multi-layer perceptron, random forest and an RBF-kernel SVM: the MLP for Census income at 82% test accuracy and the RBF SVM for COMPAS at 68%. Both figures match published benchmarks, the COMPAS figure matching ProPublica's.
- H1: Explainable AI enhances decision-making process compared to only an AI prediction (without explanation) and a control condition with no AI.
- H1.a: A participant performs above chance in prediction tasks.
- H2: A participant's decision accuracy is positively associated with AI accuracy.
- H3: Average participant's decision accuracy does not outperform AI accuracy.
- H4: Participants outperform AI accuracy more often with explainable AI over AI prediction alone.
- H5: Participants follow explainable AI recommendation more often than AI only recommendation.
- H6: Explainable AI increases participants' decision confidence.
- H7: A participant's decision confidence is positively correlated with the accuracy of his/her decision.
"providing any kind of AI prediction tends to improve user decision accuracy, but no conclusive evidence that explainable AI has a meaningful impact"
abstract
"the strongest predictor for human decision accuracy was AI accuracy"
abstract
"If we did not have an AI only (decision aid) condition a reasonable but flawed inference would have been that explainable AI enhances decisions"
discussion
Experimental design
A 2 by 3 fully between-participants factorial experiment on Amazon Mechanical Turk, with 50 participants in each of six cells for a total of 300.
Factors. Dataset had two levels, Census income and COMPAS. AI condition had three levels: Control, with no prediction or information from the AI; AI, with an AI prediction only; and AI with Explanation, with the prediction plus an anchor LIME explanation.
The three-arm structure is the design's main methodological point. The authors argue that without an AI-only arm one would wrongly conclude that explanation improves decisions, because the comparison against a no-AI control conflates the value of the prediction with the value of the explanation.
Datasets. COMPAS, 6,479 instances and 7 features (gender, age, race, priors count, charge degree, risk score), with the class being whether the defendant re-offended within two years. Census income, 32,561 instances and 14 features (age, workclass, education, marital status, occupation, relationship, race, sex, capital gain, capital loss, hours per week, country), with the class being income at or below versus above 50K, preprocessed to give equal class distribution and adjusted for inflation to a present value of 88k. The two were chosen because they differ widely in feature count and in achievable classifier accuracy, so the findings would not be limited to a single dataset.
Explanations. Anchor LIME, a model-agnostic rule-based method producing if-then anchors, selected as one of the best available explanations and because rule-based explanations have been shown to give higher human precision than feature-based ones. Only one explanation type was used; the study does not compare explanation types.
Trials. Each trial presented a description of one individual and a two-alternative forced choice, with each option correct on 50% of trials so chance performance was exactly 50%. Fifty instances were sampled at random without replacement per participant, which means AI accuracy varied naturally across participants rather than being fixed. After each decision participants rated confidence from 1 (No Confidence) to 5 (Full Confidence), then received feedback on whether the choice was correct. Ten practice trials preceded 50 test trials, followed by a demographic questionnaire and free-text comments on strategy.
Full findings
Showing an AI prediction improved decisions; adding an explanation to it did not. The AI condition had a significant but small main effect on accuracy (F(2,294) = 8.19, p < .001, eta squared = .04), qualified by a larger dataset effect (F(1,294) = 46.51, p < .001, .12) and a significant interaction (F(2,294) = 8.38, p < .001, .05). Post-hoc tests located the benefit entirely in the Census condition, where both AI (p < .01) and AI with Explanation (p < .001) beat the control, but AI and AI with Explanation did not differ from each other (p = .28). On COMPAS no pairwise comparison was significant at all (all ps > .75). H1 was not supported.
The methodological lesson the authors draw is the row's most useful contribution: had the AI-only arm been omitted, the significant advantage of explanation over the no-AI control would have licensed exactly the wrong conclusion. Any study contrasting explanation only against an unassisted baseline cannot separate the value of the advice from the value of the explanation.
The strongest determinant of human accuracy was the AI's accuracy, not anything about the interface. Because 50 instances were sampled at random per participant, each participant experienced a slightly different AI accuracy, and regressing human accuracy on it produced a large main effect (beta = 0.70, p < .001, R squared = .28) within an overall model at R squared adjusted = .29. Explanation contributed nothing detectable to this relationship (ps > .60). H2 was supported.
Participants could partly tell when the AI was wrong, and explanation did not help them do it. Adherence was significantly higher when the AI was correct than when it was incorrect (F(1,196) = 36.15, p < .001, partial eta squared = .16), with better discrimination on the Census dataset (interaction F(1,196) = 9.01, p < .01). But there was no main effect of explanation on adherence (p = .62), so H5 was rejected. Complementarity was rare: only 14 of 200 participants (7%) exceeded their own AI's accuracy, and those were concentrated in the COMPAS AI-only cell (10 of 14), supporting H3 and giving no support to H4.
Confidence rose slightly with AI but not with explanation. The AI condition effect on mean confidence was small (F(2,294) = 3.58, p = .03, eta squared = .02), with only the control-versus-AI comparison significant (p < .03) and all others null (ps > .25), contradicting H6. Confidence was nonetheless well calibrated to accuracy within individuals (repeated measures correlation r = 0.48, p < .001), confirming H7.