Effects of XAI on trust and human behavior in a high-risk decision task
N = 410 Mushroom classificationThis study in the framework
Held constant, and deliberately mediocre. A ResNet50 backbone with ImageNet pre-trained weights, retopped with a flattening layer and two dropout and linear layers, fine-tuned with Adam and categorical cross-entropy to classify 18 mushroom species. Training used 3480 images, with 241 test and 219 validation images, mostly from the Danish Mycological Society database and topped up from Google Images and Wikimedia to at least 80 images per species, with class weighting to offset imbalance. Accuracy was 94% on training data, 82% on validation and 71% on test. The authors state plainly that 71% would be far too low for a real mushroom assistant and that the low accuracy was retained because it generates the incorrect predictions the study needs.
- RQ1. Explore the effects of an XAI interface with attribution-based and example-based visual explanations by comparing it with an interface that does not explain its recommendations.
- RQ2. Explore the effects of an educational intervention presented as texts about and illustrations of how machine learning works by comparing two groups, one with and one without such an intervention.
- RQ3. Understand the underlying psychological mechanisms by assessing users' prior knowledge of AI, trust, and domain-specific knowledge about mushrooms as major factors commonly used to explain differences in comprehension and system use.
"we found that the educational intervention, domain-specific (i.e., mushroom) knowledge, and AI knowledge had no effect on performance"
abstract
"the results show that participants with visual explanations trusted the AI classification significantly less and reported to understand it less. This could mean that the visual explanations prevented overtrust and thus led to more adequate trust"
discussion
"participants performed better with visual explanations, and showed lower trust and lower intentions to use the app"
discussion
Experimental design
A 2 by 2 between-subjects online experiment crossing an educational intervention about how the AI works (present, absent) with visual explanations of the AI's classification (present, absent). N = 410 after exclusions, with n = 200 receiving the educational intervention and n = 210 not, and n = 202 receiving explanations and n = 208 not.
Task. A newly developed mushroom-picking task, 15 items presented in randomised order on separate pages with no back-navigation. For each mushroom photograph the participant made two separate decisions: an edibility assessment (edible or poisonous) and a take-home decision (pick it or leave it). The two can be right or wrong independently, and the authors weight the errors asymmetrically, noting that leaving an edible mushroom costs only enjoyment while taking a poisonous one may cost health. Stimuli were selected so that both edible and poisonous mushrooms appeared, so that AI certainty varied across items, and, critically, so that the classifier gave a false recommendation on at least 5 of the 15 items, which the authors state was required in order to allow overtrust to be measured. Ten items were retained for analysis after removing those with extreme difficulty, with at least four poisonous and at least four incorrectly classified items preserved. Item selection was performed before further analysis.
The interface. A fictitious app, Forestly, showing the top three species predicted by a real classifier with a certainty percentage for each, plus a bar chart and numeric value for the total predicted probability of edibility. In the explanation condition the interface additionally displayed a Grad-CAM attribution map over the image and, for each of the top three species, the most similar training image, an example-based technique adapted from ExMatchina.
The educational intervention. A short text-and-illustration introduction to machine learning covering high-level concepts of image classification networks, given before the task to one group only.
Measures. Edibility accuracy and picking accuracy, each summed over items and also computed on the poisonous-only and incorrect-AI subsets. Per-item single-item trust ("I TRUST this mushroom identification of the AI") and self-reported comprehension ("I UNDERSTAND how the AI arrives at this mushroom classification") on 5-point scales. A five-item task-specific AI comprehension test administered after the task, an eight-item general AI knowledge test and an eleven-item Austrian mushroom knowledge test developed with mycologists, both administered before. App evaluation and intention to use, before and after.
Full findings
Explanations improved decisions while reducing trust, self-reported understanding, app evaluation and intention to use. Everything else the authors expected to matter, including teaching people how the AI works, did nothing.
Explanations improved accuracy. On the edibility assessment, participants with visual explanations outperformed those without across all 10 analysed items (BM(398.68) = 4.65, p < .001, d = .44, mean 5.29 (SD 1.36) against 4.72 (SD 1.22)) and, more importantly, on the four items where the AI recommended a false classification (BM(401.73) = 3.48, p = .004, d = .35, mean 1.00 (SD .97) against .69 (SD .84)). The benefit is therefore concentrated exactly where blind acceptance would hurt. On the take-home decision the effect was weaker: significant over all 10 items (BM(400.71) = 2.69, p = .03, d = .26) but not on the poisonous subset or the incorrect-AI subset. The authors read the gap as edibility assessment being a maximal-effort measure while picking intention also reflects traits such as risk aversion.
Explanations reduced trust and perceived understanding. Participants without explanations trusted the system more (M = 3.64, SD .74) than those with them (BM(403) = 3.91, p < .001, d = -.36) and reported understanding it better (M = 3.57, SD .87; BM(404) = 2.49, p = .02, d = -.21). The authors interpret this as better calibration rather than as damage: the explanations exposed the fallibility of a 71%-accurate classifier, so lower trust is the appropriate response, and it coincides with better performance on precisely the items where the AI was wrong.
The same reversal appears in system evaluation. Participants without explanations rated the app better (M = 4.12, SD .82 against M = 3.77, SD 1.03; BM(390) = 3.44, p = .003, d = .38) and reported higher intention to use it (M = 3.58, SD 1.10 against M = 3.24, SD 1.23; BM(404) = 2.73, p = .01, d = .29). Subjective preference and objective performance point in opposite directions across the whole study.
The educational intervention did nothing. No significant effect on any dependent variable, on any item subset: edibility across all 10 items BM(403.89) = .59, p = .56; picking intention BM(407.75) = 0.53, p = .77; and nothing on evaluation or intention to use. Teaching people in the abstract how image classifiers work did not transfer to using this one. The authors are careful not to generalise from one implementation and attribute the null partly to the brevity of the intervention.
Prior knowledge did nothing either. Neither mushroom knowledge (beta = .06, p = .25) nor AI knowledge (beta = .05, p = .30) predicted edibility performance, and neither predicted picking behaviour (beta = .02, p = .75 and beta = .08, p = .11). Explainability was a significant predictor in both models (beta = .22, p < .001 and beta = .13, p = .007) while the educational intervention was not. Both regression models fit poorly (edibility F(4,405) = 5.73, p < .001, R2 = .05; picking F(4,405) = 2.43, p = .05, R2 = .02), so almost none of the variance in performance is explained by anything measured. The authors offer two candidate reasons: the new knowledge tests had low internal consistency (omega .42 to .50) and may not discriminate at the higher expertise levels that mushroom picking demands, and the static-screenshot design gave AI-knowledgeable participants no way to probe the system, which is how such knowledge would ordinarily be applied.
The overall shape is a case-level intervention working where a general one failed: the explanation attached to the individual prediction changed behaviour, while education about the class of system did not.