Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
N = 199 Deception detectionThis study in the framework
The model is a support vector machine trained on the Ott et al. deceptive hotel review dataset, reaching 86% accuracy, which the authors note is in line with published work on this task. Its LIME explanations are computed from that model and are therefore genuine. The AI as experienced by participants performed at 50%. From a 32-review test set the authors drew four reviews from each cell of the confusion matrix, true positive, false positive, true negative and false negative, balanced two-and-two on sentiment, giving the 16 reviews each participant saw. Balancing the confusion matrix by construction sets the down-sampled AI performance to chance in both conditions.
H1a: Providing explanations of the AI advisor influences the relative self-reliance (RSR). H1b: Providing explanations of the AI advisor increases the relative AI reliance (RAIR). H2: Providing explanations of the AI advisor increases the change in self-confidence. H3a: An increased change in human self-confidence increases the relative self-reliance (RSR). H3b: An increased change in human self-confidence increases the relative AI reliance (RAIR). H4: Providing explanations of the AI advisor increases trust in the AI advisor. H5a: Trust decreases the relative self-reliance (RSR). H5b: Trust increases the relative AI reliance (RAIR).
"explanations of AI decisions can reduce the share of under-reliance"
results, Appropriateness of Reliance
"RAIR is not increased simply by relying more often on AI advice, as this would have also reduced the RSR significantly"
results, Appropriateness of Reliance
"The human-AI team performance is not significantly different from the human accuracy which means we do not reach CTP and therefore AR is not displayed"
results, Appropriateness of Reliance
Experimental design
A between-subjects online experiment on Prolific with two conditions and 199 participants analysed.
Task. Deceptive hotel review classification: judge whether a given review is deceptive or genuine. Stimuli come from the Ott et al. dataset of 400 deceptive reviews written by crowdworkers and 400 genuine ones, so ground truth is known by construction.
AI advisor. A support vector machine achieving 86% accuracy, comparable to published work on this dataset.
Explanations. LIME feature importance, chosen as the most common technique for textual data. Influential words are highlighted, with the direction of the effect indicated and magnitudes bucketed into three effect sizes, following the interface design of Lai et al.
Conditions. Control, in which participants receive the AI's advice as a bare statement such as that the AI predicts the review is fake. Feature importance, in which the same advice is accompanied by the LIME highlighting. Both arms therefore receive AI advice; only the explanation varies.
Stimulus sampling. From a test set of 32 reviews the authors drew four from each cell of the confusion matrix (true positive, false positive, true negative, false negative), two with positive and two with negative sentiment, giving 16 reviews per participant. Balancing the confusion matrix means the AI as experienced by participants performed at 50%, not 86%, even though the underlying model is strong and its explanations are genuine.
Procedure. An attention check on the colour of grass, random assignment, a task introduction covering the AI and, where applicable, its explanations, then two training tasks with feedback. Participants were given a general intuition of the AI but no performance figure. The 16 main tasks then ran as a strict two-step sequence: the participant reads the review alone, classifies it, and rates confidence on a 7-point scale; the AI advice then appears, with or without explanation; the participant may revise the classification and re-rates confidence. No feedback was given during the main tasks. Trust and demographics were collected afterwards.
Measures. RAIR and RSR as defined in the paper. Change in self-confidence as the summed difference between post-advice and pre-advice confidence across all 16 instances, on 7-point scales. Trust as a latent construct from four 7-point items (Cronbach's alpha 0.89). Human accuracy from the initial decisions and AI-assisted accuracy from the revised decisions.
Full findings
Explanations improved reliance on one dimension only, and the gain was real rather than a byproduct of general compliance.
Relative AI reliance rose from 29.59% in the control condition to 38.87% with feature importance explanations (t = -1.95, p = .05), while relative self-reliance was unchanged (71.87% against 69.45%, t = 0.61, p = .54). The authors make the interpretive point that matters: if explanations had simply made people more compliant, RAIR would have risen and RSR would have fallen. Because RSR held, the extra reliance was discriminating. Explanations reduced under-reliance without creating over-reliance. H1b supported, H1a not.
The baseline itself is informative. In the control condition participants already showed high self-reliance (71.87%) and low AI reliance (29.59%), which the authors read as a severe share of under-reliance: people rejected the AI's correct advice roughly seven times in ten when they had initially been wrong.
The accuracy consequence is small but real. Neither human accuracy nor AI-assisted accuracy differed significantly between conditions, but the change from the participant's own accuracy to their assisted accuracy did: +2.45 percentage points with explanations against -1.56 without (t = 2.29, p = .02). Without explanations, taking AI advice made participants worse than they had been; with explanations it made them better.
The headline null is that none of this reached appropriate reliance as the authors define it. Because the confusion-matrix sampling put the experienced AI at 50%, and human-AI team performance did not significantly exceed human performance alone, complementary team performance was not achieved and, by their own definition, appropriate reliance was not displayed in either condition.
The structural model localises the mechanism, and it is not trust. Explanations did not increase trust at all (standardised path -0.04, not supported), so H4 failed. Trust nonetheless had the strongest effects in the model once present, lowering RSR (-0.05) and raising RAIR (0.06), both at the highest significance level, so trust drives reliance in both directions but explanations do not drive trust.
What explanations did move was confidence. They increased the change in self-confidence (0.13, supported), and that change in turn raised RAIR (0.11, supported) but had no effect on RSR (0.00, not supported). The path from explanation to better reliance therefore runs through self-confidence, not through trust in the system, and it operates only on the dimension of taking good advice rather than on the dimension of resisting bad advice.