Explanations Considered Harmful: The Impact of Misleading Explanations on Accuracy in Hybrid Human-AI Decision Making
N = 22 Logical / reasoning taskThis study in the framework
Held constant and scripted. The system was a Wizard-of-Oz simulation of ChatGPT Plus built in Figma, with no underlying model, so every participant saw the identical sequence of 19 responses: 13 correct and 6 incorrect, an accuracy of about 68%. Participants were informed of the AI's accuracy, and a concurrent study by the same authors in the same setting established that unaided participants got slightly more than half the items wrong against a 70% accurate system, so the AI held a real advantage. What varied was not the advice but the explanation attached to it. Within the 13 correct-advice items, 7 carried explanations that were both coherent with the advice and relevant to the case, and 6 carried explanations that were coherent with the advice but irrelevant to the case. All 6 incorrect-advice items carried misleading explanations, so the design contains no cell in which a good explanation accompanies bad advice.
- RQ1: Does a correct AI advice coupled with a misleading explanation affect user accuracy? In particular, would misleading explanations induce users in error? And is there any user strata that is more susceptible to this effect?
- RQ2: Does a correct AI advice coupled with a misleading explanation affect user confidence in their response?
"participants were influenced by the misleading explanations to the extent that they did not verify the correctness of the advice, despite its accuracy"
abstract
"misleading explanations seem to have no bad smell, so to say"
discussion
"misleading explanations are more harmful for low-performers than top-performers"
discussion
Experimental design
A within-subjects study with no control group, run in person. Twenty-two participants each answered 19 items, all seeing the same sequence.
Participants. Twenty-two Master's students enrolled in an Artificial Intelligence Master's degree, recruited and run in person in university spaces.
Task. Nineteen moderate-to-hard logic puzzles sourced from the Youmath website, of the kind used in psychometric assessment of the General Intelligence Factor. Types included numerical and alphanumeric logic, deductive reasoning, graphic interpretation and anagrams. Each puzzle was answered by choosing one of four options.
Apparatus. A simulated ChatGPT Plus interface built in Figma, that is a Wizard-of-Oz setup with no underlying model. Participants engaged with each puzzle by uploading an image of the pattern task or typing out the word puzzle, after which the simulated system returned an answer together with an explanation. Responses and confidence were collected in a LimeSurvey questionnaire, confidence on a four-value ordinal semantic differential from 1 (not sure at all) to 4 (almost certain), the four-point scale chosen to mitigate central tendency bias.
The manipulation. All 19 system responses were scripted in advance and identical for every participant. Thirteen carried correct advice and six carried incorrect advice. Within the 13 correct-advice items, 7 came with correct explanations, meaning coherent with the advice and relevant to the case, and 6 came with misleading explanations, meaning coherent with the advice but irrelevant to the case. All 6 incorrect-advice items came with misleading explanations. The comparison that carries the paper is therefore within-subject and within the correct-advice items only: 7 items with good explanations against 6 items with misleading ones.
An example of the manipulation: for an odd-one-out puzzle the system correctly advises discarding the word flowers, but justifies it by saying the remaining words denote colours, when in fact the remaining words are all names of flowers. The advice is right, the reasoning offered for it is irrelevant to why it is right.
Trust establishment. Nine of the 13 correct answers were placed consecutively at the start of the sequence, deliberately, to establish trust and prevent negative bias or poorly calibrated distrust from forming before the manipulated items appeared.
Analysis. Hypothesis testing at a .95 confidence level with alpha .05. Mann-Whitney U tests with normal approximation on average error rates and on average confidence, chosen because the data are respectively non-normal and ordinal. Reliance patterns were additionally summarised with the group's own Technology Impact and Conservatism Bias diagrams, both expressed as odds ratios. Participants were stratified into top and low performers using the extreme quartiles of the accuracy distribution.
Full findings
A correct recommendation accompanied by an irrelevant explanation produced significantly worse decisions than the same correct recommendation with a relevant one. Accuracy on the correct-advice-plus-correct-explanation items was significantly higher than on the correct-advice-plus-misleading-explanation items (Mann-Whitney U, p < .001), with a large standardised effect size of 0.51 and a common language effect size of 0.79. The advice was right in both sets of items; only the reasoning offered for it differed.
The authors stress why this is not obvious. These are logic puzzles with verifiable answers: a participant could have checked the advice against the puzzle directly and would have found it correct. Instead they appear to have taken the explanation as a proxy for the advice's soundness and stopped short of verifying it. The authors describe this as the explanation generating a kind of temporary blindness about the correctness of the advice, and name the pattern the XAI halo effect.
The design recommendation the authors draw is to withhold explanations rather than risk poor ones: XAI systems should not generate an explanation when the advice module's confidence falls below a conservatively high threshold.