Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

A moderated, think-aloud mixed-methods study with 26 participants. Explanation type was within-subjects; prediction task was between-subjects.

Tasks. Two were selected against four criteria: no specialised expertise required, hard enough that unaided accuracy would be low, each decision under a couple of minutes, and different data modalities. Income prediction: judge whether an individual earns more than $50,000 from a profile of age, class of work, education, marital status, occupation, place of birth, hours worked per week, sex and race. Data from Folktables, sampled from the 2018 US Census. A random forest reached 80.8% on held-out data. Biography classification: guess an individual's profession from their online biography, narrowed from the BIOS dataset's 29 professions to five (psychologist, physician, surgeon, teacher, professor). Bag-of-words embedding, random forest, 75.4% on held-out data.

Explanations. Feature-based used LIME. For income, coefficients were shown as a bar chart with direction fixed to mean increasing income, bars right for positive and left for negative, length showing magnitude. For biographies, the top ten highest-magnitude words were colour-highlighted in the text, red for supporting the prediction and blue for supporting other professions, with shade encoding magnitude. Example-based showed the two nearest neighbours from the training set by Euclidean distance, with their AI predictions and their ground-truth labels. The authors note that simply copying the nearest neighbours' ground truth would give only 63% on income and 60% on biographies, so extracting benefit required non-trivial reasoning.

Stimulus sampling. Sixteen instances per participant, drawn by stratified sampling as 10 on which the model was correct and 6 on which it was wrong, giving an experienced AI accuracy of 62.5% rather than the models' true 80.8% and 75.4%. Errors were deliberately over-sampled so overreliance could be observed; the authors chose this over training a weaker model because a weaker model's explanations and errors would be unrealistic for a deployable system. Within phase 3, each explanation type covered 8 instances, 5 AI-correct and 3 AI-incorrect.

Procedure, four phases, conducted online with camera and screen recording and moderated by the first author. Phase 1: five practice instances with correctness feedback, exposing participants to ground truth. Phase 2: the 16 instances unaided and without feedback, establishing a baseline and recording where the participant would initially agree or disagree with the AI. Phase 3: the same 16 instances in random order with AI prediction and explanation, split into two blocks of 8 by explanation type in randomised order, with think-aloud throughout. Phase 4: post-task interview. Mean duration 45 minutes; $35 compensation; IRB approved. A five-participant pilot informed the design, and participants often did not recognise that instances repeated.

Example-based explanations produced complementary human-AI performance on both tasks; feature-based explanations produced none, and actively hurt when the AI was wrong.

On income prediction, mean accuracy was 61.1% unaided, 60.6% with feature-based explanations and 71.1% with example-based. On biography classification the figures were 60.0%, 64.4% and 71.2%. Since the models' accuracy on the specific instances shown was 62.5%, the example-based condition exceeded both the human alone and the AI alone on both tasks. In the mixed-effect regressions against the No AI baseline, feature-based explanations had no significant effect (p = .88 income, p = .51 biography), while example-based explanations were marginally significant on income (p = .07) and significant on biographies (p = .04).

Splitting by AI correctness shows why. On instances where the AI was right, both explanation types raised accuracy (feature-based p = .06 income, p = .01 biography; example-based p = .01 income, p = .06 biography). On instances where the AI was wrong, feature-based explanations lowered accuracy below the unaided baseline (p = .09 income, p < .01 biography), whereas example-based explanations did not (p = .80 and p = .43), leaving participants at roughly their unaided accuracy. Feature-based explanations raise reliance indiscriminately; example-based explanations raise it selectively.

The think-aloud data supplies the mechanism, in the form of three types of intuition. Intuition about the outcome is a gut feeling about the answer, often formed before attending to the AI. Intuition about features concerns whether a feature's weight or relevance is plausible. Intuition about AI limitations covers perceived signals that a particular prediction is unreliable, plus general beliefs about model bias, inability to handle context, and poor performance on rare instances.

From these the authors derive three pathways by which a decision-maker overrides the AI: forming a strong outcome intuition that disagrees with the prediction; using feature intuition to find evidence in the explanation that discredits the prediction; and recognising signals of prediction unreliability. They are careful that these are not mutually exclusive, cannot be quantified from this data, and that overriding the AI is not automatically appropriate.

The pathways explain the quantitative result through an interface effect. Participants were more likely to report weak outcome intuition when working with feature-based explanations, which the authors attribute to those explanations being visually dominant, especially the text highlights overlaid directly on the biography. One participant described their approach as scanning the pink and the blue instead of reading. Example-based explanations, by contrast, let participants form their own view of the instance first and even skip the explanation entirely when their intuition was strong. Weak outcome intuition then leads to deference: participants who found no evidence of error in an explanation they could not reason about meaningfully defaulted to the AI.

The two explanation types also differed in how well they signalled unreliability, which is Pathway 3. With example-based explanations most participants noticed when the AI had been wrong on the similar cases shown, a direct and legible cue. With feature-based explanations only a few participants picked up the much subtler cue of weights not trending strongly in either direction. Notably, the AI-limitations pathway was used primarily by participants with ML experience.