Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making

Two randomised between-subjects experiments on Amazon Mechanical Turk sharing one task and one model.

Task. Income prediction from the 1994 Census Adult Data Set in the UCI repository (48,842 records). Participants predicted whether a person's annual income exceeded $50K from eight attributes, selected as the eight most important of the original fourteen by the feature importance of a gradient boosting decision tree. Ground truth was the recorded income category. Each profile was shown as a table listing the eight attribute values alongside, for each value, the chance out of 10 that a person with that attribute value earns above $50K, computed from the training data and expressed as a frequency rather than a probability. For continuous attributes the chance was computed over a range revealed on hover. The authors describe these chance numbers as analogous to the historical statistics a real expert would have, and note that participants still had to combine single-attribute probabilities themselves.

Experiment 1. N = 72, a 2 by 2 by 2 between-subjects design with 9 participants randomly assigned to each of the 8 cells. Factor 1, show versus not show AI confidence: confidence was the model's predicted probability for the most likely outcome, stated as a frequency in the form that the model's prediction is correct N times out of 10 on individuals similar to this one. Factor 2, show versus not show AI prediction: in the show condition participants saw the model's prediction before committing; in the not-show condition they had to decide whether to delegate the trial to the AI without seeing its output, which the authors treat as a stricter test of trust. Outcome feedback was given in both, so participants experienced the AI's performance either way. Factor 3, full versus partial model: the partial model was retrained without marital status, the single most important attribute, and participants in that condition were told so explicitly and shown marital status in the profile table labelled as extra information for you. The point was not to test an inferior model but to test the effect of knowing one holds knowledge the AI lacks.

Procedure. Instructions tailored to condition, then 20 training trials with full feedback on the true income, the AI's prediction and, where applicable, the AI's confidence, followed by a report of the participant's and the model's accuracy over the last 10 training trials. Then 40 task trials in randomised order. On each trial the participant made an independent prediction first, then saw the assigned AI information, then chose their own or the model's prediction as the final answer; where the two already agreed, the agreement was taken as final automatically. Feedback on both the participant's and the model's correctness followed every trial. A 10-second countdown gated the submission button on each trial to encourage attention. Base pay was $3 with a bonus of 5 cents for each correct final prediction and a penalty of 2 cents for each incorrect one; average bonus was $1.16 and average total compensation $4.16 for roughly half an hour.

Experiment 2. N = 9, run only in the full-model, with-prediction cell, with the confidence score replaced by a local explanation: a bar chart of each attribute's contribution to this particular prediction computed by the Shapley method, with orange bars indicating attributes pushing towards income above $50K, blue bars pushing below, and a base-chance bar. The without-prediction condition was not run because participants could reconstruct the prediction from the direction of the bars, and the partial model was not run because model completeness had shown no effect. Analysis pooled these 9 participants with the baseline and with-confidence cells of experiment 1 that shared the same full-model, with-prediction setting.

Outcomes. Trust was measured behaviourally, not by self-report, on the explicit grounds that self-reported trust is an unreliable indicator of trusting behaviour. Two indicators were used: switch percentage, the share of trials on which the participant took the AI's prediction as final, computed among disagreement trials in the with-prediction conditions and among all trials in the without-prediction conditions; and agreement percentage, the share of trials on which the final prediction matched the AI's. Accuracy was recorded separately for the participant's own prediction, the AI's prediction and the final AI-assisted prediction. Analysis used four-factor and three-factor ANOVA with Tukey HSD post-hoc tests.

Confidence scores calibrated trust but did not improve decisions, and local explanations did neither. This dissociation is the paper's contribution.

Displaying confidence significantly raised overall reliance on the AI (main effect on switch percentage F(1,64) = 4.64, p = .035) and, more importantly, made reliance track the model's confidence: the interaction between showing confidence and the model's confidence level was strong (F(4,256) = 15.8, p < .001). Below 60% model confidence participants trusted the model slightly less when the score was shown; above 80% their trust rose sharply. Without the score, reliance was flat across all five confidence bins (F(4,128) = 1.84, p = .126), meaning participants had no independent read on when the model was reliable. The same pattern held on agreement percentage (interaction F(4,256) = 3.82, p = .005). H1 was supported.

The calibration effect survived blind delegation. Whether or not the AI's prediction was visible made no difference to reliance (F(1,64) = 0.217, p = .643) and did not moderate the calibration effect (three-way interaction F(4,256) = 0.266, p = .899; on agreement percentage F(4,256) = 0.331, p = .857). High confidence scores induced participants to hand the decision to the AI without seeing what it would say, which answers RQ1 and is the strictest version of the trust result.

None of this improved accuracy. AI-assisted accuracy did not differ significantly across the confidence or prediction conditions; only the model's confidence level itself (F(4,256) = 79.6, p < .001) and its interaction with model completeness (F(4,256) = 2.95, p = .021) mattered. The difference between AI-assisted accuracy and AI-alone accuracy was unaffected by any factor. H2 was rejected.

The mechanism the authors identify is the alignment of human and AI error boundaries. Participants' unaided accuracy was 65% against the AI's 75% on the stratified trial set, so the AI held an advantage but not a large one. Critically, the trials on which the model was uncertain were also the trials humans found hard: humans were less accurate than the AI at every confidence level, with the gap merely narrower at low confidence. Better calibration therefore had nowhere to pay off. In the high-confidence zone where trust rose, human and AI rarely disagreed to begin with; in the low-confidence zone, the human's judgement was not a better substitute. The authors are explicit that a setting where human and model held genuinely complementary knowledge might yield a different answer.