Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Logical / reasoning taskThis study in the framework
Manipulated. Marvin was correct 80% of the time initially and was updated to 85% at cycle 75 of 150. The update simultaneously changed where the errors fell, in one of three between-groups conditions: the same error boundary, a compatible boundary that introduced no new errors, and an incompatible boundary that introduced new errors. Error-boundary complexity, set by the number of human-visible features from three to six, and error stochasticity, set across four pairs of conditional error probabilities, were varied separately. In the accompanying simulation, compatibility scores produced by standard retraining ranged from 0.40 to 0.76 across datasets and model classes.
Q1. Do better mental models of AI lead to higher team performance? Q2. Do more compatible updates lead to higher team performance than incompatible updates? Q3. Do current machine learning classifiers produce compatible updates? Q4. Does there exist a tradeoff between the performance and the compatibility of an update to AI? Q5. What is the relative performance of the different dissonance functions?
"We show that updates that increase AI performance may actually hurt team performance."
abstract
"A more accurate but incompatible classifier results in lower team performance than a less accurate but compatible classifier (no update)."
results, Q2
"Unless the particular domain and the interaction allows the human to validate the correctness of the machine recommendation efficiently and effectively, extracting benefits from collaboration with the AI system depends on the human developing insights (i.e., a mental model) of when to trust the AI system with its recommendations."
introduction
"Thus, compatibility of updates becomes an essential determinant of team performance, and developers should factor it in system design."
discussion
Experimental design
Two-part study. The human component used CAJA, a web-based assembly-line game in which a participant and an AI assistant named Marvin classify box-like objects as defective or not defective. Object labels are generated at random so that no participant can become expert at the task itself; the only learnable structure is when Marvin errs. For each object the AI recommends a label and the participant either accepts it or pays an opportunity cost to use a compute option that always returns the correct answer. The reward matrix was set to imitate high stakes: $0.04 for a correct decision against minus $0.16 for an incorrect one, so that a pure accept or pure compute strategy has an expected value of zero and the only way to score is to learn when to trust Marvin.
Participants were crowdworkers recruited on MTurk, 25 per condition, with the bottom quartile removed as suspected spammers. Marvin was correct 80% of the time initially and its errors followed a logical formula over object features, which participants were never shown.
Three factors were varied. Task complexity was varied through the number of human-visible features, from three to six. Error-boundary stochasticity was varied across four settings of the conditional error probabilities, three one-sided and one two-sided. Update type was varied between groups: at cycle 75 of 150 the classifier was updated from 80% to 85% accuracy, either with the error boundary unchanged, with a compatible boundary that introduced no new errors, or with an incompatible boundary that introduced new errors. A no-update baseline was extrapolated from the first study.
Full findings
Team performance fell as the number of human-visible features rose from three to six, and fell further as the error boundary became more stochastic, with the sharpest drop for two-sided errors. Participants who self-reported a correct mental model of the error boundary scored highest, followed by those with a partially correct model; unsure and wrong models scored lowest. Mental model quality therefore stands between AI accuracy and team performance rather than accuracy acting on the team directly.
The update result is the central one. An update that raised accuracy from 80% to 85% while introducing new errors produced lower team performance than no update at all, because participants had to unlearn and relearn the error boundary. An update that raised accuracy without introducing new errors improved team performance. The interaction shows four stages: learning the original boundary, stability, disruption at the update, and restabilisation.