Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Logical / reasoning taskThis study in the framework
Manipulated, but through the shape of the error boundary rather than through headline accuracy. Accuracy was deliberately held constant across several comparisons so that the effect of error-boundary structure could be isolated. Three properties were varied: parsimony, set by the number of conjunctions and literals in the formula describing where the AI errs; stochasticity, set by the conditional error probabilities across four configurations, one non-stochastic, two one-sided and one two-sided; and task dimensionality, set by the number of human-visible features.
- RQ1. Do people create mental models of the error boundary? How do mental models evolve with interaction?
- RQ2. Do more parsimonious error boundaries facilitate mental model creation?
- RQ3. Do less stochastic error boundaries lead to better mental models?
"We find that systems with exactly the same accuracy can lead to different team performance depending upon the parsimony, non-stochasticity, and dimensionality of error boundaries."
conclusion
"Achieving high team performance depends on more than just the accuracy of the AI system: Since the human and the AI may have different expertise, the highest team performance is often reached when they both know how and when to complement one another."
abstract
"While AI accuracy has been traditionally considered a convenient proxy for predicting human-AI team performance, our findings motivate investing effort to understand how to develop AI systems to support teamwork."
conclusion
Experimental design
Controlled user studies on CAJA, the assembly-line platform introduced in the companion AAAI paper. Participants decide whether objects passing along a pipeline are defective. For each object an AI assistant named Marvin recommends a label and the participant either accepts it or clicks compute, which returns the correct answer at an opportunity cost. Feedback and monetary reward follow each decision. The payoff matrix imitates high stakes: $0.04 for accepting Marvin when correct against minus $0.16 for accepting Marvin when wrong. The decision boundary of the underlying task is generated at random, so participants cannot become expert at the task itself and the only route to a high score is learning where Marvin errs. Machine-visible features are a superset of human-visible ones.
Three properties were varied. Parsimony of the error boundary was set by the number of conjunctions and literals in the logical formula describing where Marvin errs, comparing one conjunction of two literals against two conjunctions of two literals each. Task dimensionality was set by the number of human-visible features. Stochasticity was set by the pair of conditional error probabilities, using four configurations: one non-stochastic, two one-sided, and one two-sided in which errors also occur where the formula is not satisfied. Isomorphic error boundaries were assigned at random across workers so that feature salience did not confound the comparison.
Full findings
Participants do build mental models of the error boundary, and they build them gradually. Average simple regret, the gap between the observed reward and that of an oracle with access to the true error boundary, fell as rounds accumulated. A trace of one worker shows the mechanism: an initially over-fitted model, errors on the cases it excluded, surprise, and eventual generalisation to the correct rule. Partially correct models fail in two distinct ways, over-generalisation and over-fitting.
Parsimony matters. Team performance was higher with a single-conjunction error boundary than with a two-conjunction boundary of the same number of literals. Performance also fell as task dimensionality rose, and for a fixed number of features rose with the number of literals, because more literals make the error region more specific.
Stochasticity matters more sharply. For one-sided boundaries the proportion of workers choosing the optimal action climbed towards 100% over rounds. For the two-sided boundary it stayed near 50% with a best-fit slope close to zero: workers never learned it. Even among one-sided conditions, greater stochasticity depressed both trust in Marvin and the accuracy of the mental model.
The authors' conclusion is that systems of identical accuracy can produce different team performance depending on the parsimony, stochasticity and dimensionality of the error boundary, and that these properties belong in model selection alongside accuracy.