The Principles and Limits of Algorithm-in-the-Loop Decision Making
N = 1888 Recidivism risk assessmentLoan default predictionThis study in the framework
Held constant. The pretrial model reached a test AUC of 0.66 and the loans model 0.71 under five-fold cross-validation, which the authors describe as comparable to COMPAS and the Public Safety Assessment and as good to excellent by a published meta-analysis. The same model was used in every condition of a given setting, so accuracy was not manipulated. Because each participant saw a different random sample of 40 of the 300 cases, the observed quality of the model varied across participants.
- RQ1: What criteria characterise an ethical and responsible decision when a person is informed by an algorithm?
- RQ2. Do the ways that people make decisions when informed by an algorithm satisfy these criteria?
"Under all conditions, our study participants 1) were unable to effectively evaluate the accuracy of their own or the risk assessment's predictions, 2) did not calibrate their reliance on the risk assessment based on the risk assessment's performance, and 3) exhibited bias in their interactions with the risk assessment."
abstract
"People may make more accurate predictions on average when informed by an algorithm, but they are unlikely to recognize and discount any errors that arise."
discussion
"That assumptions about human oversight are so central to risk assessment advocacy and governance is particularly troubling given the inability of algorithms to reason about novel or marginal cases."
discussion
Experimental design
Behavioural experiment on Amazon Mechanical Turk, run in March 2019, using two risk assessments built by the authors with gradient boosted trees. The pretrial model predicts violation of pretrial release from five defendant features; the loans model predicts default from seven loan and applicant features.
Participants were randomly assigned to one of two settings, pretrial release or financial lending, and then to one of six conditions. In each setting the same 300 cases were used across all conditions, and each participant made 40 predictions on a scale from 0% to 100% in steps of 10%.
The six conditions were: baseline, narrative profile only and no algorithm; RA prediction, profile plus the model's prediction in numeric form; default, as RA prediction but with the answer field pre-filled at the model's value, which participants could change; update, participants predicted from the profile alone, were then shown the model's prediction for the same case and predicted again; explanation, RA prediction plus a list of the features that pushed the prediction notably higher or lower, derived from log-odds influence; and feedback, RA prediction plus the true outcome of each case revealed after each prediction.
Full findings
On accuracy, every condition except feedback improved on baseline in both settings, but no condition matched the model working alone. Update performed best, with a gain of 0.603 in pretrial and 0.821 in loans against 0.464 and 0.682 for the plain RA prediction. Feedback performed worst and was statistically indistinguishable from baseline in pretrial. The mechanism for feedback is instructive: after the first outcome was revealed, participants shifted towards extreme predictions of 0% and 100%, apparently treating their own probability estimates as binary because the feedback was binary, and their match rate with the model collapsed. The update advantage was twofold. Participants' initial unaided predictions in update were better than those in baseline and improved over the 40 trials, so exposure to the model trained them; the second, model-informed prediction then improved further.
On reliability, participants failed both components in every condition. Self-reported confidence was unrelated to actual performance, and in two cases was negatively related. Participant estimates of the model's accuracy were unrelated to its actual performance, and in three cases negatively related. Reliance was not calibrated to model quality: of ten condition-by-setting cells only one, explanation in loans, showed people relying more on the model as it performed better, while three showed the opposite.
On fairness, every condition exhibited disparate interactions. Where the model pushed towards higher risk, it moved predictions about black defendants more than white ones in every condition, with disparities up to 0.38 in RA prediction and feedback. Participants also deviated upward for black defendants and downward for white ones in every condition. Update produced the smallest disparities on both measures, reducing them by 81.5% and 73.9% relative to RA prediction, but did not eliminate them.
The authors read this as undermining the assumption that a human in the loop provides a meaningful check on a model, since overriding requires evaluating quality and their participants could not.