Majority Voting of Doctors Improves Appropriateness of AI Reliance in Pathology

Two-stage within-subjects design run over two data-collection periods separated by a three-month wash-out, with the group conditions constructed offline afterwards.

Task. Detecting mitoses in histopathology images of meningioma, a count that drives tumour grading. Each participant examined 48 test images of 1,600 x 1,600 pixels, equivalent to 400 x 400 micrometres, matching the field of view of a 40x objective. The 48 images contained 88 mitoses in total, between zero and six per image. Ground truth came from two residents annotating all 51 candidate regions of interest independently, with a specialist pathologist adjudicating using phosphohistone-H3 immunohistochemistry as reference.

Stages. Stage 1, February to April 2023, was unaided detection with no AI. Stage 2, July to August 2023, was the same task with AI assistance. Between stages the images were randomly flipped and rotated by 0, 90, 180 or 270 degrees, presented in random order, and feedback was withheld, all to suppress learning and anchoring across the three-month gap.

AI system. An EfficientNet-b3 convolutional neural network trained on a meningioma mitosis dataset, reaching precision 0.961 and recall 0.841 standalone. Recommendations appeared as boxes overlaid on the image. Participants could switch recommendations off with a Show AI toggle and could set detection sensitivity to one of five levels from lowest to highest. Participants were deliberately not told anything about the model, which the authors justify by the time cost of that education.

Explanations. An XAI evidence card available on demand by right-clicking a recommendation box, containing a GradCAM++ saliency map, confidence information as probability and trust scores, a bar plot over five subclasses (pro-phase, meta-phase, ana/telo-phase, atypical, not mitosis), and ten retrieved similar examples. The interface also carried an FDA-style black-box warning advising that the pathologist's own judgement should govern.

Majority voting. Synthesised after data collection rather than run as a live condition. An odd-numbered group of k participants was sampled at random, their stage 2 mitosis reports pooled, and any candidate reported by more than k/2 of them retained as the group decision. Groups of k = 3, 5, 7, 9 and 11 were evaluated. The same sampled group was also majority-voted on its stage 1 responses, which preserves the within-subjects structure and gives an unaided group baseline against the AI-assisted group.

Measures. Appropriateness of reliance through relative AI reliance, RAIR = correct AI reliance / (correct AI reliance + under-reliance), and relative self-reliance, RSR = correct self-reliance / (correct self-reliance + over-reliance), following Schemmer et al. Detection performance as precision and recall against the adjudicated ground truth, and the frequency of super-AI performance, meaning outcomes that beat the model alone. Interaction logs covering the Show AI toggle, sensitivity settings and evidence-card openings support RQ1.

Aggregating several independently AI-assisted pathologists by majority vote made reliance more appropriate, and the gain was concentrated on the self-reliance side rather than the AI-acceptance side.

Relative self-reliance rose by roughly 31% for groups of three against a single pathologist working with AI, and relative AI reliance by roughly 9% (abstract figures). Both effects were significant across group sizes: RAIR gave a Kruskal-Wallis result of eta-squared_H = 0.043, p < 0.001, with post-hoc contrasts against the one-pathologist condition at p = 0.012 for k = 3, p < 0.001 for k = 5, p = 0.004 for k = 7 and p < 0.001 for k = 9; RSR gave eta-squared_H = 0.178, p < 0.001, with every group size differing from the single-pathologist condition at p < 0.001. The much larger effect size for RSR is the substantive result: what majority voting mainly fixes is over-reliance, the case where the pathologist abandons a correct own judgement for a wrong AI recommendation.

The mechanism the authors give is dilution rather than deliberation. Over-reliance on the model's false positives is idiosyncratic across pathologists, so it rarely commands a majority: in their words, it was not usual for the majority of participants to consistently over-rely when AI made false-positive mistakes. Because the votes were pooled from people who never spoke to each other, the effect cannot be attributed to discussion, and the authors present it as a lightweight alternative to cognitive forcing functions, which they argue risk algorithm aversion in time-pressured clinical work, and to richer XAI, whose benefits they take to be offset by cognitive burden.

Detection performance improved only in precision, and only for groups of five or more. Precision gave eta-squared_H = 0.150, p < 0.001, but the contrast between one pathologist with AI and a group of three was not significant (p = 0.715); k = 5, 7 and 9 all differed at p < 0.001. Reported means are precision 0.902 and recall 0.843 for three-pathologist majority voting against 0.824 and 0.817 for a single pathologist with AI. Recall did not move at all: eta-squared_H < 0.001, p = 0.774. Majority voting therefore suppresses false positives and does nothing about missed mitoses, which is what one would expect from a rule that requires more than half the group to flag a candidate. Groups did have a higher chance of achieving super-AI performance on recall.