Ignore, Trust, or Negotiate: Clinician Acceptance of AI-Based Treatment Recommendations

A mixed-methods think-aloud study with 24 ICU clinicians, each making four AI-assisted treatment decisions on real de-identified patient data, with visualisation condition varied within subjects.

The AI. The AI Clinician (Komorowski et al. 2018), a reinforcement learning model for sepsis treatment, reimplemented by the authors on MIMIC-IV and trained on 18,143 patients meeting standard sepsis criteria. The model clusters patient timesteps into 750 states by k-means and uses policy iteration to select among 25 actions, formed as 5 IV fluid levels crossed with 5 vasopressor levels, binned by quantiles, over 4-hour intervals. The authors chose this model because it is the best known of its kind and therefore the most likely to be accepted, while noting it has been criticised for deviating from bedside judgement and that its evaluation rests on off-policy estimation.

The interface. A purpose-built system, the AI Clinician Explorer, supporting patient search, trajectory visualisation with vitals and labs as small-multiple line charts with abnormal values in red, a heatmap of estimated action values, a heatmap of historical clinician action frequencies for the same state, a SHAP-based state interpretation chart, and observed mortality for the state. During the study participants saw a controlled subset of these views rather than the full explorer, to approximate the information available in practice.

Explanations. Because explainable reinforcement learning methods would have required changing the validated model, the authors explained the state clustering instead: for each of the 750 states they trained an XGBoost classifier to predict membership and used SHAP to identify the features driving it. XGBoost with SHAP was chosen over random forests, SVMs, permutation importance and SVM coefficients because it produced more parsimonious and clinically sensible explanations.

Conditions, within subjects, presented in fixed order of increasing information:

• 1. No AI. Decision made without a recommendation.
• 2. Text Only. A textual recommendation, introduced as coming from "Sepsis-AI".
• 3. Feature Explanation. Text plus a SHAP chart showing the five variables that contributed most to the patient's state assignment and how each deviates from average.
• 4. Alternative Treatments. Text plus a bar chart of five possible actions ranked by AI value score, with bars colour-coded by how often clinicians historically took each action for similar patients.

The fixed order was chosen to limit cognitive burden and let participants become progressively familiar with the interface; patients were presented in randomised order, so patient and condition are crossed. The AI was described as analysing electronic health records to recommend fluid and vasopressor doses that optimise mortality based on historical data. It was deliberately not called the AI Clinician, to avoid participants recognising the published model.

Case selection. Because the accuracy of a treatment recommendation cannot be established, the authors did not select cases to hit a target accuracy. They instead selected four patients and timesteps where the AI Clinician's recommendation differed substantially from what a plurality of clinicians had historically done, which guarantees the recommendation challenges clinical judgement. Each case was given a pseudonym and a generic clinical vignette carrying no treatment-relevant information beyond the data.

Response format. Participants chose, for each of IV fluids and vasopressors, to begin or increase, end or decrease, or leave unchanged, giving 4 to 6 options per patient. The three-way choice was adopted to capture first-order resuscitation reasoning as it happens at the bedside and to make the results analytically tractable, even though the AI's recommendation specified exact doses.

Measures. After each decision, confidence in the treatment choice and perceived case difficulty on 7-point scales; in the three AI conditions, also the usefulness of the recommendation and the degree to which it changed their confidence. Sessions ended with a semi-structured interview.

Explanations changed what clinicians thought of the AI and left what they did with it almost untouched. The important result is that binary acceptance was the wrong model of what they were doing.

Attitudes moved with explanations. The AI was rated more useful in the Feature Explanation condition than in Text Only (F(2,69) = 4.251, p = .03; pairwise delta 0.83, 95% CI [0.24, 1.43], p = .018) and reported to raise confidence more (F(2,69) = 7.946, p = .002; delta 1.08, 95% CI [0.51, 1.66], p < .001). Alternative Treatments sat between the two, directionally below Feature Explanation.

Own confidence did not move significantly (F(3,92) = 2.220, p = .11, all pairwise p > .17), though it trended upward with explanations, most so in the Alternative Treatments condition.

Perceived difficulty went the wrong way. Cases were rated significantly harder with AI and explanations than without: No AI against Alternative Treatments, delta 1.08, 95% CI [0.46, 1.71], p = .003 (F(3,92) = 4.112, p = .02). The authors read this as explanations prompting clinicians to weigh more factors, especially when the explanation did not match their mental model of the patient. One participant described having to guess where the recommendation came from after seeing a blood urea nitrogen feature they had not expected, spending extra mental energy on it.

Decisions did not move. Full concordance with the AI was about 42% across all three AI conditions, against a 33% base rate with no recommendation shown, and the visualisation condition made no meaningful difference. Concordance with the actual treating clinician and with the majority attending decision fell slightly when AI was present: 50% full concordance with the majority attending across the AI conditions against 63% without. Yet of the 36 AI-assisted decisions in which a participant departed from the attending majority, only 6 fully matched the AI. Participants were being moved off the typical clinical action without being moved onto the AI's.

The qualitative analysis explains the flat aggregate. Four behaviour patterns emerged, and they are the paper's contribution:

Ignore (7 participants, 21 AI decisions). The AI never influenced the decision. These clinicians were already confident from their own assessment; in 3 of 21 decisions they did not verbally acknowledge the recommendation at all. Engagement, when it happened, took the form of critiquing the recommendation while holding their own judgement fixed. Notably, the AI still affected them: it induced doubt without changing behaviour, so for this group the system only lowered confidence and raised perceived difficulty.

Negotiate (12 participants, 36 AI decisions), the largest group and the unexpected one. These clinicians decomposed the recommendation and accepted parts of it. In 14 of 36 decisions they agreed on either fluids or vasopressors but not both. In 12 of 36 they engaged specifically with the dosage level, usually rejecting it, though one noted that when the overall direction agreed, deferring the dose to the AI removed a decision they would otherwise have had to make. Some accepted the recommendation but changed its timing, starting fluids and deferring vasopressors by a couple of hours pending response. Two things drove what they prioritised: perceived risk and urgency (in 12 of 36 decisions, with discordant advice more acceptable for stable patients than for deteriorating ones), and the explanatory evidence itself, which this group used more than any other. Negotiators rated the AI far more useful than the rest, 4.6 (SD 1.27) against 2.8 (SD 1.79).

Consider (3 participants). The binary pattern that the prior literature predicts: fully defer or fully override, driven by their own uncertainty. In 3 of 9 decisions they yielded to the AI on cases where they were ambivalent, and dismissed it outright otherwise. Only these three behaved the way the field's dominant reliance model assumes.

Trust (2 participants). Accepted at least part of the recommendation in every decision, emphasising that the AI was based on objective data.

The behaviour patterns were not explained by seniority. Two of the seven Ignore participants were not attendings, and both Trust participants were attendings. The authors flag this as contrary to prior work reporting that experienced clinicians dismiss AI and novices over-rely, and suggest the operative difference is instead pre-existing attitudes to evidence derived from historical data and expert committees.

Four further barriers emerged from the interviews. Clinicians said their decisions rest on bedside information the AI cannot see (straight leg raise, ultrasound, how ill the patient looks) and treated this as grounds for the superiority of human judgement while not rejecting the AI outright. The model's discretised dose bins did not match practice, with the third fluid level of 75 mL over four hours described as sprinkling water. The 4-hour decision interval was acceptable for stable patients but too long to commit to for higher-risk decisions, which pushed clinicians to postpone agreement to a later timestep, an important measurement problem for reliance. And clinicians became sceptical when the AI deviated from sepsis guidelines or from their own personal practice, with eight expecting the AI to recapitulate guidelines rather than depart from them, even though the guidelines themselves invite individualisation.

Finally, trust was described as something established before use, not during it. Several participants said their overall trust would be settled by the methodology and validation evidence presented to them before they ever touched the tool, naming journal reputation as a criterion, and that even then upfront quality evidence would not supersede case-by-case clinical judgement.