Consultant or Decision-Maker? How AI's Role Defines Clinical Accountability

Challenges in AI Implementation in Healthcare and the Importance of the “Human in the Loop” Approach: Theoretical, Empirical and Operational Analysis

Abstract

Objective

Both theoretical and empirical analysis was conducted on the impact of the context in which automation is introduced within a decision-making system. The operational objective focuses on understanding how AI integration influences clinical workflows, especially in the link between human function and automation.

Background

Previous work addressed causality and accountability in human-automation systems without considering the effects of how the automation’s role is presented to users. In a healthcare setting, this means that professionals may assume different roles depending on how the system is designed and deployed, which impacts their participation and autonomy.

Methods

An existing analytical model was adapted to predict human contribution to outcomes, incorporating the automation context. Operationally, this approach is linked to how AI is presented in clinical interfaces (advisory vs. decisional). A signal detection experiment assisted by automation was carried out with 400 participants to evaluate the correspondence between observed behaviour and model predictions.

Results

The context in which the automation’s role is presented influenced users’ tendency to follow its recommendations. When automation made decisions and users only supervised it, they tended to contribute less to the outcome than in systems where automation had a merely advisory function. This manifests directly in operational workflows where the professional may shift from being an active evaluator to a mere supervisor, affecting safety, efficiency and compliance.

Conclusion

The specific way in which automation is integrated into a system affects its use and the perception of user involvement, potentially altering the overall system performance. Systems that maintain significant human control tend to favour greater accountability, regulatory alignment and better operational outcomes.

Application

This research can help design automation-assisted decision-making systems and provide information on regulatory requirements and operational processes for such systems. In the healthcare context, it implies the need for flexible structures that facilitate the adjustment between automation and human control, ensuring regulatory compliance and optimal clinical experience.

Introduction

Human operators collaborate with automated decision support systems to improve outcomes in complex tasks. In the healthcare field, this collaboration materialises in tools such as clinical support systems, automated triage solutions and remote monitoring applications.

These systems combine human and automated capabilities, but raise key questions about accountability, influence and decision-making. AI integration demands that design carefully considers the distribution of roles, to avoid the inadvertent reduction of human involvement that can lead to diminished control, security breaches or lack of compliance.

Douer and Meyer proposed an information theory-based model to quantify human accountability as the unique information contributed to the outcome. This is essential in clinical environments where traceability and accountability are fundamental regulatory requirements.

However, this model does not consider the context in which automation is presented, which can alter both real and perceived accountability. For example, the healthcare professional may find their capacity for intervention diminished if AI is perceived as decisional rather than advisory, which translates into operational challenges, lower individual contributions and possible conflicts in the distribution of responsibilities.

Contextual Operational Example

In HIS (Hospital Information System) systems, the automatic decision module can limit doctors from entering data or adjusting recommendations, relegating them to supervisors. This affects their perception of accountability and modifies their workload, as reflected in experiences of technological adaptation and administrative efficiency (RAG).

Conceptual and Theoretical Framework

The Douer and Meyer model formulates human accountability as the amount of unique information that the operator contributes to the final outcome of the joint human-machine decision. This approach is based on information theory and uses variables such as:

• I(U): Information provided by the user (healthcare professional)
• I(A): Information provided by automation (AI)
• I(T): Total system information

It is formally defined as:

I(R) = I(T) - I(A)

Where I(R) represents residual human accountability. In the clinical environment, I(R) translates into the capacity of the doctor or health professional to influence the diagnosis, treatment or final administrative decision, even when AI contributes.

This model allows human participation to be quantified in terms of information, taking into account the system design. A system where AI decides and the human supervises reduces I(R); conversely, an advisory system where the professional can accept/reject recommendations increases I(R).

Operational Context Variable

The automation context (decisional vs. advisory) determines the way in which the user perceives their role and contributes information. This directly affects workflows and regulatory compliance, because traceability and imputability requirements differ according to the degree of human intervention.

Analytical Model

The adapted model incorporates the context of automation presentation. If AI is implemented as decisional, the user tends to rely on it more and contribute less information. Formally,

I(R) dec = I(T) - I(A_dec) I(R) cons = I(T) - I(A_cons)

where I(A_dec) > I(A_cons).

In operational terms, this translates into the difference between:

• Systems where AI offers a recommendation and the human decides (advisory)
• Systems where AI makes the decision and the human supervises (decisional)

The prediction is clear: the more decisional the AI, the lower the healthcare professional’s contribution to the outcome, which can diminish human control and the capacity to correct errors.

Formal Example in Healthcare Systems

In remote monitoring workflows, when software generates automatic alerts and only requests professional validation, the following is observed:

• Lower human intervention
• Possible automatism in alarm management (transferred workload)
• Risk of omission of relevant clinical nuances

Methodology / Experiment

An automation-assisted signal detection experiment was designed, involving 400 participants. In the healthcare field, this can be equated to the process of validating AI algorithms to detect pathologies in radiology, where the professional reviews the result generated by AI.

Participant behaviour was evaluated under two conditions:

1. Decisional automation: AI makes a decision, user supervises it.
2. Advisory automation: AI provides a recommendation, user decides.

The correspondence between the human contribution predicted by the model and actual behaviour was measured, analysing variables such as:

• Level of adherence to recommendations
• Residual human contribution
• Corrected error rate
• Perceived involvement

Operational Implication

The experiment reflects the dilemma of real clinical practice: in diagnostic support systems, if the professional only validates the AI’s decision, their intervention is reduced and the opportunity to identify atypical errors or relevant nuances diminishes. This affects patient safety and efficiency in the management of complex cases.

Results

The context in which the automation’s role is presented influenced users’ tendency to follow its recommendations. In decisional systems, users tended to contribute less to the outcome than in advisory systems.

Operational Impact

• Healthcare professionals operating systems where AI is decisional show lower active involvement, which in practice can lead to:

  • Decreased control over the clinical outcome
  • Reduced capacity to correct errors
  • Increased blind trust in the automated system
  • Potential regulatory breaches due to lack of traceability

• In advisory systems, where the professional must interpret and decide on the AI’s recommendation, the following is maintained:

  • Greater human intervention
  • Better overall system performance
  • Higher levels of regulatory compliance due to greater traceability and accountability

Contextual Clinical Example

The challenge of technological adaptation in hospitals reflects the need for professionals to maintain control over the decision. The integration of decisional modules in HIS systems can generate resistance, unwanted administrative burden or loss of trust if clear mechanisms for human intervention and correction are not designed.

Discussion

The specific way in which automation is integrated into a system affects its use and the perception of user involvement, altering the overall system performance.

Cause → Effect Reasoning

• Decisional automation design → User supervises without intervening → Lower individual contribution → Greater risk of undetected errors
• Advisory design → User evaluates and decides → Active and accountable contribution → Better safety and regulatory compliance

Practical Implications

1. Patient Safety
   • An advisory design allows the detection of clinical errors inadvertently made by AI.
   • Passive supervision reduces the capacity to detect anomalies.
2. Efficiency
   • Overly automated systems can reduce human workload, but may sacrifice care quality.
   • Advisory systems may increase the burden on the professional, but this is offset by better overall performance and error correction.
3. Regulatory Compliance
   • Decision traceability requires active human participation, therefore the design must ensure that AI does not completely replace the professional’s role.
   • Regulatory processes value significant human control, to avoid conflicts in imputability and incident management.

Application Example

In the deployment of remote monitoring solutions integrated into clinical management software, it is observed that automation must balance the reduction of administrative burden with the need for direct medical intervention to guarantee quality and compliance.

Conclusions

The context in which automation is presented is key to determining:

• human accountability
• user behaviour
• regulatory compliance

Designing this context correctly is essential to guarantee significant human control in automated systems.

Implication for Healthcare Operations

This research ratifies the importance of implementing AI in healthcare under “human in the loop” principles, where the professional has tools to exercise effective control, correct errors and contribute meaningfully to the outcome. The operational challenge is to modulate the degree of automation while ensuring flexibility and traceability, aligning workflows with regulatory and clinical standards.

Source

https://pmc.ncbi.nlm.nih.gov/articles/PMC12231881/

Suggested Next Step

It is recommended to evaluate the current AI integration processes in your organisation, identifying critical points where human-machine interaction could be optimised or adjusted to meet regulatory and clinical safety standards. Initiating an operational diagnosis will allow the definition of a governance, integration and continuous improvement framework oriented towards outcomes.