The Consequences of the AI Act and Proposed AI Liability Directive on Medical Negligence

1. Introduction

Artificial intelligence (AI) has grown tremendously in the last few decades, bringing significant benefits to many fields, including the healthcare field. There has been a multitude of studies comparing AI decisions to physician counterparts, to show the success of such systems within the field and how they can simplify and relieve physicians from difficult decision-making.¹ Regardless of its benefits, growing concerns towards the use of AI in healthcare have been raised, including the risk of bias and unfair decisions,² and more predominantly, the issue of explainability,³ particularly when the use of AI in decision-making results in unexpected consequences, leading to liability claims. Fault-based negligence regimes have existed for many years; however, the use of AI has brought unprecedented challenges to these regimes, forcing regulators to rethink the legislative landscape to account for the growth of technology within industries, including the healthcare sector.

This paper showcases the impact of AI decision-making in medical negligence claims through a consideration of case-law from the England and Wales jurisdiction. In examining the challenges brought to medical negligence claims involving AI, this paper intends to evaluate whether the AI Act and proposed AI Liability Directive from the European Union (EU) achieve their aim of reducing the complexities of applying the law where AI is involved in decision-making. In doing so, this paper argues that the EU’s AI Act, combined with the proposed AI Liability Directive, does not go far enough to solve the current liability issues physicians face when making decisions with AI. This paper also looks to the future, by assessing how these challenges will develop as the use and understanding of AI advances further. A case study is used to showcase the failings of the legislative stance, with suggested recommendations striking a balance between protecting physicians and compensating victims who suffer an injury, in order to further contribute to the discourse on explainable AI in the healthcare sector.

2. Standard Medical Negligence Claims

Medical negligence claims rely on the traditional rules of tort law: a breach of an established duty of care, and a causal link between the breach of care and the injury suffered. With the ever-growing inclusion of AI-enabled decisions and treatments used in healthcare, this traditional application of negligence is flawed. In the English legal system, standard medical negligence claims have relied on the common law through the use of the Bolam and Bolitho test⁴ to assess the standard of care of a physician; rules which have been subject to scrutiny of their own for a number of years.⁵ Through an explanation of this precedent within the English legal system, the next section of the paper addresses how this system is challenged when physicians use AI to assist with decision-making.

In arguing whether a physician has breached the standard of care expected of them, the test first established in Bolam v Friern Hospital Management Committee⁶ provided the starting point for medical negligence claims within the English legal system. Now known as the ʻBolam test’, this case established that physicians must act in accordance with a responsible body of medical opinion to uphold their duty; if it can be proven that an act has fallen outside of this standard, a physician has breached their duty.

The Bolam test was subject to major criticism for providing excessive immunity to physicians and making it difficult for patients to succeed in negligence claims.⁷ Regardless of this, the test existed for four decades before the much-needed extension was given through the case of Bolitho v City and Hackney Health Authority,⁸ which adapted the Bolam test to state that the act from the physician also has to ʻwithstand logical analysis’. This addition was seen to achieve more of a balance for victims, empowering the court to question the credibility of medical opinion.⁹

It is important to note that the standard of care in medical negligence differs within the English legal system for cases of informed consent, which is governed by the more recent case of Montgomery v Lanarkshire Health Board,¹⁰ forming the rules for disclosure of risk. According to the precedent of Montgomery, physicians have the duty to disclose all material risks of treatment and any reasonable alternatives to the suggested treatments. A risk is deemed ʻmaterial’ both subjectively, in that the particular patient would be likely to attach significance to the risk, or objectively, where a reasonable person in the patient’s position would likely attach significance to the risk.¹¹ If a healthcare professional fails to disclose a material risk or reasonable alternatives, this would amount to a breach of care.

The third element needed for a successful medical negligence claim after proving a breach of care is causation. To establish causation, a link needs to be shown between the physician’s act and the injury suffered by the patient. Depending on the circumstances and context of the claim, a number of precedents can be relied on to assist in satisfying this element within the English legal system.¹² The starting point for proving causation is known as the ʻbut-for test’, introduced in Barnett v Chelsea and Kensington Hospital,¹³ which poses the question of whether the injuries suffered would have occurred but for the act or omission of the healthcare professional; if the injuries suffered were due to the act or omission, causation would be established.

It is not always clear whether a physician’s act caused the injury suffered, and in cases where this is difficult to establish, the precedent of Bailey v Ministry of Defence¹⁴ can be relied on, which relaxes and broadens the but-for test, whereby it only needs to be proven whether the breach of care materially contributed to the injury suffered. If the act or omission made by the healthcare professional materially contributed to the injury, then causation would be established. As noted, the procedure for establishing a successful medical negligence case is strongly backed by case-law, allowing the likelihood of a successful case to be estimated effectively when advising clients. However, this process has faced a complex challenge with the introduction of AI within the healthcare setting, particularly when used to assist with decision-making.¹⁵

3. The Challenges Brought by AI to Medical Negligence Claims

The influx of AI systems in healthcare has brought several benefits to the field through assisting physicians with decision-making and robo-surgeries, and bringing more efficiency to treatment plans, and due to these benefits, there is a steady increase in their use.¹⁶ Regardless of the benefits, there are some instances where unexpected injuries may occur, leading to the bringing of a new era of medical negligence claims, not only questioning the actions and decisions of physicians, but physicians who use and rely on AI when making decisions. This use of AI poses unprecedented challenges to the standard legislative regime, particularly concerning the establishment of a breach of care.

To begin, the Bolam and Bolitho test, explained in the above section, works on the assumption that doctors use their own judgement alone and are sole decision-makers. This assumption is clearly challenged when those circumstances differ, in scenarios where doctors make decisions with AI systems, or where decisions are made without doctors altogether. According to Bolam and Bolitho, the claimant would have to show that the physician has acted outside of what is considered responsible by other physicians to establish a breach of care, and hence, that they acted outside what is considered the ʻnorm’ in the standard healthcare setting. AI is built on the idea of innovation, changing the standard and ways in which things have been done to achieve maximum efficiency.¹⁷ This idea of AI is hoped to affect the healthcare setting in the same way, allowing AI to pave new ways in which illnesses can be treated, and hence going against the principles that underpin Bolam and Bolitho. Applying this idea to the Bolam and Bolitho test, the following scenarios could raise some challenging questions:

This continuance of these scenarios also raises the complex consideration of the possible future reliance on AI systems. Currently, AI systems can only be used to assist physicians with decision-making, in that the final decision needs to be made by a human, due to the current lack of public trust and accuracy in machines.¹⁸ However, in the future, if AI continues in its current trajectory, it can be assumed that some decisions may be made wholly by AI itself, which could lead to an over-reliance on such systems to improve efficiency within the health sector. This future standpoint could lead to further complex questions in scenarios where unexpected injuries occur, as liability would have to fall onto a human counterpart to ensure that claimants could be sufficiently compensated, and could include an example such as the following:

In scenarios 3 and 5, the question of explainability becomes prevalent. The concept of explainability has always been a major discussion point in the academic literature surrounding AI systems, particularly when systems are described as ʻblack-boxed’ systems. The term ʻblack-box’ refers to the technical infeasibility of gaining an understanding of how systems arrive at their output decisions, where machines are also unable to provide this explanation themselves. Even if systems were explainable and could generate an explanation on an arrived decision, would this healthcare professional themselves be expected to interpret that explanation and make it understandable to a patient?

It is not unusual for physicians to adapt the way they explain surgeries and decisions so that a patient can understand the information, and if this expectation was consistent for explaining the use and decision-making process of AI, physicians would need more than medical knowledge. It would be unfair to expect on top of an average of seven years of education to become a physician,¹⁹ that healthcare professionals then need additional technological understanding and training to enable them the ability to provide patients with full explanations of the future of healthcare.

The application of Montgomery²⁰ is also an interesting discussion point when considering the collaboration of healthcare professionals and AI. As explained in the above section, this precedent only extends to the area of informed consent in the disclosure of risks, where the healthcare professional must disclose all material (viewed both subjectively and objectively) risks and alternative treatments to the suggested surgery.

In a scenario where a physician intends to complete a ʻrobo-surgery’ with assistance from AI robotics, it can be questioned whether the risks of using AI could amount to a material risk to the patient. Usually, the risks that need to be disclosed are from a medical standpoint, but if for example, the robo-surgery tool is a ʻblack-boxed’ system, would this need to be explained to the patient before they give consent to surgery, as this would still amount to something the patient themself, or objectively, a reasonable patient would want to know?

If this was the case, there is a question of whether this is sufficient, as based on the assumption of public trust, especially given the current stance, disclosure of AI risks and the lack of explainability would evidently ʻscare’ patients, making them opt for ʻnormal’ surgery rather than ʻrobo-surgery’. This would then be similar to scenario 1, in that the integration of AI into surgery would be rendered useless. The EU’s AI Act, discussed more thoroughly in the next section, includes the obligation that if not obvious, consumers or citizens must be informed that AI is in use²¹ which, particularly in this scenario, could delay the integration of AI into the healthcare world with the current lack of public trust.

Another issue stems from the concept of vicarious liability. In the English legal system, the healthcare provider is vicariously liable where a healthcare professional has breached their duty,²² allowing a scheme to ensure that the claimant can be compensated. It could be questioned whether vicarious liability would extend to harm caused by AI in hospitals or other facilities, or whether liability would instead extend to the manufacturer or software developer of the system. In the case where physician liability would extend to healthcare providers, yet AI liability would extend to a body external to the healthcare provider, it is possible that medical negligence cases involving AI will turn into warfare, with both providers intending for the other to be held responsible.

The above would increase the challenge to establish causation. If future estimations of AI ʻtaking over’ sectors come to fruition, AI could play a part in the majority of decisions made by physicians, meaning that physicians would have a basis for escaping liability by establishing AI as a novus actus interveniens and relinquishing liability on the external party. In scenarios where causation has been established, the use of AI could also bring issues in considering the remoteness of damage, in that the assessment of damages is usually based on the degree of foreseeability. If systems used have self-learning capabilities, as many AI systems do, the essence of foreseeability dissolves the longer along the lifecycle a system is, allowing circumvention of liability as it could be argued that the injury sustained was unforeseen.

These complexities ultimately highlight the need for specific and strict rules to address the distinctive issues this technology brings to traditional liability regimes. In the first of its kind, the AI Act,²³ introduced by the European Commission, sets a standard for AI regulation. The Act is likely to influence nations across the globe, similarly to the effect of the EU’s General Data Protection Regulation (GDPR),²⁴ highlighting the power of the ʻBrussels Effect’.²⁵

4. Current AI Regulatory Landscape

The European Union adopted its AI Act in June 2024. The Act applies horizontally and harmonises rules for the development and use of AI. The Act is part of the EU’s two-fold approach to regulating AI, alongside the proposal for an AI Liability Directive.²⁶ The AI Liability Directive proposal was introduced in 2022 and aims to clarify the rules on non-contractual civil liability to ensure liability claims involving AI can be dealt with at a consistent level to human counterpart claims.

The Artificial Intelligence Act works on a risk-based approach, categorising AI systems upon the purpose in which they are used, with those posing an unacceptable risk banned. Systems that pose a limited risk are subject to heightened transparency obligations,²⁷ including the disclosure of their use. Minimal-risk systems are permitted to be used without restriction, but voluntary codes of conduct are expected to regulate these systems, which include spam filters and AI-enabled gaming. The Act itself predominantly focuses on those systems categorised as high risk, with a number of requirements imposed on providers and deployers of such systems. AI used in the healthcare field, that fall under the Medical Devices Regulation (MDR),²⁸ are considered to be high risk, and hence the majority of AI systems and devices used in the healthcare sector are categorised as high risk.

The AI Act applies to both providers (developers of AI systems) and deployers (any person, body, authority, or agency who put AI systems on the market, and not the end-user),²⁹ and therefore, healthcare facilities that use AI would be considered deployers. There are a number of obligations placed on providers of systems, before such systems can be placed lawfully on the market, but the scope of this paper is focusing on deployers of such systems. Although not subject to as many obligations as providers, deployers of AI have requirements that they must uphold, including operating systems in accordance with the instructions of use, ensuring human oversight, monitoring the use of AI systems, informing providers regarding any serious incidents or malfunctioning, and conforming to existing legal obligations such as the GDPR and MDR.

The proposed AI Liability Directive, which is still under consideration, has introduced two main provisions for non-contractual liability. These include the power given to national courts to order the disclosure of evidence relating to high-risk systems that are suspected of having caused damage, and the introduction of a rebuttable dual presumption; one regarding causation, and the other regarding fault. Under Article 3 of the proposed Directive, which governs the court’s power to order the disclosure of evidence, Article 3(5) introduces a presumption of non-compliance with the duties of care, relevant to those who fail to disclose or preserve evidence as required.³⁰ This presumption works as a deterrent to encourage the disclosure of evidence but also to expedite court proceedings.

The second presumption introduced by the proposed Directive is the existence of the causal link between fault and damage, which is based on a tri-fold criterion. For this rebuttable presumption to be triggered, there must be:

(1) non-compliance with an obligation relevant to the harm that amounts to a breach of care (which can be presumed under Article 3(5)³¹),

(2) that it be reasonably likely that the defendant’s negligent conduct has influenced the output that gave rise to the damage, and

(3) that the output produced by the system gives rise to the damage.³²

5. Does the AI Act and Proposed Directive Address the Challenges to Medical Negligence?

The obligations imposed under the AI Act provide reassurance to deployers and those subject to AI decisions that the AI available on the market has gone through a series of testing and assessments; however, due to the nature of AI, these safeguards do not ensure that AI will always perform as expected. The AI Act solves issues for healthcare providers where systems act against what is expected and there has been non-compliance under the AI Act obligations by a provider, but a loophole remains in scenarios where compliance with the obligations has been satisfied by both providers and deployers, but unexpected injuries can still occur.

Without the triggering of the rebuttable presumption under the proposed AI Liability Directive through non-compliance, the affliction of liability will fall onto the involved physician and related healthcare provider, putting a weighted burden on those who choose to utilise AI. In addition to this, Recital 15 of the proposed Directive states that the Directive is intended to only apply to fully automated decisions, meaning that in the healthcare scenario where a physician is involved in ʻtaking’ the decision to the patient, the Directive would not apply. This limited scope has received major criticism, and its removal has been advocated for, particularly given its failure to uphold and respect the right to redress for consumers,³³ in addition to its conflict with the right not to be subject to automated decisions under the GDPR.³⁴

Without the application of the proposed AI Liability Directive, it is unclear how effective enforcement will be relating to non-compliance with the AI Act, particularly when the provider and deployer of AI differ from one another. With the lack of the rebuttable presumption, the weight of liability falls back onto the physician, which re-emphasises the re-occurring question throughout this paper: to what extent does a physician need to understand an AI decision to rely on it and escape liability? Applying the current standard within the English legal system for medical negligence, this would mean that if other healthcare professionals had also relied on the decision, and that was deemed as ʻlogical’, a breach of care would not be proven, which would leave victims uncompensated, making it even more difficult to establish a successful medical negligence case.

Currently with civil claims, the burden of proof falls on the claimant, but if the claimant could argue that the physician should not have relied on a system (particularly if not used or opposed in other jurisdictions), would it be left to the physician to justify their reliance? If so, would this reliance entail an understanding of the system? If so, would they need to have the technical expertise to understand and explain the decision? If so, how would this work with systems described as ʻblack-boxed’ and inherently unexplainable? This takes the discussion back to scenarios 3 and 5 discussed earlier in this paper, showing that the EU legislative stance is inadequate in solving the main questions surrounding medical negligence claims.

6. Conclusions

It is clear that AI will continue to advance and transform sectors of society with the benefits of improving efficiency, reducing human error, and assisting with decision-making. In order to reap these benefits, it is imperative that regulation catches up with the technology in order to reduce the existence of the ʻpacing problem’. Currently, the EU’s AI Act and proposal for an AI Liability Directive fail to address the issues caused by negligence regimes applicable to the integration of AI into the healthcare setting. The current regulatory position leaves several technical questions unanswered, highlighting the future difficulties the courts will face when they apply the new rules.

A balance needs to be struck in medical negligence claims to protect victims and ensure that they can be fairly compensated, whilst also ensuring fairness and clarity to physicians. In an era where AI is already beginning to be a major feature in the sector, there needs to be a common understanding of what is expected by providers, and how citizens will be fairly compensated in the inevitable scenario of AI not performing as expected.

Acknowledgements

I would like to express my gratitude and thanks to Lee Andrew Bygrave and Vera Lúcia Raposo for the opportunity and support in writing and submitting this article.