AI Robotics in Healthcare Between the EU Medical Device Regulation and the Artificial Intelligence Act

2.1.1 Definition of Medical Device

According to Article 2(1) MDR, a medical device is a product (including a software) which is intended by the manufacturer to be used for diagnosis, monitoring, treatment or alleviation of a disease, an injury or disability. In contrast, software cannot be regarded as a medical device if it fulfills a purely informational, archiving or displaying function and only stores or graphically displays patient data without evaluating the data and thus influencing a medical decision.¹⁶

A further distinction must be made with respect to so-called lifestyle and wellness applications, which cannot be considered medical devices according to Recital 19 MDR either.¹⁷ These include, for example, applications that provide instructions for training exercises, give tips on nutrition or store the user’s weight or pulse.¹⁸ As the software in these cases does not serve a medical purpose, but only transmits information to the user without deciding specifics with regard to a disease or a disability, the MDR does not apply.¹⁹ In such cases, it seems problematic that no other special EU regulation exists to protect users.²⁰

2.2.2 Intended Purpose

A particular problem arises when the device can be used for medical purposes, but the manufacturer nevertheless tries to label the system as a non-medical product, for example, by relying on disclaimers indicating that the system should not be used for medical purposes beyond its intended use in a lifestyle environment. According to Article 2(1) and Article 2(12) MDR, the decisive factor for a product to qualify as medical device is the intended purpose which results from the labeling, instructions for use or advertising materials of the manufacturer. Therefore, the classification does not depend on the objective purpose, but on the subjective purpose of the manufacturer. As a result, a product can objectively serve medical purposes, but is not classified as a medical device due to the subjective intended purpose. At first glance, it is therefore within the manufacturer’s power to define whether the product is a medical device. Consequently, more and more manufacturers try to label their products as non-medicinal products to avoid the MDR regime.²¹

However, the manufacturer’s right to determine the intended purpose is limited. The MDR specifies details that must be included in the description of the intended purpose and these consequently limit the manufacturer’s freedom to determine the intended purpose. For example, the manufacturer must provide information on the safety and performance of the product in the instructions and in the labeling,²² including information concerning the indication as well as contraindication, the patient group and the user profile (Annex I No 23.4(b) MDR).²³ If contraindications or risks associated with the product are known or if the use of the medical device requires certain qualifications and knowledge of the user, the information must be provided by the manufacturer.²⁴ Therefore, the subjective intended purpose is limited by objectively assessed risks, which must be set out in the instructions for use.

Whether the MDR places further limits on the manufacturer’s intended use to avoid circumvention of the MDR is currently unclear. In Germany, the Federal Supreme Court ruled in its landmark ʻActive Two’ case that the subjective purpose requirement is limited in terms of arbitrariness.²⁵ If the manufacturer restricts the use of its product to certain purposes, these must be comprehensible and not arbitrary. If a manufacturer violates these requirements, the objective purpose becomes the decisive criterion.²⁶

The CJEU, however, has regularly emphasised in its decisions on the Medical Device Directive 93/42/EEC²⁷ that it is the subjective intended purpose by the manufacturer that ultimately matters.²⁸ On the other hand, the Court has not yet had to deal with cases in which a circumvention of the MDR by the manufacturer seemed likely. Therefore, it remains to be seen whether the Court will limit the manufacturer’s freedom to determine the intended purpose—for example by drawing on its case law on the prohibition of abuse of rights.²⁹

2.2.1 Overview

If an AI system or a robot qualifies as a medical device, this has broad consequences for the development and use of the product. Manufacturers must pass a conformity assessment procedure before they are allowed to release the device onto the market. Only if they successfully complete this procedure, can they CE-mark their product and place it on the market (Article 20 MDR). Medical devices are classified into four classes (I, IIa, IIb and III) as per Article 51(1) MDR, according to the intended purpose of the devices and their inherent risks.³⁰ The higher the risk, the higher the class and the more stringent the conformity assessment procedure.

2.2.2 Static vs Dynamic Systems

Medical devices including robotic systems that are leveraging AI technology can be on a spectrum from ʻstatic’ systems that have learned and operate in a learned state to ʻdynamic’ systems that continuously learn from real world experience in the field.³¹ The specific characteristics of dynamic systems pose major challenges to the existing medical device legislation. The conformity assessment procedure is intended to demonstrate that the requirements for medical devices are fulfilled at the time of the conformity assessment. In other words, the focus is primarily on safety risks that exist already at the time the product is placed on the market, while risks that may arise later as a result of continuous learning are not sufficiently taken into account.³²

Consequently, notified bodies in the EU³³ and regulators elsewhere³⁴ do not currently certify dynamic AI systems. Rather, they require the software to be ʻfrozen’ at a certain point in the learning process to evaluate the software at that point. As a result, manufacturers must carry out a new conformity assessment every time a medical device learns after being placed on the market from real-world data resulting in a significant change in its safety and performance characteristics.³⁵

This requirement could hinder the development of innovative AI systems.³⁶ However, considering the high risk of physical harm such as death, disability, or other diseases, the requirement seems appropriate. To mitigate its impact, there is a proposal that manufacturers train the AI system separately from the software which is already on the market.³⁷ In this way, they could continue to train the product in a safe environment (for example, in a research department) without the need to re-certify the software at each learning step. Rather, the manufacturer could wait until, after a longer process of continuous learning, the product has improved to the point where they can release an update or upgrade that passes a new conformity assessment procedure.

Another way to address these issues could be to create a list of allowable changes and modifications that AI/ML-based medical devices can use to adapt to new data in real time without having to go through a new conformity assessment procedure, as long as the manufacturer has a quality management system in place to accommodate the modifications.³⁸ According to this proposal, documented changes that alter performance but do not change the intended use or input type could be allowed as long as they do not pose a safety risk to patients.³⁹ Only if the AI system exceeds the pre-defined scope of change or the original intended purpose, a new conformity assessment would be required.⁴⁰ However, it should be kept in mind that even foreseeable and permissible changes may accumulate over time and lead to a deviation from the performance for which conformity was certified. For this reason, it seems appropriate to require regular audits for systems that are allowed to alter their performance.⁴¹

2.2.3 Conformity Criteria for AI-based Medical Devices

The distinctive characteristics of medical devices based on ML raise a number of unresolved questions, even for ʻstatic’ or ʻlocked’ systems, regarding the criteria that must be met during the conformity assessment procedure. Two specific characteristics of ML systems seem particularly problematic. First, ML is functionally dependent on data and the quality of data. As a result, a medical device trained on a particular dataset may only be suitable for certain purposes or groups of people, such that it can predict diseases with sufficient accuracy only in certain individuals, but not in others. Closely related to this is the lack of predictability of some AI/ML models. Since learning systems are not explicitly programmed but are trained using thousands and millions of examples, so that the system evolves by learning from experience, the predictions or decisions of such systems might not be foreseeable in a particular case, even if the system relies on ʻlocked’ algorithms that no longer learn. In contrast, Annex I No 17.1 MDR requires that medical devices based on software or software that are devices themselves, shall be designed ʻto ensure repeatability, reliability and performance in line with their intended use’. Meeting these requirements can be challenging for ML systems, as their ʻautonomous behavior’ to generate outputs with limited or no human intervention can lead to unpredictable results that are difficult to assess at the time of the conformity assessment procedure.⁴²

A second concern is the ʻblack box’ nature of some ML systems, ie their opacity. Arguably, the MDR does not contain explicit provisions requiring medical devices to explain or render transparent decisions to the user, nor does the regulation preclude manufacturers from placing opaque devices on the market. However, according to the MDR, manufacturers have to demonstrate (and document) that the safety and performance of their devices reach the state of the art considering the devices’ intended use. Moreover, manufacturers must provide ʻsafety and performance information relevant to the user, or any other person, as appropriate’, tailored to ʻthe technical knowledge, experience, education or training of the intended user(s)’ (Article 23(1)(a) Annex I MDR). In order to comply with these requirements, a certain level of transparency is a prerequisite.⁴³

So far, neither regulators, notified bodies nor the literature have come to a consensus on how the MDR should be applied and, if necessary, improved or supplemented by the AI Act to address the previously described issues of AI-based medical devices (data dependency, lack of foreseeability due to autonomy, opacity). The last point mentioned, opacity, is of particular importance here, as it is currently unclear whether AI-based medical devices should have a minimum level of transparency before they can be released on the market. Some data scientists argue that regulators should only allow inherently interpretable algorithmic models but ban AI systems with algorithmic opacity that cannot be solved technically.⁴⁴ However, studies show that some opaque AI systems (eg deep neural networks) show a much higher degree of accuracy and efficiency than transparent systems (eg deductive and rule-based systems).⁴⁵ In such a situation, a trade-off must be made between AI’s accuracy and transparency.

The MDR (in contrast to the AIA)⁴⁶ provides for exactly such a balancing in that it allows certain risks to be admitted as acceptable if they are outweighed by the corresponding benefits (Annex I No 4 MDR). Hence, inherent algorithmic opacity could be considered an acceptable risk, if the manufacturer can provide evidence that the benefits of using such an AI medical device outweigh the risks.⁴⁷

2.2.4 Lack of Standards for Human-Robot Interaction

Finally, there is also the problem that there are currently no established standards for the communication and interaction between AI-based devices and patients. Current requirements are primarily concerned with the physical safety of medical devices. In contrast, there is still a lack of standards for the specific dangers to mental health and other patient rights arising from the human-robot interaction.⁴⁸

In the EU, the current legal framework relies on the basic divide between medical device law, on the one hand, and medical services law, on the other. While medical device law regulates (at European level) the physical hazards of a product before it is placed on the market, the laws and ethical principles governing medical practice (at Member State level) are intended to ensure high quality health care services and, in particular, the protection of core patient rights (such as autonomy, human dignity and trust).

However, this regulatory dualism of product versus service is called into question, as medical treatment is no longer provided by humans, but by AI systems interacting directly with patients. Obviously, the law applicable to healthcare professionals cannot guarantee that the AI system will be of sufficient quality when interacting with patients. After all, the quality of an AI system is determined by the program code and not by the skills of the human operating it. Policymakers are therefore rightly advised that robotic and AI applications engaged with patients will likely also need to be bound by similar ethical guidelines and legal requirements as those that apply to health professionals.⁴⁹

3.1.1 Scope of Application: The Contentious Definition of ʻAI’

The AIA has a rather broad scope of application, not only with regard to its territorial scope,⁵⁰ but also with regard to the definition of AI systems. According to the Act, an AI system is a ʻmachine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments’ (Article 3(1)).⁵¹ Recital 12 AIA states that the reference to ʻvarying levels of autonomy’ means that AI systems ʻhave some degree of independence of actions from human involvement and of capabilities to operate without human intervention’. However, it remains unclear what level of autonomy is required for software and robots to be considered ʻAI systems’. As a result, the scope of the AI Act appears to cover all types of medical software ranging from chatbots, medical device software embedded in electronic thermometers, blood glucose meters and patient ventilators, migraine or asthma episode prediction apps, to medical image analysis software for tumor detection.⁵²

3.1.2 Medical Devices as High-Risk AI Systems

The AI Act puts most emphasis on the regulation of so-called high-risk AI systems. According to the Act, high-risk AI systems are permitted on the European market; however, they are subject to compliance with mandatory requirements and an ex-ante conformity assessment before their market deployment (Article 8ff AIA). The mandatory requirements include: the creation of a risk management system (Article 9); quality criteria for training, validation and testing data (Article 10); technical documentation (Article 11 and Annex IV) and record-keeping (Article 12); provisions on transparency and user information (Article 13); obligations for human oversight (Article 14); and obligations concerning the accuracy, robustness, and cybersecurity of systems (Article 15).

However, all these requirements do not apply automatically. Most notably, the Act does not classify all AI applications used in the medical field as high-risk systems. Rather, ʻhigh risk’ includes only AI systems that are medical devices as defined in the MDR and subject to a third-party conformity assessment under that regulation (Article 6(1) and Annex I Section A, No 11 AIA). Since only medical devices classified as risk IIa MDR or higher under the MDR are required to undergo third-party conformity assessment, only these systems are regarded as high-risk under the AIA.⁵³

If a medical device is a high-risk AI system, the AIA complements the MDR in that the above-mentioned essential requirements for high-risk AI systems must be assessed as part of the existing conformity assessment procedure under the MDR (Articles 16(f) and 43(3) AIA). In practice, this means that the notified body conducting the conformity assessment under the MDR must not only review the requirements of the MDR, but also the specific requirements of the AIA for high-risk AI systems.