Doctor Chatbot: The EUʼs Regulatory Prescription for Generative Medical AI

1. Introduction

In recent years, the development and use of technologies involving artificial intelligence (AI) has proliferated in the EU and internationally. In the health sector, the increased use of AI raises several legal and policy issues.¹ With this article, we contribute to the discussion on the EUʼs regulation of medical AI from a product safety perspective. We focus on the EUʼs emerging product safety framework as it will exist after the future introduction of the proposed EU Artificial Intelligence Act (AIA).² In the context of medical AI systems, there will be considerable interplay between the AIA and the EUʼs Medical Device Regulation (MDR).³

The main purpose of this article is to search for regulatory lacunae that may exist in the space between the AIA proposal and the MDR in respect of medical AI systems. Because of the way in which the two legal frameworks are interconnected, we focus on potential lacunae stemming from the two risk classification schemes for medical AI systems, respectively included in the AIA and in the MDR. While others have also argued that there may exist lacunae in the future EU legal framework, consisting of risks that are not sufficiently managed when the two laws are read in conjunction,⁴ we rely on two specific case studies to demonstrate the problem and propose solutions.

We use the term ʻmedical AI systemsʼ to refer to health-related AI systems, regardless of whether they fall under the scope of the MDR. Like other products with medical purposes, AI systems may harm patients if they do not function as intended, or if they are developed or used without appropriate safeguards. Some emerging applications of AI for health may present similar safety risks as non-AI products, while other applications may pose risks that are more specific to AI-based products. The goal of the AIA is primarily to mitigate the latter category of risks, as it is assumed that the existing product safety framework can handle risks that are similar to those posed by non-AI products. If there are risks related to AI products that are not addressed either by the existing product safety framework (especially the MDR) or by the AIA, those risks constitute what we regard as regulatory lacunae for the purposes of this article.

While the AIA considers a broader range of technologies as ʻartificial intelligenceʼ, this article primarily contemplates generative AI systems employing technologies such as machine learning (ML) methods,⁵ including deep learning, as well as natural language processing (NLP) techniques. Of particular interest is the NLP-based subset of generative Large Language Models (LLMs).⁶ When referring to generative AI, or simply AI, we allude to the various technologies mentioned above.

This article is structured as follows. Section 2 introduces the prospects and risks of medical AI systems, exemplified by recent advances in generative LLMs. Section 3 presents an overview of the emerging regulatory framework for medical AI systems, which represents the necessary background for the present article. Subsequently, Sections 4 and 5 narrow the focus on the proposed AI Act and the Medical Device Regulation. Both laws present a form of risk classification, as explained in Section 6, but this implies the possibility that some risks remain unaddressed in the intersection between the two laws. We therefore examine the potential for lacunae between the two acts in two case studies. Section 7 addresses the case of therapeutic chatbots. These systems, potentially falling into the minimal risk class, do not need to comply with the safety requirement of the AIA, even though they arguably represent examples of relatively risky medical AI systems. Section 8 addresses the case of an automated documentation system for electronic health records, which potentially may not be subject to the main AIA requirements since its classification as a ʻmedical deviceʼ is uncertain. Section 9 discusses implications of the proposed case studies and engages with existing proposals of how to address the risks of medical AI systems. Section 10 proposes solutions that we find to be compatible with the intentions of the EU legislature when it comes to the regulation of medical AI systems.

2. The Prospects and Risks of Generative Medical AI Systems

AI systems are already being used to some extent in the health sector and are expected to be used more widely in the near future.⁷ AI can be used to support clinical tasks such as diagnoses,⁸ treatment recommendations,⁹ patient monitoring,¹⁰ and risk assessments,¹¹ as well as administrative tasks related to, eg, resource management,¹² or to automate certain workflows or procedures, such as surgical procedures.¹³ Moreover, AI systems may be used to monitor, assess, or treat patients remotely through mobile and wearable devices.¹⁴

Recent advances in AI development particularly illustrate the future potential of generative AI technologies. The term ʻgenerative AIʼ refers to AI models that are designed to generate or create new content, such as images, video, or text, that is similar to the training data the models have been exposed to. For example, the development of Large Language Models (LLMs) has enabled the creation of advanced chatbots such as ChatGPT. Because systems like ChatGPT are optimised for dialogue, they can respond to conversational prompts and adapt their responses to new input. They can produce answers that seem coherent and thoughtful, despite the fact that these systems cannot ʻthinkʼ in a traditional sense.¹⁵ The conversational capabilities of current LLMs provide a preliminary indication of the prospects of future generative medical AI systems. It is not inconceivable that conversational chatbots could be used to treat certain types of mental illness, mimicking conversations with a therapist.¹⁶ As such, chatbots could act as virtual therapists and make therapy – a limited resource – more easily accessible to patients. In this case, it would be important to ensure that the system responds adequately to problems, such as indications of suicide risk, or that it does not suggest the consumption of alcohol to treat depression. Risks related to therapeutic chatbots are further discussed in section 7.

Different applications may expose patients to different kinds of risk. This article further discusses how risks related to medical AI systems with different degrees of generativity can be mitigated through EU regulatory measures. The wider legal context for such measures can be found either in medical law, AI law or in a combination of both fields. This article focuses especially on the intersection of AI and medical law, since certain risks associated with medical AI systems may slip through the gaps between two key legal instruments, the AIA and the MDR.

3. The Emerging EU Regulatory Framework for Medical AI Systems

The use of medical AI systems can trigger, in principle, a variety of legal rules, including those related to the processing of personal data in the EU General Data Protection Regulation (and supplementing national laws),¹⁷ or rules governing the provision of healthcare, which are typically found in national law. In addition, the introduction of AI systems on the European market must comply with current and upcoming safety rules, which are the main focus of this article. For certain types of products which are deemed to present risks to the health and safety of EU citizens, access to the common market is contingent on conformity with harmonised safety and performance requirements laid down in EU law.

The purpose of product safety legislation at the EU level is twofold: to ensure a high level of protection from health and safety risks, and to facilitate innovation and free trade of products across EU member states. These legislative aims are equally important. Therefore, the substantive requirements to which products need to conform, as well as the procedures through which conformity may be demonstrated (conformity assessment procedures), are designed with the intent of striking an appropriate balance between them. To determine what the appropriate balance is, the EU legislature relies on risk assessments.¹⁸ Through risk assessments, the legislature anticipates health and safety risks associated with a category of products (eg toys, lifts, radio equipment, medical devices and soon also AI systems). Each product category is regulated according to the anticipated types of risk. For instance, toy manufacturers are obligated to consider and address elements such as a toyʼs physical attributes (eg its sharpness and the possibility of it obstructing a childʼs airway), its flammability and its toxicity.¹⁹ Medical AI systems can imply risks from different spheres, including risks typical to the medical context (eg misdiagnosis), as well as typical AI risks (eg automation bias). These risk types are addressed in current and proposed EU laws.

Certain medical AI systems will be classified as medical devices and thus be regulated under the MDR, particularly when they are incorporated within a physical medical device.²⁰ For instance, if an AI system is incorporated into a pacemaker, the pacemaker is the medical device, and the AI system is an integrated component of the pacemaker. However, the focus of this article is on ʻstandaloneʼ medical AI systems, ie medical AI systems that serve their purpose without being incorporated into another medical device. Such a ʻstandaloneʼ AI system may in itself constitute a medical device under the MDR. The term ʻAI medical devicesʼ is used in this article to denote standalone AI systems that constitute medical devices.

Medical devices generally come in many different shapes and are used for a variety of tasks. Because of their build and the use for which they are intended, some devices are more dangerous (risky) than others. The MDRʼs risk classification scheme is meant to ensure that more risky devices are subject to heavier scrutiny before they can be placed on the common market. For less risky devices, the path to the common market is easier. This is how the EU strikes a balance between the dual aims of facilitating innovation in the medical space and protecting citizens from unjustified health and safety risks. Therefore, the risk classification schemes – for medical devices and for other product categories – are crucial to the functioning of the common market. When new technologies emerge – potentially presenting other types of risk than those anticipated by the existing legislative risk assessments – the ability of existing risk classification schemes to ensure a well-functioning marketplace may be challenged.

The EU Commission has proposed a common European framework for artificial intelligence (the AIA).²¹ The proposed AIA lays down requirements that AI systems and the ʻprovidersʼ of AI systems (in various sectors) must satisfy before the systems can be placed on the common market.²² In this sense, the proposal appears as a product safety act for AI systems, and applies on top of the already existing sector-specific product requirements that follow from the EUʼs product safety framework (the ʻNew Legislative Frameworkʼ/ʼNLFʼ), including the MDR.²³ In addition to addressing safety aspects of AI systems, the AIA aims to ensure that AI systems respect existing laws on fundamental rights and Union values and to enhance the effective enforcement of fundamental rights.²⁴ The EU Commission has made it clear that it only intends to impose new rules for AI to the extent that is necessary to protect EU citizens from AI-related risks in so far as those risks are not already addressed in existing EU legislation.²⁵ Thus, the AIA is meant to address the specific risks posed by AI systems.²⁶ These risks are typically not addressed by the existing legal framework, which focuses on more traditional products and their related risks.²⁷

To ensure that the scope of the AIA is adapted to the risks posed by AI systems, the Commissionʼs proposal relies on a preparatory risk assessment.²⁸ Based on this legislative risk assessment, the Commissionʼs proposal prohibits certain AI systems because the risks associated with these systems are deemed to be unacceptable.²⁹ Further, the proposed AIA sets out classification rules for distinguishing between ʻhigh-riskʼ and non-ʻhigh-riskʼ AI systems. The classification determines the requirements that will apply to the AI systems. While ʻhigh-riskʼ AI systems are subject to a set of material requirements in Title III of the proposed AIA, AI systems classified as lower risk will only be subject to minimal transparency requirements or not be regulated by the AIA. In practice, this means that the most important requirements of the AIA only apply to AI systems that are classified as ʻhigh-riskʼ. Therefore, it is crucial to determine the risk classification of any AI system, including medical AI systems, in order then to determine whether they reach the ʻhigh-riskʼ threshold. If the threshold is reached, the AIAʼs requirements apply, in addition to any other requirements, for example those included in the MDR. If the threshold is not reached, the respective AI system does not need to conform to the main requirements of the AIA. This is where the AIA is linked to the MDR and its risk classification scheme for medical devices.

As mentioned above, medical devices are also placed in different risk classes under the MDR according to a set of risk classification rules. The AIA relies on this risk classification scheme in order to determine whether additional requirements (ie those that are found in the AIA) apply. Although the details are slightly more complicated, as elaborated below, the AIAʼs requirements essentially apply in addition to the requirements of the MDR, if a medical AI system reaches a specific risk threshold in the MDR. On the flip side, the AIAʼs requirements are not applicable if a medical AI system is not classified as a medical device at all, or if it is classified in a risk class below a specific threshold.³⁰ This implies the possibility that some medical AI systems might not be subject to the AIAʼs requirements. This is not necessarily a problem as long as these systems do not imply a high level of AI-specific risks.

4. ʻHigh-Riskʼ AI systems in the Artificial Intelligence Act

In the following, we concentrate on the requirements for ʻhigh-riskʼ AI systems in the AIA.³¹ This means that we disregard, inter alia, the prohibited AI systems, as it is unlikely that many medical AI systems will fall into this category, although the possibility cannot be completely excluded.³² This article is primarily interested in understanding the delimitation between ʻhigh-riskʼ AI systems, which are subject to specific requirements, and non-ʻhigh-riskʼ AI systems, which remain essentially³³ unregulated by the AIA. Many AI systems do not reach the threshold of ʻhigh-riskʼ and are therefore not subject to the AIAʼs rules for ʻhigh-riskʼ AI systems. For example, a system to detect spam in an email inbox is not considered sufficiently risky to justify stringent requirements.

Title III of the proposed AIA sets out the requirements aimed at AI systems classified as ʻhigh-riskʼ. Providers of ʻhigh-riskʼ AI systems must establish a risk management system and a quality management system, implement data governance measures including an examination in view of possible biases, and draw up technical documentation.³⁴ Furthermore, the ʻhigh-riskʼ AI system and the training data used to develop the system must comply with the requirements concerning the statistical properties of the data, technical documentation, record-keeping functionalities, system transparency, information to users, human oversight functionalities and requirements relating to accuracy, robustness and cybersecurity.³⁵ Finally, the provider must ensure that AI systems undergo the relevant conformity assessment procedure.³⁶ Having demonstrated compliance, the provider shall draw up an EU declaration of conformity and affix the CE marking of conformity.³⁷

Because the Title III requirements in the AIA do not apply to non-ʻhigh-riskʼ AI systems, there will be a significant difference in the material requirements that are applicable to ʻhigh-riskʼ AI systems and other AI systems. It follows that the role of the AIAʼs risk classification scheme is to distinguish between AI systems on which it is justified to impose the material requirements, and AI systems that ought to be allowed market access without adhering to those requirements, because they present less risk. The risk classification scheme is how the AIA strikes a balance between the two foundational values of the traditional common market for products: health/safety and innovation/trade.³⁸

Although this could be further discussed, this article assumes that the AIA works well if it regulates an AI system that in fact presents risks that justify regulation. This leaves open the possibility of two types of errors. This article does not further address the possibility that some AI systems are regulated under the AIA despite the fact that they do not present sufficient risks. Instead, we concentrate on the other type of error, ie the possibility that a medical AI system remains unregulated even though it should be, because it is risky. This means that we address the possibility of a mismatch, or lacuna, in which a medical AI system is not classified as ʻhigh-riskʼ in the AIA, despite the system implying risks that ought to be managed by applying the requirements of the AIA. With this focus in mind, we now turn to the mechanism for classifying ʻhigh-riskʼ AI systems in the AIA.

There are two ways through which an AI system can fall into the AIAʼs ʻhigh-riskʼ category. First, Annex III in the proposed AIA lists AI systems that shall be considered as ʻhigh-riskʼ.³⁹ Most systems in this list are of limited relevance to medical AI, as they concentrate on contexts such as employment, law enforcement, the administration of justice, etc.⁴⁰ Annex III focuses on specific purposes for AI systems that are classified as ʻhigh-riskʼ. In addition, in light of the discussions within the European Council and Parliament, it is anticipated that the AIA will take into account the significance of general-purpose AI systems, such as LLMs not intended for specific purposes. These systems will also be subject to the requirements for ʻhigh-riskʼ AI systems if they can be used for ʻhigh-riskʼ applications. The specific details of these regulations are still being negotiated on a political level. However, it seems that complying with at least some key requirements of the AIA will likely be mandatory, unless the provider of a general-purpose AI system explicitly excludes all ʻhigh-riskʼ uses in the systemʼs instructions.

The second option is more cumbersome to explain, as it consists in a connection between the risk levels in the AIA and other NLF acts. The most significant NLF act in the present context is the MDR. In a slightly simplified manner, we can say the AIA classifies certain medical devices – considered particularly risky – as ʻhigh-riskʼ. To determine whether a medical device is ʻhigh-riskʼ, the AIA relies on one specific criterion: whether the MDR stipulates that a device is subject to a conformity assessment procedure involving a third party.⁴¹ The significance of conformity assessments is further explained below. At this point, it suffices to state that the AIAʼs risk classification rules directly connect to the MDR risk classification rules. This makes the classification rules in the MDR the central object of analysis when exploring the risk classification scheme for AI medical devices. In order to determine whether a medical AI system is regulated by the requirements of the AIA, we need to assess its classification under the MDR. This implies that the AI system must be classified as a medical device (as further discussed in Section 5) and, in addition, reach the MDR-specific risk threshold where the AI system is deemed risky enough to warrant the involvement of a third party in the conformity assessment of the system (discussed in Section 6).

5. The Medical Device Regulation and AI as a Software Medical Device

For stakeholders in the health sector, the AIA is a new layer of product safety legislation which is added on top of the MDR. The MDR lays down requirements to which medical devices must conform before they can be placed on the EU market. The MDR, besides governing manufacturers, also imposes specific obligations on distributors, importers and users, which include healthcare institutions.⁴² However, the manufacturer remains the most salient duty-subject under the MDR.⁴³ Above all, the manufacturer is responsible for ensuring that their devices conform to the applicable MDR requirements. It is the manufacturer who initially determines which risk category a medical device belongs to.⁴⁴ Thus, it is important that manufacturers have proper methods for assessing the risk posed by their devices.⁴⁵

AI medical devices, as defined in Section 3, are a subcategory of software medical devices and regulated as such under the MDR.⁴⁶ A ʻmedical deviceʼ, according to Article 2(1) MDR, is ʻany instrument, apparatus, appliance, software […] intended by the manufacturer to be used, alone or in combination, for human beingsʼ for one or more specific medical purposes, including ʻdiagnosis, prevention, monitoring, prediction, prognosis, treatment or alleviation of diseaseʼ.⁴⁷ Medical devices can, according to this definition, be as basic as hospital beds or face masks. Software systems are medical devices when they are intended to be used for medical purposes, unless their intended use is too generic; guidance from the Medical Device Coordination Group (MDCG) indicates that software systems only qualify as medical devices if they are intended for medical uses directed at individual patients.⁴⁸ Thus, AI systems intended to be used to analyse large amounts of patient data to discover patterns and extract new knowledge, or to aid in strictly administrative decision-making processes, are not medical devices. For example, AI can be used to support decisions concerning high-level management of (human or other) resources, or to provide the basis for changes in how a health institution is organised or how health services are provided (eg, due to indications of a potential for quality improvement, increased efficiency, etc). Such AI systems can all be referred to as ʻmedical AIʼ, but they are not ʻmedical devicesʼ pursuant to the MDR.⁴⁹ As a consequence, they would only be subject to the requirements for ʻhigh-riskʼ AI systems under the AIA if integrated in the list included in Annex III to the AIA.

An AI medical device may take the shape of an application that anyone can download and use anytime, anywhere, on their personal device (ʻmHealthʼ).⁵⁰ An AI medical device may also be a software system operating on a computer in a physicianʼs office, in an intensive care unit or in another setting where a health institution and/or clinician is in charge of how the device is used.⁵¹ As long as an AI system that falls under the definition of a ʻmedical deviceʼ operates on a hardware device that is not in itself a medical device, the AI system constitutes a so-called ʻstandaloneʼ software medical device.⁵² For risk classification purposes, standalone software medical devices are classified in their own right under the MDRʼs risk classification scheme.⁵³

AI medical devices set themselves apart from conventional, hard-coded medical software systems. This is due to their capacity to learn from examples and formulate their own decision rules, rather than relying on human software engineers to generate decision rules for every potential scenario. Consequently, medical AI systems may present risks that differ from those posed by traditional software systems.⁵⁴ The AIAʼs explanatory memorandum highlights several key traits of AI systems that are closely linked to safety and fundamental rights risks. These relate to ʻopacity, complexity, bias, a certain degree of unpredictability and partially autonomous behaviourʼ.⁵⁵ Such traits may be especially pertinent to generative AI systems. Thus, to align with the objectives of the EU legislature in mitigating AI-specific risks, it is vital that the risk classification scheme appropriately addresses generative AI systems. Bearing this consideration in mind, we now turn our attention to the AIAʼs risk classification scheme for medical AI systems.

6. Risk Classification of Medical AI Systems

7. Case Study 1: Therapeutic Chatbots

8. Case study 2: An Automated Health Record Documentation System

8.3 Conclusion for Case Study 2

The degree of generativity in an AI system for automated EHR documentation can determine whether the system qualifies as a medical device under the MDR and whether the main requirements in the AIA apply to the system. Arguably, the directions drawn up by the CJEU for the assessment of whether a software is a medical device indicate a low threshold of generativity; if a software systemʼs data processing involves only a modicum of interpretation or modification, it seems likely that the system should be deemed as a medical device. However, erroneous information outputted from an AI system can have serious consequences regardless of the systemʼs degree of generativity. From a safety perspective, the transcription of spoken language into text (low generativity) poses risks that are similar to those posed by more generative systems. This arguably represents an example of a borderline case where an AI system with health and safety impacts may or may not qualify as a medical device. Other borderline cases may occur as AI systems continue to spread throughout the health sector.

Figure 1

9. Discussion

The EU legislatureʼs anticipation of product risks is crucial to facilitate a marketplace where citizens can trust that the available products are safe to use. AI products may present new risks that are specific to AI technologies and risks that are similar to well-known risks associated with existing product types. The proposed AIA is meant to protect EU citizens from AI-specific risks. When it comes to more traditional risks, the AIA relies on existing EU product safety legislation (the NLF).

There are good reasons why the AIA relies on existing product safety legislation. When an AI system is regulated by an existing NLF act, it means that the AI system falls into a category of products generally associated with certain safety risks. Those safety risks are addressed by the existing requirements and conformity assessment procedures for the relevant product category. A significant infrastructure, involving public authorities and private industry, is already established around the process of verifying different productsʼ conformity with the applicable requirements.¹¹¹ Therefore, it is perfectly reasonable for the EU legislature to focus its legislative and regulatory efforts on new, AI-specific risks. However, this article has demonstrated how the AIAʼs risk classification scheme for medical devices may potentially fail to capture certain AI-specific risks as well as certain traditional risks associated with medical devices. In other words, we have identified potential regulatory lacunae that are not filled in by the proposed AIA.

First, this article has demonstrated that certain AI medical devices that pose AI-specific risks might not be subject to the AIA requirements, even though those requirements are designed to mitigate such risks. This is because the MDR risk classification is decisive for their applicability. The MDRʼs risk classification scheme primarily focuses on risks arising from inaccurate information used as a basis for clinical decisions. However, some AI medical devices may present AI-specific risks unrelated to the provision of inaccurate information and could be classified as class I under the MDR. These risks may not be captured by the risk classification scheme for AI medical devices, as a result of the AIAʼs reliance on the MDR classification scheme. The implications of this issue have been illustrated by the example of AI-based therapeutic chatbots. The potential for AI-based therapeutic chatbots operating with limited clinical oversight has been fortified by recent advancements in generative LLMs and is not anticipated by the MDR.¹¹²

Second, the article has confirmed that the risk classification scheme for medical AI systems opens up to a potential regulatory lacuna at the outskirts of the MDRʼs area of application: medical AI systems not mentioned in Annex III of the AIA are only placed in the AIAʼs ʻhigh-riskʼ category if they are defined as medical devices.¹¹³ Therefore, medical AI systems that are not medical devices escape the requirements for ʻhigh-riskʼ AI systems, even if they pose more traditional types of risks, ie risks that the MDR is designed to mitigate. This leaves the AIAʼs risk classification scheme vulnerable to speculative formulations of a medical AI systemʼs intended purpose. While others have made this point in relation to potentially risky wellness applications, which may be marketed as being intended for non-medical purposes,¹¹⁴ we have shown that the intended generativity of a medical AI system may determine whether it constitutes a medical device. Different levels of generativity can produce borderline cases that may be challenging to regulate. In borderline cases, providers might be inclined to downplay the degree of generativity in a medical AI system, thus evading MDR and AIA requirements.

To address the gap concerning medical AI systems falling outside the scope of the MDR, one specific solution proposed in the literature is to add these systems to the list of ʻhigh-riskʼ AI systems in Annex III of the AIA.¹¹⁵ This approach suggests including ʻmedical or health-related AIʼ in the list, which would effectively address the challenges outlined in this article. However, it is important to evaluate carefully whether this solution may generate new challenges.¹¹⁶ If all health-related AI systems are categorically defined as ʻhigh-riskʼ, there would be no opt-out for providers of applications that operate at the periphery of the medical context and potentially pose lower risks compared to other systems listed in Annex III. Such a broad inclusion would undoubtedly create compliance challenges for a diverse and loosely defined set of AI systems connected to the health sector. Therefore, justification for this inclusion would be warranted only if these systems exhibit a high level of risk specifically associated with their utilisation of AI.

10. Conclusion and Recommendation

The EU legislature and national legislators are advised to consider the possibility of medical AI systems that either fall into MDR class I or fall outside the scope of the MDR because they are not medical devices. Because the AIA relies on MDR classification, these systems are not covered by the AIAʼs ʻhigh-riskʼ category, unless they are listed in Annex III of the AIA. To improve the risk classification scheme for medical AI in terms of capturing relevant health and safety risks, there are at least two approaches that may be considered: (i) the addition of certain (but not necessarily all) medical AI systems to the list of ʻhigh-riskʼ AI systems in Annex III of the AIA; and (ii) the addition of AI-specific classification rules in the MDR (at least to the extent needed to address therapeutic chatbots).

In relation to option (i), it was noted in section 9 above that an inclusion of all medical AI systems in the AIAʼs list of ʻhigh-riskʼ systems might not be justified. An alternative solution could be to use the MDRʼs definition of a medical device as a foundation for a more targeted and specific inclusion of essential medical purposes within Annex III of the AIA. This solution would imply that all AI medical devices, but not all medical AI systems, should be placed in the AIAʼs ʻhigh-riskʼ category. A key advantage of this solution is that it enables the AIAʼs risk classification scheme to capture a medical AI systemʼs degree of generativity (which is associated with AI-specific risk). In our interpretation of a ʻmedical deviceʼ under the MDR, we found that the degree of generativity may be a decisive criterion.¹¹⁷

If the EU legislature prefers for the AIA to give deference to the MDRʼs risk classification scheme, rather than adding certain medical AI systems to the list in Annex III of the AIA, option (ii) may be chosen. To address the problem of AI medical devices potentially being classified as low risk (MDR class I), the MDRʼs classification rules could be amended with a risk classification rule for generative AI medical devices capable of providing treatment responses autonomously. This would be an even more targeted solution than option (i), with less risk of unintentionally subjecting low-risk AI systems to ʻhigh-riskʼ requirements. However, this approach would not remedy the lacunae of risky medical AI systems that are potentially not defined as medical devices. To some extent, it may be possible to mend this gap through teleological interpretation of the MDRʼs definition of a ʻmedical deviceʼ. Given the MDRʼs objective to effectively manage specific medical risks, a teleological interpretation could permit those risks to be considered when assessing whether an AI system is a medical device. For example, it could be argued that the borderline case of a low-generativity AI system for EHR documentation should be deemed a medical device because it poses a type of risk that is typical of medical devices (the risk of erroneous information serving as a basis for clinical decisions). By incorporating risk considerations into the interpretation of the ʻmedical deviceʼ definition, the MDR could uphold proportionality between innovation and regulation, while remaining aligned with its intended objectives.

Funding and Acknowledgments

Parts of this research are supported by the project ʻData-Driven Health Technologyʼ (project number 310230101), funded by UiT the Arctic University of Norway, and the VIROS project, funded by the Norwegian Research Council (project number 247947). We thank Timo Minssen for extensive comments and suggestions to earlier versions of the manuscript. We also thank Christian Johner and Brita Elvevåg for valuable comments to an earlier version. Finally, we thank the reviewer for insightful comments. Any remaining errors are the sole responsibility of the authors.