Conceptualizing an AI-based Police Robot for Preventing Online Child Sexual Exploitation and Abuse:

2.1 Understanding PrevBOTʼs functions

Understanding PrevBOTʼs functions concerning categorization and identification is crucial to the legal analysis. This section describes their purpose and how they relate to the crime preventive mandate of the police. As for the latter, a three-tiered crime preventive model is applied. While generally applied in public health intervention, the modelʼs broader relevance, i.a., to the field of crime prevention in the police and criminal justice sector is increasingly recognized (Stokols, 2018; Muir, 2021).

2.2 PrevBOTʼs functions

ʻCategorizationʼ refers to PrevBOTʼs capability to predict online participantsʼ age and/or gender and the occurrence of sexualized speech in a public chat room. ʻIdentificationʼ refers to its capability to predict the presence of known CSEA offenders. In real time, it compares the writing style in chat conversations to ʻlinguistic fingerprintsʼ of known CSEA offenders in a police reference repository. The reference material is made by computation of chat logs collected during investigations of previous offenses. Rather than suggesting a single match, PrevBOT produces a list of linguistic fingerprint candidates to a match, similar to the procedure applied to fingerprints and DNA samples.² Both functions are enabled by machine learning and Authorship Analysis.

PrevBOT is rooted in situational crime perspectives (SCP) and extension theory. SCP takes the offenderʼs motivation for granted, “instead seeking to limit their scope for offending, and influence their perception and decisions regarding criminal action” (Part I, section 5.1). Further, with reference to extension theory, PrevBOT “extends the ability of [police] presence in more spaces than otherwise possible, allowing the entering and monitoring of more chatrooms per police officer” (Part I, section 5.5).

PrevBOTʼs categorizations are output in the form of predictions. Predictions of the existence and frequency of sexualized speech provide the police with information relevant to classifying specific forums as high-risk to children (ʻProblematic Spacesʼ). Problematic Spaces can then be prioritized for closer monitoring, enabling PrevBOT to detect persons who target children while pretending to be younger and/or of different gender than indicated in the user profile or the chat (ʻProblematic Personʼ (PP)). The human operator may also use PrevBOT as vehicle for sending a preventative message to the PP.

By ʻunmaskingʼ adults who pose as youngsters and/or lie about their gender, PrevBOTʼs predictions make it more difficult for would-be offenders, thus limiting the ʻscopeʼ for offending. Besides, the preventative message may influence their perceptions and decisions regarding CSEA. More generally, police ʻvisibilityʼ may be a psychological effect of the policeʼs improved ability to identify and monitor Problematic Spaces, detect PPs, and intervene against CSEA. Like police visibility in physical space, this may have a preventive effect.

PrevBOTʼs identifications (a linguistic fingerprint match) indicate that the PP is a known CSEA offender. In combination with the asymmetric age constellation between the parties to the conversation, plus the fact that online CSEA offenders often have many victims (Part I, section 2; Sunde 2019; 2020; Nilssen 2021), there might be reasonable ground to open a criminal investigation where the PP is suspected of still abusing children sexually. In line with SCP, identification limits the offenderʼs possibility for reoffending, while criminal investigation provides an entry point for intervention that could end CSEA abuse of other children.

2.3 Placing the functions in the preventive model

In the preventive model, primary and tertiary prevention respectively refer to prevention addressing causes and symptoms, while secondary prevention refers to early intervention (Muir, 2021, p. 9). Application of the model must be adapted to the nature of the problem at hand and the actors involved in preventing it. Crime prevention often requires multi-agency collaboration, involving the efforts of more actors than the police.

As a rule, the police should stay out of primary prevention, as other actors regularly are better placed to address the cause of a problem (Muir, 2021, p. 30–33). There is also the risk that the police “[become] intrusively pervasive across much of social life” (Loader, 2020, p. 12), resulting in increased surveillance and interference with personsʼ fundamental rights.

PrevBOTʼs categorizations aim at early intervention. Although online CSEA became a manifest problem of worrying dimensions many years ago, the ʻwindowʼ for secondary prevention is open with respect to the occurrence of each criminal offense. It is highly relevant that the police target social groups deemed to be a risk of committing certain crimes. The European Code of Police Ethics (2001) (ECPE) states that crime prevention is a main objective of the police (Article I.1), and a mandate to proactively intervene may be unequivocally expressed in law, e.g., section 7(1)(3) of the Norwegian Police Act (1995).

Muir takes CSEA as a use case to illustrate the model. Muir relies, like PrevBOT, on SPC theory, proposing to implement processes of “identity authentication and age verification in the online environment” to “design out” the CSEA problem (2021, p. 14). This would “reduce the opportunities to offend” by making it more difficult for would-be offenders to give false information about their age, thus making online spaces safer for children.

Addressing causes, Muir places responsibility on ʻthose best placedʼ, i.e., the internet and software industry. PrevBOTʼs intervention, however, is not primary akin to Muirʼs, but secondary, the purpose being to intervene early, before the PP gets control of the victim. In this situation, the police are clearly mandated to intervene.

The identifying function is mainly reactive, resting on the preventive effects of criminal investigation and punishment. Incapacitation, combined with the renewed possibility to approach the offender and the general deterrent effect of the criminal justice process, is deemed to have a (tertiary) preventive effect. Furthermore, private vigilantism against CSEA offenders is an international phenomenon (Lomell, 2020) and cautioned against (Halnes & Ugelvik, 2019; Lomell, 2020). The phenomenon could be reduced through application of PrevBOT, as increased effectivity of the police may reinforce public trust in the rule of law (Muir, 2021, p. 31).

There is also a proactive dimension to criminal investigation and prosecution: CSEA victims regularly keep the sexual abuse secret due to shame, and because they are silenced by threats from the offender to publish sexualized images on social media for schoolmates and parents to see (Part I, section 2; Sunde, 2019; 2020; Nilssen, 2021). Terre des Hommes (2013) emphasized that proactive investigation by the police is a precondition for coming to grips with the CSEA problem. PrevBOT may provide information necessary for investigating known CSEA offenders who resume the criminal conduct. This can proactively end the continued victimization of children who do not report, thus reducing harm.

Summarized, PrevBOTʼs categorization and identification functions answer to an evidence-based, problem-oriented approach anchored in sound theoretical foundations, enabling interventions on secondary and tertiary levels, where the police properly may use powers.

3.1 Efficiency and effectivity

PrevBOTʼs autonomous properties enable policing of online forums and are superior to manual patrolling against CSEA both in terms of efficiency and effectivity. As PrevBOT can be put into operation simultaneously in several chat rooms and monitor significantly more chat conversations than would be possible for a police officer, it is more efficient than manual patrolling. Moreover, in public chat rooms, the conversations are open for everyone to see. However, CSEA offenders are risk-sensitive and make quick moves (Part I, section 2.2). To minimize the risk of detection they are likely to try to move the chat to a private channel for the conversation to continue one-to-one (Broome, Izura & Davies, 2020; Lorenzo-Dus, Kinzel & Cristofaro, 2020). This significantly reduces the possibility of visually observing CSEA risk indications in public chat rooms, which is the option available to a police officer.

PrevBOT is better equipped for the task through automated processing of available data, thus categorizing participants according to relevant risk indicators and identifying former CSEA offenders present in the chat room. Furthermore, PrevBOT is superior in collecting the digital traces needed to retrace offenders. This makes PrevBOT a more effective alternative to manual patrolling, improving the rate at which the police may submit preventative messages or initiate a criminal investigation.

3.2 The preventative message

PrevBOT may engage in secondary intervention by submitting a preventative message to the PP. The message informs the PP that s/he has chatted with the police and that contacting a child for sexual purposes is a punishable offense. In addition, the message may motivate the PP to seek professional help and inform about relevant programs for sex offender treatment. To tackle the risk of messaging an innocent (false positive), the message can facilitate easy feedback, e.g., through a link which provides information on how to file a complaint if s/he feels unfairly targeted and contact information for a complaint service in the police.

The police officer and PrevBOT are equally positioned concerning the preventative message: as participants to the chat, both can submit a message to the conversation partner (the PP). However, as predictions of age and/or gender are needed, only PrevBOT may effectively get the opportunity to submit such a message.

3.3 Criminal investigation of serial CSEA offenders

As per criminal procedural law, a criminal investigation is opened pursuant to a decision by a duly authorized person. The decision must be based on a reasonable suspicion that a crime was committed, substantiated by concrete indications. PrevBOT is a tool for gaining such information through linguistic fingerprinting, age/gender classifications, and information obtained in the conversation with the PP. In addition, it collects electronic traces that are important in the initial phase of the investigation. PrevBOT does not play a role in the investigation per se. The investigation makes use of traditional investigation methods by retracing the suspect and following up with methods such as a house and computer search and police interview.

Some options are equally open to police officers and PrevBOT, such as posting a general warning in the chat room or alerting the platform owner/moderator of PP presence. Yet, the important point remains: that the police officer is less efficient and less likely to gain actionable information in these situations and is thus less effective.

5.1 Accessing the public chat room

The first question concerns PrevBOT entering a publicly accessible online forum. PrevBOT may enter the forum with a user profile indicating a child, for instance ʻLisa12ʼ or ʻRoger14ʼ. Another option is using a profile indicating an adult. There is even the option to use a profile that discloses the police identity. It is uncontroversial that the police may enter open sources on the internet, as expressed in Article 32(a) of the Cybercrime Convention, further clarified in T-CY (2014) section 3 (see also Lentz, 2019, p. 32). The right to access is not conditional on the disclosure of police identity.

As noted by Lentz (2018; 2019, p. 244) and Rønn (2022, p. 159), the distinction between public and private forums is not always clear due to the semi-private character of many online forums and groups. This might introduce uncertainty in concrete situations as to whether the police may enter and observe. A discussion of these problems is beyond the scope of this article hence the analysis assumes that the forum is an open source within the meaning of the law. Insofar as the police officer lawfully could have entered the forum, so may PrevBOT.

5.2 Observation

5.3 Infiltration

ʻInfiltrationʼ means a police officer has engaged in a conversation without revealing the police identity (the General Attorney, 2018). At present, the assumption is that PrevBOT uses a profile indicating a child. This could attract the interest of would-be CSEA offenders.

Should the PP perform speech acts, e.g., ʻlascivious speechʼ, ʻgroomingʼ, or attempt to obtain sexualized child images from PrevBOT, these are punishable offenses. The fact that PrevBOT is a computer system and not a child is not exonerating; the acts are still punishable as criminal attempts (Schermer et al., 2016, section 3.5; Sunde, 2019).

The PP may suggest moving to a private channel to improve the chances of succeeding in obtaining sexual ʻservicesʼ from the ʻchildʼ. Depending on the circumstances, the activity in the private chat room could constitute a criminal CSEA attempt against PrevBOT, perhaps also revealing indications that the PP is a serial offender.

6.1 The problem

The LED Article 10 imposes stricter conditions for processing of special categories of personal data than personal data in general. The categories in Article 10 include, i.a., “biometric data for the purpose of uniquely identifying a natural person”, further discussed below in relation to PrevBOT.

6.2 Biometric data

In a study for the European Parliament, Fuster and Peeters (2021, p. 6) note that biometric technologies utilize various body characteristics. These are ʻsourcesʼ or ʻreference measuresʼ of biometric data because they “can be used for the collection of personal data” (Fuster & Peeters 2021, p. 6, with reference to Opinion 3/2012 of the Article 29 Working Party).⁴ Further, the fact that biometric data are “by definition enabling the unique identification of individuals, is of particular value” (Fuster & Peeters 2021, p. 6).

For PrevBOT, the relationship between personal/biometric data and the source needs conceptual clarification. As a start, biological material such as blood samples or human tissue is illustrative: the material is the source, affording a possibility to collect personal data from it. Because the data are extracted from unique bodily characteristics, they are also biometric. The biological material per se is not personal data (Fuster & Peeters, 2021, p. 6, with reference to the Article 29 Working Party, 2012: “sources of biometric data shall not be considered biometric data themselves”). In the case of PrevBOT, the chatlogs are the source from which the biometric data are collected through technical processing. The chatlogs are also personal data because the content relates to a person who can be identified. The content probably provides personal details as well, as the purpose is to get in contact with others. The chat logs in visual form are thus personal data.

The source of the biometric data is not the visual content but the writing style of the participant. This is a behavioral characteristic, like gait, hand-written signature, way of using a keyboard or a mouse. PrevBOT analyses writing style based on machine learning applied to Authorship Analysis, and the output is biometric data.

The LED qualifies the definition of biometric data to comprise only such data that “allow or confirm the unique identification” of a person (Article 3(13); see also Article 10). Hence for Article 10 to be relevant to PrevBOT, the output must allow or confirm the unique identification of the participant to the chat conversation. The question is whether this is the case or not.

6.3 The linguistic fingerprint

PrevBOTʼs linguistic fingerprints do not ʻconfirmʼ identity, as the identities of participants in chat rooms are unknown to the police, but a match may ʻallowʼ identification of a person. However, the identification must also be ʻuniqueʼ, which raises the question of whether the match must be 100 percent accurate. False positives may “identify” more persons than one, in which case the data do not uniquely identify the correct person. Does this render Article 10 inapplicable? Arguably, the application of such logic would contradict the very purpose of the directive, i.e., to provide “a high level of protection of personal data” (Preamble, Recitals 4 and 7). Motivated by the great potential for person-oriented mass surveillance and risk of harm to fundamental rights by police use of biometric data, the LED aims at securing an even higher level of protection of such data. Against the backdrop of these risks, highlighted in the general debate relating to AI and biometrics and thoroughly discussed by Fuster & Peeters (2021), to interpret the directive as providing weaker protection because the technology cannot eliminate false positives is genuinely illogical. In comparison, physical fingerprints also involve a risk of false positives (see interesting examples offered by Searston (2019)) yet are nonetheless included in Article 3(13) as an example of biometric data and hence protected by Article 10. As PrevBOTʼs identification function is comparable to physical fingerprints, the protection must be the same. Consequently, it may be used only when “strictly necessary” for the purposes mentioned in Article 10 litra a to c, subject to “appropriate safeguards”.

The conditions for processing special categories of personal data in Article 10 are addressed rather cursorily here as a detailed examination is beyond the scope of the article. However, it should first be noted that strict necessity must be assessed relative to the characteristics of the problem at hand. The PrevBOT concept was born in recognition of the shortcomings of current police strategies against CSEA, and, so far, less intrusive methods promising to achieve the same goal are not available. Importantly, it was demonstrated that PrevBOT affords the police capabilities that are not available through personal faculties (vision), without which a police officer is not in position to recognize a former CSEA convict who has resumed the activity of approaching children for sexual purposes. Appropriate safeguards are of course needed, some of which are indicated in the LED, Preamble Recital 37. Third, the use of linguistic fingerprints is limited to the purposes mentioned in Article 10 litra a to c, and of these litra b seems to fit: “to protect the vital interests … of another natural person,” e.g., children at risk of online CSEA.

6.4 Categorization of age and/or gender

The purpose of PrevBOTʼs categorizations of age/gender is to detect indications of CSEA risk, not to uniquely identify a person. PrevBOTʼs intervention strategy shows this, the plan being to submit preventative messages to PPs who remain anonymous. Yet, in the age of AI and high-powered data computations, the possibility that the classification could narrow down a group of people, indirectly help ʻallowingʼ the unique identification of a person, cannot be excluded. Still, the data are not biometric within the meaning of the LED as in this regard the original purpose for which the data are processed is decisive (Fuster & Peeters, 2021, p. 24-25). Put differently, for the data to be biometric the purpose of the processing must be to uniquely identify a person. As this is not the case for PrevBOTʼs categorizations, the data are not covered by Article 10.

Article 10 also includes the processing of ʻdata concerning a natural personʼs … sexual orientationʼ. A PP is targeted because of the risk of committing a CSEA crime. Indirectly, the police have assumed something about the PPʼs sexual orientation, although this is not expressed by the data extracted by PrevBOT, which only concern age and gender, and are clearly outside the scope of Article 10. Inferences about the PPʼs sexual orientation need contextual data such as participation in a PS, seeking contact with children, misrepresentation of age/gender. These data are not processed by PrevBOT but by the police officer. It is therefore unclear whether PrevBOTʼs categorization of data falls under the scope of Article 10. If it does, the assessment should be the same as set out above.

6.5 Automated processing of personal data

As PrevBOTʼs output is produced automatically by machine learning, the applicability of LED Article 11 becomes an issue. This can be done quite briefly as the provision only prohibits decisions based “solely” on automated processing (if producing “an adverse legal effect concerning the data subject or significantly affects him or her.”) PrevBOT is not conceptualized as being fully autonomous, neither in the decision to submit a preventative message nor for opening a criminal investigation. Instead, it acts at the order of a human operator whose decision is based on the merits of the available information (see Section 3.2). By securing meaningful human involvement in the decision-making, PrevBOT does not fall under the scope of Article 11 (cf. Enarsson et al. (2021, section 3.1) in relation to the GDPR Article 22, which applies mutatis mutandis to the LED Article 11; and Sajfert and Quintel (2018)).

The analysis has shown that the LED does not preclude the use of PrevBOT, only prescribes conditions for it to be lawful. Importantly, a purpose-specific legal framework for the reference linguistic fingerprints repository and the biometric data processing must be established. The framework could be inspired by the corresponding regulation of DNA and fingerprints. Furthermore, safeguards must be established to secure that no data are retained other than those giving reason for intervention. All data processed but not acted upon, should be immediately discharged. With a suitable legal framework along these lines in place, PrevBOT may be realized within the LED framework.

7.1 The ban on real-time biometric identification systems in publicly accessible spaces

The AIA prohibits the use of “ʻreal-timeʼ biometric identification systems in publicly accessible spaces by law enforcement,” except in three “exhaustively listed and narrowly defined situations” (Preamble, Recital 19) (Article 5.1.d (i)–(iii)). Recital 18 explains the reasons for the prohibition:

[Such use] is considered particularly intrusive in the rights and freedoms of the concerned persons, to the extent that it may affect the private life of a large part of the population, evoke a feeling of constant surveillance and indirectly dissuade the exercise of the freedom of assembly and other fundamental rights. In addition, the immediacy of the impact and the limited opportunities for further checks or corrections in relation to the use of such systems operating in ʻreal-timeʼ carry heightened risks for the rights and freedoms of the persons that are concerned by law enforcement activities.

PrevBOTʼs linguistic fingerprint function is a ʻreal-time biometric identification systemʼ within the meaning of the AIA (see Article 3(33) (36) and (37)), thus encompassed by the prohibition. However, the prohibition concerns ʻpublicly accessible spacesʼ, a notion which only comprises “physical” places (Article 3(39)), leaving out online spaces. This is surprising since the reasons for the prohibition (above) arguably are relevant to online forums as well. However, the Preamble Recital 9 emphasizes that “[o]nline spaces are not covered as they are not physical spaces.” As per the current text of the AIA, PrevBOTʼs identification function thus falls outside the scope of the prohibition. Then, what remains is to consider PrevBOT in relation to the provisions of ʻhigh-riskʼ AI systems.

7.2 High-risk AI systems

Pursuant to Article 6 of the proposed AIA, the AI systems referred to in Annex III shall be considered ʻhigh-riskʼ. Any function of PrevBOT covered by Annex III must comply with a comprehensive set of conditions pertaining to ʻhigh-riskʼ AI systems in the AIA Title III. Annex III Point 1 and 6 are relevant to PrevBOT. Point 1 concerns biometric identification systems “intended to be used for the ʻreal-timeʼ … biometric identification of natural persons without their agreement.” PrevBOTʼs real-time identification function fits this description and is hence high-risk.

Biometric categorization systems per se were originally included in Point 1, later removed by the Council. For PrevBOTʼs categorization function to be ʻhigh-riskʼ, it must be covered by an alternative in Point 6 litra a to f (law enforcement), and of these, litra a, e and f may be relevant. The “intended use” is decisive, and, as one may recall, one purpose is to predict the existence of sexualized speech in a chat room, relevant for classifying a forum as a Problematic Space; another is to assign persons participating in Problematic Spaces to age and/or gender categories, which (depending on the context) may define them as PPs who may be targeted by a preventative message.

As Point 6 litra a, e, or f address use of AI by the police intended to expose persons, they seem not applicable to the categorizations of sexualized speech. Although speech is generated by persons, this is not decisive – the point being that the processing does not target any person but rather helps the operator to assess whether the online forum is a Problematic Space. The categorization of speech is therefore not high-risk.

Categorizations of age and gender on the other hand concern persons. Point 6 litra a addresses AI systems “intended to be used … for making individual risk assessment of natural persons in order to assess the risk for … offending … .” The categorization may lead the person to be regarded as ʻproblematicʼ. This means that the person has been subject to an “individual risk assessment” concerning “the risk for offending,” entailing the conclusion that this part of the categorization function is covered by litra a and is thus ʻhigh-riskʼ. This being the conclusion, there is no need to consider litra e and f.

The Council of Europe

The European Union

Norwegian legal sources

The Norwegian Police Act, 4.8.1995/ 53.

The General Attorney (2018) Circular 2/2018 of the Norwegian General Attorney in Criminal Matters concerning ʻInfiltration and provocation as methods in criminal investigationʼ.