So Man Created Robot in His Own Image: The Anthropomorphism of Autonomous Weapon Systems and the Law of Armed Conflict¹

2.1 Unpredictable but not self-aware systems

Definitions are crucial for legal regulation. It is impossible to ban or restrain an unknown entity. Furthermore, how a leading State or the organised international community choose to define autonomous weapon systems will determine whether these weapon systems are seen as existing systems or solely as weapons of the future.⁶ Yet, a commonly agreed definition appears to be truly difficult to achieve.

In the past, treaty prohibitions governing the means of warfare have emerged after the international community witnessed a specific use, such as the use of gas during the First World War and the vast civilian damage caused by cluster munitions and anti-personnel mines.⁷ Some prohibitions have been introduced due to the potential devastating use of particular weapons (for example, explosive projectiles under a certain weight and blinding laser weapons),⁸ either with regard to the indiscriminate effect on civilians or the potential suffering inflicted on combatants. AWS have not (yet) proved devastating in use and opinions differ as to whether they are even in use. Partly because of this, the definitional challenge becomes even more strenuous.

Above, in the introduction, I have already presented a preliminary definition of an autonomous weapon as one that operates on its own in a qualified way, which implies both that the system is technologically advanced enough to sense its surroundings and act accordingly, and that it is actually left alone.

How one defines a ‘fully autonomous weapon system’ beyond this preliminary approach depends on whom you ask. There are at least three approaches to autonomy. According to the first, the concept is used in a rather wide sense, referring to the absence in intensity of human control in time and space – namely, that the system is left alone. This category encapsulates existing automatic weapons systems,⁹ from the simplest forms of pre-programmed mines with the use of trip-wires or signatures, to the most advanced automatic air-defence systems.¹⁰ Some also divide this broad category into two, arguing that the way a weapon is used – that it is actually left alone – is a separate category and that the true wide approach embraces systems that are able to select and engage targets without human intervention.¹¹ In this article I view these as one broad category. If one uses a broad definition, this entails that the same weapon system may be both autonomous and automatic (or semi-autonomous). In cases where the system is operating under a certain degree of human control, it will be automatic, and where it is operating under less (or hardly any) human-imposed limitations in time and space, it will not. As opposed to others, I do not link the definition of autonomy to whether the system, when used in an autonomous mode, is used in accordance with LOAC. Very often, the degree of human control will determine whether the system is used lawfully. This does not, however, necessarily influence the definitional question.¹² Having said this, some do link the definition to the fact whether the system can be used in accordance with the law – this the third category, dealt with below.

A second (more narrow) category refers to the inherent technical features of the weapon system – and reserving the term autonomy for systems that are technically advanced enough to understand a mission, that is to search for, identify and attack targets without any human operator intervening¹³ – or simply that they are able (through technology) to change their behaviour according to changes in circumstances, i.e. that they are adaptive.¹⁴ In the latter cases, some refer explicitly to artificial intelligence, others not. The core of the first part of the second category, however, is that none of the existing systems are (usually considered to be) capable of understanding the commander’s intent, and thus capable of being fully autonomous.¹⁵ A system capable of knowing its commander’s intent would know its commander very well and, more often than not, act according to his intentions. This approach leaves us with the impression that the term ‘autonomy’ is reserved for machines that operate in a way beyond human understanding. When finally understood, they are called ‘automatic’. If understood in this way, the group of autonomous systems will change in accordance with technological development.

A third category links the definition to the ability of the system to abide by international law. The following ‘working definition’ was proposed by Switzerland during the April 2016 Expert meeting on CCW: ‘weapons systems that are capable of carrying out tasks governed by IHL in partial or full replacement of a human in the use of force, notably in the targeting cycle’.¹⁶ Some commentators have also presented a ‘functional approach’ to AWS, leaving aside the challenge of definitions and focusing on the different challenges AWS represent for the law when looking at the different elements of the system: its munitions and platforms.¹⁷ This approach resembles the approach that would, in any case, necessarily have to be taken while applying the law as it is to new technology.

While the first definition suffers from challenges of being over-inclusive, the latter (third) definition suffers from being circular. The second definition understood in the UK view may be under-inclusive when referring to a ‘human level’ of understanding a mission, appearing to reserve the term ‘autonomy’ in practice for human-like robots only. In this paper, I understand autonomy according to the second view above (but not identical to the UK view), as a system’s technical ability to adapt to changes in circumstances and, due to this, its ability to search for, identify and attack targets without human intervention. This means that I exclude from the category of autonomous weapons, for example, loitering missiles which are pre-programmed to search for certain targets according to a pre-defined algorithm, and to attack these without human intervention. These weapons are, when operating without errors, predictable. Autonomous systems are, by their very nature, unpredictable. The scope of unpredictability is, however, limited because AWS do not possess self-awareness. Even though the scope of unpredictability is limited, the deeply rooted scepticism towards autonomous weapon systems appears to lie precisely in the fear of the unpredictable features of these systems. I argue here that the anthropomorphism of autonomy contributes to the fear of the systems and to our expectations as to their behaviour. Hence, because we think their behaviour is autonomous in the human meaning of the word, we fear the machines and expect similar behaviour from them as from a human being. I shall pursue the core of anthropomorphism in the following.

2.2 Autonomy and machines: putting a dress on a cow?

‘I am not a gun’ says the cartoon robot the ‘Iron Giant’, in the movie by Brad Wird from 1999, when the Giant discovers that the boy is his friend.¹⁸ The Iron Giant is in possession of one of the most human-specific characteristics: the ability of perception and self-awareness. That is, the ability to model a picture of the real-life world based on its own senses and, thereafter, to place itself within that picture.¹⁹ The ability to perceive and be self-aware is among the core abilities expected from military commanders: to be able to adjust the use of force to circumstances ruling at the time. Autonomy is thus an intrinsically human term.²⁰ An autonomous person has the freedom and ability to govern itself and to act according to its perceptions of the world around. Using the term with machines entails a humanisation of the machine – an autonomous machine constitutes an anthropomorphism.²¹ By necessity, this task becomes partly strained, which in turn may contribute to the difficulties of agreeing upon a definition. Furthermore, the anthropomorphism may create an illusion that an ‘autonomous’ machine will behave just as a human being going rogue, and that the machine may decide its own enemies and create its own goals and values.²² I do not, however, believe that a machine going rogue in the same manner as a human being is the real issue of concern. This is why the heading of this sub-section asks whether we are ‘putting a dress on a cow’ when using the language of autonomy on machines. Instead of ‘dressing up the cow’ and expecting human-like behaviour, the focus could more fruitfully be on those characteristics of the weapon system suited to describe the system as ‘autonomous’ and, subsequently to deal with these characteristics according to the law.²³

The linguistic aspects of autonomy also invite us to address a deeper issue already launched at the outset of this article – do we deal with machines or men before the law? Are weapons (and systems) becoming humans – or are they simply a means of warfare? Are the weapons becoming their own users?²⁴ As already noted in the introduction, I argue that machines remain a means of warfare, regardless of whether they resemble humans in certain respects.²⁵ A number of other aspects of whether machines, in one way or another, become their own users and thus remove human presence are, in my view, primarily but not exclusively of ethical concern. The question ‘who are they?’ when speaking of autonomous weapon systems also relates to issues of accountability – namely, the question of where to place responsibility. Beyond what is discussed in sections 3 and 4 below, I do not address issues of accountability in this article. I do, however, address those legal assessments that presumably require a human touch. Let us, therefore, turn to the main question: what are the legal requirements that autonomous weapon systems can or cannot comply with?

4.1 Artificial intelligence: the rise or fall of the reasonable commander?

As introduced above, military commanders must assess the existence and weight of the ‘military advantage’ of an attack, as well as whether the attack must be expected to cause ‘excessive’ collateral damage compared to the concrete and direct military advantage anticipated. I shall mainly use the terms ‘advantage’ and ‘excessiveness’ in the following analysis. Both terms are not only highly discretionary, as pointed out above, but also relative: ‘advantage of what?’, ‘to achieve what?’,³⁷ ‘excessive compared to what?’ and so on. Furthermore, a military advantage is a highly subjective concept: advantage for whom?³⁸ Consequently, those military commanders assessing the anticipated military advantage must know the aim of the military operation they are taking part in, and they ought to know the relative strength of their opposing forces. By way of illustration, let us assume that the operational objective of a phase of an operation is to establish air superiority in the area of operations. The military advantage relates to this objective.³⁹ In order to create air superiority it may prove necessary to attack enemy air defence systems, command and control systems, as well as airfields. The relative importance of these targets is subject to the assessment of the reasonable commander. On the other hand, the assessment of the probability of tactical success in attacking each one of these targets by the use of a specific kind of ammunition, may be better performed by machines. Just as a machine can calculate the next move in the game of chess, predictions of enemy behaviour in the air (‘dogfights’) may be better performed by machines. The first example (the relative importance of the targets) belongs to reasoning, the latter example is a question of artificial intelligence. Weapons may be smart and even intelligent, but they do not yet possess reason in the ‘human’ definition of the concept.

At a certain stage, the law of armed conflict becomes a question of whether we can accept that human reason shall yield to intelligent weapons. Or rather, smart weapons, as they obviously lack the social/human dimension of intelligence. Does international law accept that the reasonable commander is being replaced by artificial intelligence?

We may therefore ask, as in the heading of this subsection, what does the technological development represent – a rise or a fall of the reasonable commander? Does he or she have to be even more reasonable than before to administer the advanced level of technology, or will he or she simply be out-conquered by algorithms? In the following I shall argue that commanders i) will have an even more onerous burden of reasonableness than before and ii) cannot be replaced in their performance of assessment by machines. Eventually, this is also why machines ought not to be compared to humans in this regard.

4.2 The expansive duty to take feasible precautions in attack: a matter of the commander’s choice

The reason for my position, as stated above, lies at least in part in the expansive duty to take feasible precautions in attack, to spare the civilian population, civilians and civilian objects.⁴⁰ First, I will briefly address the ‘commander’, and then move on to the precautionary duties. The heading of this sub-section refers to the ‘commander’s’ choice, but who is he or she? Upon whom does the duty lie? The wording of treaty law refers to ‘those who plan or decide upon an attack’, which in principle addresses all levels in the military chain of command.⁴¹ Some assessments, though, are in certain cases presumed to be carried out on a relatively high level of command, such as the assessment of the military advantage anticipated from an attack as a whole.⁴² As the example above shows, the attack as a whole may encompass the coordinated effort to achieve air superiority. In these cases, the military advantage is presumed to be assessed on the operational level of command and reassessed on tactical levels of command if circumstances change.⁴³

What about the precautionary duties of the commander? Commanders involved in both planning and/or conduct of an attack shall take all feasible precautions to:

I shall address each of these two interrelated duties in some more detail. The duty to ‘verify’ that the target is a military objective invites the obvious questions: what does it mean to ‘verify’, which measures are ‘feasible’ in order to achieve verification, and how much of the verification process can be left to a machine – to an autonomous weapon? In simple terms, ‘verification’ means ‘to prove’ that the target is a lawful target, but the rule must not be confused with a duty to be certain.⁴⁵ Nor is it a general requirement of ‘eyes on target’, as some doubt will always exist. Using the qualifying term ‘feasible’, the precautionary rule addresses the measures required in order to be as certain as possible under the circumstances.⁴⁶ Obviously, one cannot verify having found a military objective unless one knows what the objective looks like. Verification therefore presumes recognition of some sort.⁴⁷

The duty to suspend or cancel attacks is closely linked to the duty to verify that the target is a military objective – with the additional requirement of observing the proportionality rule, namely that an attack shall be cancelled or suspended if it becomes apparent that the target is not a military objective, or that it may be expected to cause excessive collateral damage. The law, therefore, requires some sort of estimate of collateral damage. The duty encompasses an assessment of what collateral damage is reasonably foreseeable.⁴⁸ This is a duty that obviously requires situational awareness through human perception. Furthermore, if one attack goes wrong, corrections may be expected according to the assessment of the damage (the so-called battle damage assessment – BDA).

The process of recognition can be feasibly done in accordance with the law by an automated or an autonomous weapon, if the parameters to match are inputted into the weapon in advance – to hit an object with a known signature. Problematic situations may arise if the parameters are very few, given that errors in recognition may occur, or that the weapon operates on its own over a considerable time, or travels over long distances, or a combination of these.⁴⁹ These challenges are familiar to all highly automated weapons, such as with the use of unmanned aerial vehicles (UAVs or ‘drones’), combat systems on frigates (such as AEGIS, used by the Norwegian Armed Forces), as well as sea mines. Autonomous weapons, understood in the narrow sense as referred to above, pose additional challenges. If the autonomous weapon system has learned to recognise objects that have not previously been programmed into it, from its own observations – i.e. it is trying to achieve a level of perception – it may attack objects which have not been chosen by humans, but which ‘fit’ what the weapon system has learned about other objects.⁵⁰ For example, the system knows that tanks have tank turrets and tracks, and the autonomous system is able to use this information even if observing the vehicle from another angle than in the picture loaded into it. The autonomous system can add and process data beyond what is predicted by the human operating it. Such a system may prove demanding to operate by military commanders, not knowing exactly what information (data) the system will choose to collect and rely upon. Having said this, the use of autonomous processes in weapon systems may not necessarily produce more targeting errors during attacks. On the contrary, high-level technology may provide commanders with more comprehensive and up to date information about the target and the target area, creating a higher level of situational awareness. Furthermore, a number of errors are also likely to happen with humans in or on the loop, controlling the information loaded into the weapon and deciding for the system what the threshold is for verifying that an object is a military objective (a lawful target). The pending issue appears, rather, to be which errors are expected vs. which errors are accepted. An error in verification due to incorrect data may be acceptable as accidental – provided that the error appeared reasonable according to the circumstances. For example, civilian use of – or presence within – a compound expected to be a military training camp, and only this, may not have been reasonable to expect.⁵¹ If, on the other hand, the error of attacking an object that proved to be civilian was done by an autonomous system, was that error reasonable ?

Two points shall be made in this regard: first, that the legal environment does not appear to be equally forgiving for mistakes made by machines as by humans, simply because it is counter-intuitive to label machines as reasonable. On the contrary, they are, by nature, unpredictable and the potential for systemic errors appears less consistent with the law than tragic accidents. Second, the human will never be completely ‘out of the loop’. This formula rests on a relative premise, namely that the human is far away (in time or space) from the attack decision. Nevertheless, recalling that the weapons system is a means of warfare and not an actor in warfare, a human being will always be accountable. After all, it will be a human decision to use the system against the category of target, within the area of operations, with ammunition, and so on. No matter how much autonomy a weapons system may eventually possess, the ultimate control of the weapon will expectedly rest with the reasonable commander. He or she will have to rise to the expectations of advanced weaponry, in order to comply with LOAC. He or she will have to know when to reassess the legality of the attack and thus to adequately interact and, when necessary, intervene on the autonomous system in time or space. As expectations of accuracy and precision increase, a parallel expectation of situational awareness arguably arises. For example, the law may arguably require a high degree of communication between the missile launch area and the forward air controller in the target area. It should be pointed out, however, that a high degree of communication does not necessarily mean very intense or extensive communication; it may imply sharing time sensitive information at the right time – for example, on changes in the circumstances in the target area.