Meaningful Human Control over Autonomous Weapon Systems: An (International) Criminal Law Account

[Marta Bo is a Researcher at the Graduate Institute (LAWS and War Crimes
) and at the T.M.C. Asser Institute.]

Meaningful Human Control is at the core of regulatory and ethical debates on autonomous
weapon systems. In international discussions and writings, the problem of
meaningful human control has been addressed from different angles: from philosophical, ethical and legal (here and here), to operational, cognitive and technical (here and here), and recently from an engineering, socio-technical and
governance perspective

Despite the increasing use of parallel concepts
such as ‘human element’ or ‘human oversight’, a recent, very comprehensive, SIPRI-ICRC report, which combines ethical, legal and
operational considerations, adopts human
as a conceptual framework to shape limits to autonomy in weapons. In
addition, a general obligation to maintain meaningful human control and
positive obligations to ensure meaningful human control are the first and third
pillars of the banning treaty advocated by HRW and the Harvard Law School International Human Rights

This post provides an account of meaningful human control based on criminal law and on conditions to attribute principal criminal responsibility (rather than command responsibility). More specifically, it suggests that criminal law could provide elements not only to ground a theory of meaningful human control but also for the concrete operationalisation of this norm. A treaty obligation enshrining a duty to ensure meaningful human control should be adopted. This obligation should hinge on whether the individual had the capacity to fulfil the duty to take precautions in attacks and thus should provide for the attribution of criminal responsibility by omission (thus partly addressing responsibility gaps) limited to situations when it is fair to do so.

Why using criminal responsibility as conceptual framework for Meaningful Human control?

While concerns surrounding responsibility (and
in particular the so-called responsibility gap) have been raised by civil society
and academia since the inception of the debates on autonomous weapons, only
recently have international regulatory discussions opened up and moved from a
compliance-centric focus to encompassing issues of responsibility. Guiding Principles b) and d) adopted by the GGE on LAWS both refer
to accountability.

The use of autonomous weapon systems might
result in the commission of violations of the IHL principles of distinction and
proportionality potentially amounting to war crimes. Criminal responsibility for
unlawful attacks against protected persons (e.g. civilians) or objects is one
kind of responsibility that could arise in this scenario.

In light of possible responsibility gaps and
the cardinal notion of ‘control’ in criminal law, meaningful human control must
be shaped so that individual criminal responsibility for unlawful attacks
amounting to war crimes can be attributed. Amoroso and Tamburrini rightly speak of meaningful human
control as ‘responsibility attractor’. As I have previously argued ‘the degree of human involvement
and the corresponding level of autonomy of weapon systems are also crucial for
the determination of criminal responsibility [and must] be able to ensure that
the ‘human’ will be held criminally responsible.’

‘Control’ figures prominently in criminal law.
It is a crucial notion within the international criminal law doctrine of
command responsibility (Mettraux, 143 and 156 ff.),as well of some
forms of complicity (Ashworth and Horder, p. 427)including joint
perpetration before the ICC. Besides these forms of criminal participation or
accessory liability, ‘control’ also plays a central role in the attribution of principal responsibility. Arguably, the
applicability of command responsibility to an autonomous weapon is not self-evident
(since it has so far been postulated on the existence of responsibility of human subordinates for crimes that the
commander failed to prevent and punish). Thus, principal responsibility must be
the starting point. obvious

Criminal law blames individuals for what they intended or foresaw (mens rea) and
for what lay within their control (actus reus and causation). Key concepts
such as intent, foreseeability, voluntary act, and causality entail some
control conditions. Among those, importantly, criminal responsibility is causal
responsibility: it requires human causal control over events. A theory of
‘meaningful human control’ must ensure that conditions for the ascription of principal criminal responsibility– and
in particular foreseeability and influence over the act and the causal chain  – could be met if war crimes are committed.

It is beyond the scope of this post to develop
a theory of human control ensuring human dominance over the act and human
causal contribution. When human actions are
intermediated by autonomous processes,
foreseeability is irremediably affected, thus making the determination of mens rea and causality difficult.

Instead, I will focus on human omissions: that is, humans’ failures to stop autonomous systems.

Criminal responsibility for what and by what? Failures to stop and responsibility by omission for conduct of hostilities war crimes

Whilst intentionally
committing a war crime and causing the
death of civilians with an autonomous weapon system falls squarely within war
crimes provisions, this will not be the most likely scenario when these systems
are employed. Most probably, there will be some forms of (reckless or negligent) failures to stop these systems.

According to a recent SIPRI-ICRC report, ‘human supervision has been and is
likely to remain a necessary form of control measure to ensure safety,
reliability and efficiency of military operation (p. 21) and  seems ‘to be required in all but the most
static and predictable targeting and conflict environments’’ (p. 10).
Supervising the functioning of an autonomous system entails ‘authorizing,
vetoing or overriding the targeting functions, aborting a task or deactivating
the system’ (p. 28).

Thus, at the operational phase, the failure to
stop an autonomous targeting cycle will be one of the most common situations arising
when an autonomous weapons are deployed causing the violation of the principle
of distinction or proportionality. Responsibility will arise for having failed to stop an attack
against protected objects or individuals, such as civilians. This is a type of
responsibility by omission, which
refers to responsibility for not doing
(so-called commission by omission).

War crimes are defined in terms of affirmative
action. If we take grave breaches in Article 85 AP I, it is a war crime to wilfully
launch attacks against civilians causing death or serious injury and to
wilfully launch attacks causing disproportionate death or
serious injury are war crimes. According to Article 8 of the ICC Statute, it is
a war crime to intentionally direct attacks
against civilians or to intentionally launch
disproportionate attacks.

In the context of principal responsibility, can these war crimes be committed by omission
as well as by action? In other words, is ‘commission by omission’ of conduct of
hostilities war crimes punishable? Can the lack of intervention/failure to stop
an autonomous targeting process resulting in an attack in violation of the IHL
principles of distinction or proportionality qualify as a war crime? This
depends on the legal framework.

In most national systems, it is accepted that
failing to prevent harm is equated to causing it, if a positive legal duty to
act exists. The equivalence of act and omission is founded on the existence of
such legal duty to act.

AP I accepts this equivalence and at Article 86
requires states and belligerents to ‘take measures necessary to suppress all other breaches of the Conventions or
of this Protocol which result from a
failure to act when under a duty to do so

When it comes to the duty to prevent unlawful
attacks, we must turn to the IHL duty to take precautions in Articles 57 and 58
AP I and in customary law that requires, among other things, that those
responsible for planning, deciding upon or executing attacks do everything
feasible to verify the objective of an attack and suspend attacks if it becomes
apparent that they would violate the requirements of IHL – either because the
target is recognised as a protected person or object or because the attack is disproportionate.
Regardless of whether Articles 57 and 58 AP I and customary law impose these
obligations directly on individuals, these duties are enacted in most national
laws, military laws, manuals or rules of engagement, which make individuals the
bearers of these obligations.

Legal duties to act create so-called ‘guarantor’ positions. The concept of guarantor in the context of omission
responsibility in national criminal law refers to the ‘duty to safeguard any
protected interest against potential dangers which are under [one’s] control (position of supervisor)’ (Duttwiler, pp. 35 and 36).  Supervisory positions ‘can be subdivided into duties to control potential dangers posed by third persons or by things’ (Duttwiler, p. 36). Importantly, autonomous
weapons are to be equated to ‘things’. Thus, omission liability in the context
of principal responsibility is in principle applicable, whilst the
applicability of command responsibility to ‘things’ is arguably more uncertain.
Once a legal duty to act exists, the capacity
to perform the required act
and to discharge the obligation is a
requirement (in most national legal systems and
in the ICTY case law
) for responsibility by omission to be imposed.

Within the Rome Statute there is no general provision on commission by omission and the status of criminal responsibility by omission is more uncertain (save in instances of command responsibility). Dörmann contends that violations of the duty to take precautions do not give rise per se to criminal responsibility. Violations of this duty, and, importantly, failures to suspend unlawful attacks, amount to war crimes only to the extent that the violation evidences intent to unlawfully attack civilians or other protected persons or objects (p. 382). In the absence of intent, the impossibility of attributing responsibility for omission, i.e. for failures to suspend attacks launched by autonomous systems, would be very problematic and the criminal responsibility gap is apparent.

An obligation to ensure MHC as precondition for attributing criminal responsibility by omission and criminal responsibility as one of the drivers of the concept of MHC

Arguably, when commission by omission is
accepted, an obligation to ensure meaningful human control is functional to a
fair ascription of criminal responsibility. Human control is a precondition for
the duty to take precautions insofar as it allows obligations stemming from
this duty to be discharged in a comprehensive and informed manner.

According to Sassoli and Quintin (pp. 32 and 33): ‘What counts is
not the authority under rules, regulations and instructions, but only who
possesses the practical possibility to take precautions and in particular who
has the necessary knowledge of risks’. A treaty obligation to ensure meaningful human control across the
difference stages of the targeting process is key for the ‘practical
possibility’ of discharging the duty to take precautions. However, such an
obligation should also be implemented in national military laws and rules of
engagement. The obligation to ensure meaningful human control over an
autonomous system should be articulated in specific legal duties to act and be
imposed upon individuals. All bearers of these duties should be identified
taking into account the distributed nature of military decision-making
processes, the specific features of the weapon system in use, as well as the
stage of its use.

The purpose of the obligation would be twofold:
to create duties to act and to place the individual bearers of these duties in the
condition (i.e. capacity) to exercise
them and suspend the launching of unlawful attacks. Legal obligations on
individuals stemming from a general treaty obligation to ensure meaningful
human control together with the IHL duty to take precautions create so-called ‘guarantor’ or supervisory positions.

Criminal responsibility of individuals in a
position to exercise human control over the autonomous weapons system would
arise for having neglected positive duties to act, as a result of which attacks
in violation of the IHL principles of distinction or proportionality are not
prevented and launched. In case these unlawful attacks occur, criminal
responsibility would arise from failures to suspend attacks launched by
autonomous weapons based on the violation of the duty to exercise control over
the weapons system and the duty to exercise precautions. The framework of
omission responsibility offers surrogates for the arduous assessment of
causality and mens rea in cases of
responsibility by commission (Fletcher, Rethinking Criminal Law, OUP 2000, p.
606).  In the context of human omissions, the assessment of mens rea and causality becomes less
cumbersome than in cases of human actions
intermediated by autonomous processes and thus allows a partial closing of
the responsibility gap.

In sum, obligations to exercise meaningful
human control across the different stages of the targeting process are
necessary for the duty to take precautions in attacks to be discharged.

This allows for a fair attribution of criminal responsibility: only in cases where the individual/human was under a clear duty to act (e.g. control and suspend) and only when he/she was capable to do so (i.e. he/she had the ability to affect the autonomous process).


In this post, I offered an account of the
relationship between meaningful human control and criminal law whereby they can
be mutually beneficial. A treaty obligation on meaningful human control must be
adopted and articulated in terms of legal
duties to act
on individuals and must ensure that human operators are capable to exercise the duty to take
precautions. Under these conditions, criminal responsibility for failing to
stop unlawful attacks launched by autonomous weapons is fairly attributed.

The creation of supervisory positions and responsibility for omission are policy
choices, which seem appropriate in light of the present and future relevance of
supervised autonomy. This however
highlights a wider potential responsibility gap when it comes to the punishment
of conduct of hostilities war crimes committed by omission within the framework
of the Rome Statute.

Read More

Marta Bo