Outrage is cyclical and often misinformed. When Israel used ‘Lavender’ and ‘Daddy’ to select targets, there was a brief period when it became the centre of attention, and there was a fair amount of outcry. The development of these technologies continued, regulatory challenges remained unresolved, and, by and large, the world moved on. Then, earlier this year, during Operation Epic Fury, the United States struck a school in Minab, Iran. As many as 180 people were killed. Yet despite these instances, conversations about the use of AI for targeting often get stuck at a surface level, and do not unpack some of the more complicated challenges involved, such as automation bias, and what human control really means.
Initial conversations around the strikes on Iran largely questioned whether Claude could be trusted to make life-and-death decisions. Anthropic does not have a contract with the United States government pertaining to the use of its technologies for target identification. Somewhere along the way, people had conflated AI as a whole with large language models and concluded that one such model was responsible for this grave error.
In reality, Project Maven was used to coordinate airstrikes during the war. Project Maven began as an object-detection tool in 2016, developed to address the sheer volume of data intelligence analysts were unable to sift through. In 2017, the US Department of Defence partnered with Google to use its technology to build targeting systems. There was a flurry of protests in Silicon Valley as engineers questioned the ethics of the project, and Google succumbed to internal pressure and did not renew its Maven project contract.
After Google’s departure, defence-tech company Palantir Technologies came on board to develop and manage the Maven Smart System. Today, Project Maven has been codified as a ‘program of record’ or a permanent program. This means it will have guaranteed long-term funding and be allocated funds in the defence budget. In its current avatar, amongst other use cases, the system ingests and synthesises data streams from sources such as satellite imagery, videos, communications link intercepts, and radar. It then uses AI to sift through the data and flag potential targets, and generate strike options.
Military AI is often used as a catch-all term for a wide range of defence applications. Many of these applications are relatively benign, ranging from predictive maintenance to data analysis. Where it becomes complex and controversial is when artificial intelligence is used to identify, select and engage targets with lethal force. This is the case with lethal autonomous weapons, and with decision support systems like Maven or Lavender.
One of the most complex challenges that comes hand in hand with decision support systems is automation bias. Automation bias is a psychological phenomenon in which humans tend to favour decisions made by automated systems and machines.
There is an idea in psychology that human brains are ‘cognitive misers’. Thinking and decision-making expend energy, and when there is an option to take a cognitive shortcut, we are inclined to do so in an attempt to choose the easier option. This is compounded by the perception that machines are objective and data-driven, as they are exempt from human error stemming from emotion or fatigue. This common belief only reinforces the tendency to favour machine-generated outputs or decisions.
This has already been observed in autonomous and decision-support systems. An article by +972 magazine, explored how, during the early stages of the genocide, officers indiscriminately used Lavender’s recommendations, seemingly without any mandated procedure that checked the data on which the recommendations were based.
Conversations about autonomous weapons and decision-support systems tend to focus on two things: the idea of ‘meaningful human control’ and whether a human is in the loop. Systems are categorised into three buckets. Human in the loop (HITL), human on the loop (HOTL) and human out of the loop (HOOTL). They range from having a person authorise every decision to overseeing every decision to a system being completely autonomous without human oversight. Most decision support systems claim to be HITL or HOTL. However, the issue is that automation bias often renders human oversight notional, since people are inclined to trust machine-made calls. Discussions about the governance of these systems need to move beyond whether there is a human in the loop, to have clearer requirements for what that means. Global governance needs to go beyond merely whether a human is present, to look at specific use-cases and understand what kind of oversight is required to prevent civilian casualties through machine-made errors.