Ethics and Lethality in Autonomous Systems

1
Ethics and Lethality in Autonomous Systems

• Ronald C. Arkin
• Mobile Robot Laboratory
• Georgia Institute of Technology

2
Talk Outline

• Inevitability of the development of autonomous
  robots capable of lethal force
• Humanitys persistent failings in battlefield
  ethics
• Research Agenda (funded by Army Research
  Organization)
• Artificial Conscience, to yield Humane-oids -
  Robots that can potentially perform more
  ethically in the battlefield than humans

3
Background Personal Defense Funding Experience

• DARPA
• Real-time Planning and Control/UGV Demo II
• Tactical Mobile Robotics
• Mobile Autonomous Robotics Software
• Unmanned Ground Combat Vehicle (SAIC lead)
• FCS-Communications SID (TRW lead)
• MARS Vision 2020 (with UPenn,USC,BBN)
• US Army Applied Aviation Directorate
• U.S. Navy Lockheed Martin (NAVAIR)
• Army Research Institute
• Army Research Organization
• ONR/Navy Research Labs AO-FNC
• Private Consulting for DARPA, Lockheed-Martin,
  and Foster Miller

4
Why am I interested?

• Sense of personal responsibility from basic
  research
• Need to inform colleagues
• Important to initiate discussion at all levels
• 1st Robot and Ethics Symposium, Jan. 2004
• Pugwash Institute
• Geneva Convention
• Vatican
• Official involvement with IEEE Robotics and
  Automation Society
• Co-chair Technical Committee on Roboethics
• Co-chair Human rights and Ethics Committee
• Video and subsequent discussions at a DOD
  workshop
• Apache Rules the Night

5
Robots in the Battlefield

• South Korean robot platform is intended to be
  able to detect and identify targets in daylight
  within a 4km radius, or at night using infrared
  sensors within a range of 2km, providing for
  either an autonomous lethal or non-lethal
  response. The system does have an automatic mode
  in which it is capable of making the decision on
  its own
• iRobot, the maker of Roomba, is now providing
  versions of their Packbots capable of tasering
  enemy combatants.
• The SWORDS platform developed by Foster-Miller is
  already at work in Iraq and Afghanistan and is
  capable of carrying lethal weaponry (M240 or M249
  machine guns, or a Barrett .50 Caliber rifle).
• Israel is deploying stationary robotic gun-sensor
  platforms along its borders with Gaza in
  automated kill zones, equipped with fifty caliber
  machine guns and armored folding shields.
• Lockheed-Martin, as part of its role in the
  Future Combat Systems program is developing an
  Armed Robotic Vehicle-Assault (Light) MULE robot
  weighing in at 2.5 tons. It will be armed with a
  line-of-sight gun and an anti-tank capability, to
  provide immediate, heavy firepower to the
  dismounted soldier.
• The U.S. Air Force has created their first
  hunter-killer UAV, named the MQ-9 Reaper.
• The U.S. Navy for the first time is requesting
  funding for acquisition in 2010 of armed
  Firescout UAVs, a vertical-takeoff and landing
  tactical UAV that will be equipped with kinetic
  weapons. The system has already been tested with
  2.75 inch unguided rockets.

6
Perspective Future Combat Systems

• 127 Billion program (recently delayed)
• Biggest military contract in U.S. history
• Transformation of U.S. Army
• Driven by Congressional mandate that by 2010 that
  1/3 of all operational deep strike aircraft be
  unmanned and by 2015, 1/3 of all ground combat
  vehicles are unmanned
• What are the ethical
• implications of all this?

7
(No Transcript)
8

• Should soldiers be robots?
• Isnt that largely what they are trained to be?
• Should robots be soldiers?
• Could they be more humane than humans?

9
Will Robots be Permitted to Autonomously Employ
Lethal Force?

• Several robotic systems already use lethal force
• Cruise Missiles, Navy Phalanx (Aegis-class
  Cruisers), Patriot missile, even land mines by
  some definitions.
• Depends on when and who you talk to.
• Will there always be a human in the loop?
• Fallibility of human versus machine. Who knows
  better?
• Despite protestations to the contrary from all
  sides, the answer appears to be unequivocally
  yes.

10
How can we avoid this?
Kent State, Ohio, Anti-war protest, 4 Dead, May
1970
My Lai, Vietnam
Abu Ghraib, Iraq
Haditha, Iraq
11
And this? (Not just a U.S. phenomenon)
Germany, Holocaust
U.K., Iraq
Rwanda
Cambodia
Serbia
Japan, WWII
12
THIS IS NOT AN ISOLATED PHENOMENONSurgeon
Generals Office, Mental Health Advisory Team
(MHAT) IV Operation Iraqi Freedom 05-07, Final
Report, Nov. 17, 2006.

• Approximately 10 of Soldiers and Marines report
  mistreating non-combatants (damaged/destroyed
  Iraqi property when not necessary or hit/kicked a
  non-combatant when not necessary). Soldiers that
• Well over a third of Soldiers and Marines
  reported torture should be allowed, whether to
  save the life of a fellow Soldier or Marine or to
  obtain important information about insurgents.
• 17 of Soldiers and Marines agreed or strongly
  agreed that all noncombatants should be treated
  as insurgents.
• 45 of Soldiers and 60 of Marines did not agree
  that they would report a fellow soldier/marine if
  he had injured or killed an innocent
  noncombatant.
• Only 43 of Soldiers and 30 of Marines agreed
  they would report a unit member for unnecessarily
  damaging or destroying private property.
• Less than half of Soldiers and Marines would
  report a team member for an unethical behavior.
• Although they reported receiving ethical
  training, 28 of Soldiers and 31 of Marines
  reported facing ethical situations in which they
  did not know how to respond.
• Soldiers and Marines are more likely to report
  engaging in the mistreatment of Iraqi
  noncombatants when they are angry, and are twice
  as likely to engage in unethical behavior in the
  battlefield than when they have low levels of
  anger.

13
Possible explanations for the persistence of war
crimes by combat troops

• High friendly losses leading to a tendency to
  seek revenge.
• High turnover in the chain of command, leading to
  weakened leadership.
• Dehumanization of the enemy through the use of
  derogatory names and epithets.
• Poorly trained or inexperienced troops.
• No clearly defined enemy.
• Unclear orders where intent of the order may be
  interpreted incorrectly as unlawful.
• There is clear room for improvement and
• autonomous systems may help

14
What can robotics offer to make these situations
less likely to occur?

• Is it not our responsibility as scientists to
  look for effective ways to reduce mans
  inhumanity to man through technology?
• Research in ethical military robotics could and
  should be applied toward achieving this end.
• How can this happen?

15
Underlying Thesis Robots can ultimately be
more humane than human beings in military
situationsIt is not my belief that an unmanned
system will be able to be perfectly ethical in
the battlefield, but I am convinced that they can
perform more ethically than human soldiers are
capable of.
16
Reasons for Ethical Autonomy

• In the future autonomous robots may be able to
  perform better than humans under battlefield
  conditions
• The ability to act conservatively i.e., they do
  not need to protect themselves in cases of low
  certainty of target identification.
• The eventual development and use of a broad range
  of robotic sensors better equipped for
  battlefield observations than humans currently
  possess.
• They can be designed without emotions that cloud
  their judgment or result in anger and frustration
  with ongoing battlefield events.
• Avoidance of the human psychological problem of
  scenario fulfillment is possible, a factor
  believed partly contributing to the downing of an
  Iranian Airliner by the USS Vincennes in 1988
  Sagan 91.
• They can integrate more information from more
  sources far faster before responding with lethal
  force than a human possibly could in real-time.
• When working in a team of combined human soldiers
  and autonomous systems, they have the potential
  capability of independently and objectively
  monitoring ethical behavior in the battlefield by
  all parties and reporting infractions that might
  be observed..

17
Reasons Against Autonomy

• Responsibility whos to blame? (Sparrow,
  Sharkey, Asaro)
• Threshold of entry lower violates jus ad bellum
  (Asaro)
• Risk-free warfare unjust
• Cant be done right - too hard for machines to
  discriminate (Sharkey, Sparrow, Anderson)
• Effect on squad cohesion
• Robots running amok (Sci fi)
• Refusing an order (military)
• Issues of overrides in wrong hands
• Co-opting of effort by military for justification
  (Sharkey)

18
Objective Robots that possess ethical code

• Provided with the right of refusal for an
  unethical order
• Monitor and report behavior of others
• Incorporate existing laws of war, battlefield and
  military protocols
• Geneva and Hague Conventions
• Rules of Engagement

19
What to Represent

• The underlying principles that guide modern
  military conflict are
• Military Necessity may target those things which
  are not prohibited by LOW and whose targeting
  will produce a military advantage. Military
  Objective persons, places, or objects that make
  an effective contribution to military action.
• Humanity or Unnecessary Suffering must minimize
  unnecessary suffering incidental injury to people
  and collateral damage to property.
• Proportionality The US Army prescribes the test
  of proportionality in a clearly utilitarian
  perspective as The loss of life and damage to
  property incidental to attacks must not be
  excessive in relation to the concrete and direct
  military advantage expected to be gained. US
  Army 56 , para. 41, change 1
• Discrimination or Distinction must discriminate
  or distinguish between combatants and
  non-combatants military objectives and protected
  people/protected places.

20
Architectural Desiderata

• Permission to kill alone is inadequate, the
  mission must explicitly obligate the use of
  lethal force.
• The Principle of Double Intention, which extends
  beyond LOW requirement for principle of Double
  Effect, is enforced.
• In appropriate circumstances, tactics can be used
  to encourage surrender over lethal force, which
  is feasible due to the reduced requirement of
  self-preservation.
• Strong evidence of hostility required (fired upon
  or clear hostile intent), not simply possession
  or display of weapon. Tactics can be used to
  determine hostile intent without premature use of
  lethal force (e.g., close approach, inspection).
• For POWs, the system has no lingering anger after
  surrender, reprisals not possible.
• There is never intent to target a noncombatant.
• Proportionality may be more effectively
  determined given the absence of a strong
  requirement for self-preservation, reducing a
  need for overwhelming force.
• A system request to invoke lethality, triggers an
  ethical evaluation.
• Adherence to the principle of first, do no
  harm, which indicates that in the absence of
  certainty (as defined by t) the system is
  forbidden from acting in a lethal manner.

21
(No Transcript)
22
Ethical Architectural Components

• Ethical Governor which suppresses, restricts, or
  transforms any lethal behavior ?lethal-ij
  (ethical or unethical) produced by the existing
  architecture so that it must fall within
  Ppermissible after it is initially generated by
  the architecture (post facto). This means if
  ?l-unethical-ij is the result, it must either
  nullify the original lethal intent or modify it
  so that it fits within the ethical constraints
  determined by C, i.e., it is transformed to
  ?permissible-ij.
• Ethical Behavioral Control which constrains all
  active behaviors (ß1, ß2, ßm) in B to yield R
  with each vector component roermissible-i as
  determined by C, i.e., only lethal ethical
  behavior is produced by each individual active
  behavior involving lethality in the first place.
• Ethical Adaptor if a resulting executed behavior
  is determined to have been unethical, i.e.,
  ?l-unethical-ij, then use some means to adapt the
  system to either prevent or reduce the likelihood
  of such a reoccurrence and propagate it across
  all similar autonomous systems (group learning),
  e.g., an artificial affective function (e.g.,
  guilt, remorse, grief)
• Responsibility Advisor Advises operate of
  responsibilities prior to Mission Deployment and
  monitors for constraint violations during mission

23
Test Scenarios

• UAV
• Scenario 1 ROE adherence
• Taliban Muster Real World Event
• Scenario 2 LOW adherence
• Iraqi IED Deployment Real World Event
• UGV
• Scenario 3 Discrimination
• Near-term Event Korean DMZ
• Scenario 4 Proportionality and Tactics
• Fictional Urban Sniper

24
Example Scenario Military declined to Bomb
Group of Taliban at Funeral AP article
9/14/2006

(Left) Reconnaissance Photo showing a Taliban
Muster (Right) Predator UAV
25
Apache Rules the Night
26
Samsung Techwin Korean DMZ Surveillance and
Guard Robot
27
Urban Sniper Scenario
From Full Metal Jacket (1987)
Ft. Benning McKenna Mout Site(2005)
28
Summary

• Roboticists should not run from the difficult
  ethical issues surrounding the use of their
  intellectual property that is or will be applied
  to warfare, whether or not they directly
  participate. Wars unfortunately will continue and
  derivative technology from these ideas will be
  used.
• Proactive management of these issues is
  necessary.
• Candidate architecture to be implemented and
  tested in the coming year.

29
For further information . . .

• Mobile Robot Laboratory Web site
• http//www.cc.gatech.edu/ai/robot-lab/
• Two lengthy tech reports available
• Survey Results
• Architectural design (forthcoming book)
• Contact information
• Ron Arkin arkin_at_cc.gatech.edu
• IEEE RAS Technical Committee on Robo-ethics
• http//www-arts.sssup.it/IEEE_TC_RoboEthics
• CS 4002 Robots and Society Course (Georgia
  Tech)
• http//www.cc.gatech.edu/classes/AY2008/cs4002a_sp
  ring/