The Defense Innovation Board recently published a white paper with its list of five ethical principles for the use of Artificial Intelligence in DoD. These principles aren’t meant to define ethics, but to describe a framework for development of AI systems that conform to existing and widely accepted ethical principles of warfare, including the Law of War, Geneva Conventions, international law, and other treaties. The guidelines, listed below, have some significant shortcomings, and seem to lack some necessary context. Here are the guidelines as enumerated in the White Paper:
Responsible. Human beings should exercise appropriate levels of judgment and remain responsible for the development, deployment, use, and outcomes of DoD AI systems.
Equitable. DoD should take deliberate steps to avoid unintended bias in the development and deployment of combat or non-combat AI systems that would inadvertently cause harm to persons.
Traceable. DoD’s AI engineering discipline should be sufficiently advanced such that technical experts possess an appropriate understanding of the technology, development processes, and operational methods of its AI systems, including transparent and auditable methodologies, data sources, and design procedure and documentation.
Reliable. DoD AI systems should have an explicit, well-defined domain of use, and the safety, security, and robustness of such systems should be tested and assured across their entire life cycle within that domain of use.
Governable. DoD AI systems should be designed and engineered to fulfill their intended function while possessing the ability to detect and avoid unintended harm or disruption, and for human or automated disengagement or deactivation of deployed systems that demonstrate unintended escalatory or other behavior.
Some initial thoughts and concerns:
1. Responsible. Who is responsible for each aspect (development, deployment, use, and outcomes)? Over what time frame? To whom are they responsible? Through which process? Is the developer who wrote a section of code for a weapons pairing algorithm (with little or no experience with the weapons being employed, and living in a different country from the owner of the system) to be held accountable for the outcome (collateral) damage of the use of that software by a warfighter in a different country? Hopefully that isn’t the intent of the authors… perhaps they really mean that AI will have to be developed and tested by humans and not by automated coding systems (evolutionary algorithms and related technologies). It is difficult to understand the intent here. And without a better understanding of the intent, it will be even harder to implement.
2. Equitable. This is also a very difficult issue. We rightly have a strong reaction to biases in AI that discriminate against individuals by ascribing attributes of a group to a single member of the group (stereotyping). On the other hand, AI algorithms are fundamentally (but not exclusively) designed to detect patterns in large data sets. And they don’t do this in human ways; they do it in machine ways. So AI makes machine types of errors rather than human types of errors, which can have the effect of causing humans not to trust the AI, even if the AI makes fewer errors than the humans would. Care should be taken to understand and explain this to the public, so we don’t discard solutions that have the potential to save many lives simply because the mistakes are different, but less consequential (fewer, less severe, or both), than a human operator would make.
3. Traceable. Again, there is ambiguity here. While transparency and auditability of design and development are certainly possible and probable necessary, applying the same standard to the actual operation of AI systems is an area of ongoing research (AI Explainability), and one where there may be fundamental computability issues that prevent a solution. Even if AI Explainability ends up being technically possible, humans may not be able to understand the explanation (just as AI and humans make different classes of mistakes, they arrive at decisions in very different ways).
4. Reliable. There is a ‘predicting the future’ class of problem here that is as old as the history of warfare. We have never accurately predicted the nature of the next conflict, and as a result, we always go to war with tools that are reasonably well suited to a previous conflict, but frequently poorly suited to the new one. Requiring an ‘explicit, well-defined domain of use’ as an AI design requirement may be a bridge too far. Especially if one potential use of AI in warfare (and other domains) is to solve problems that humans haven’t seen before, that humans haven’t solved before, or that humans are too slow to solve.
5. Governable. This one may have the fewest potential issues, as the authors have at least partially avoided the AI safety problem of humans being too slow to stop a sufficiently powerful computer in time to avoid harm. Difficulty may arise when humans and machines differ in the way they define terms like ‘unintended escalatory behavior.' Again, machines and humans think differently, and machines may be more willing than humans to make certain trade-offs in conflict, for example accepting short-term or tactical casualties to achieve a long-term or strategic objective more effectively.
The topics of Ethics in AI are difficult and not well explored. There are very few people with sufficient expertise in both areas to understand the difficulties inherent in the topics and to help formulate the right questions (and it appears, no ethicists on the DIB). This is complicated by the fact that humans haven’t ever agreed on a definition of ethics (What is right? How do we decide? How do we develop the character to act in accordance with what is right?), we humans are terrible at explaining why we made a decision, and we are notoriously bad at behaving in consistent and repeatably ethical ways. The DIB did a pretty good job of avoiding setting explicit standards of perfection in the white paper, but also left unaddressed the question of how much better should our AI be than the average human before we decide to certify it? While 100% is never achievable, marginally better than the average person may be too low a bar to set.
Update: This week, DoD announced their adoption of those guidelines, with some modifications. DoD appears to have broadened the definition of unintended bias to include not just bias that causes harm, but all unintended bias (probably not an improvement). In the Traceability standard, DoD changed ‘technical experts’ to ‘relevant personnel’, which I think is an improvement, as technical experts could be interpreted too narrowly, whereas ‘relevant personnel’ includes non-technical users (such as warfighting tacticians and strategists). They also made a minor change to the Reliability standard, defining it around the uses of AI rather than the domains in which they are used, an improvement over the DIB draft, as DoD can control how AI is used, but the domain may be influenced by adversaries.