Military Standard on Software Control Levels

The mil-std-882e standard specifies levels of software control, i.e. how
dangerous the software can be based on what it is responsible for. Although the
standard is a little more complicated, we can simplify to essentially four
levels:

1. The most alarming case is when the software has direct control of something
   that can be immediately dangerous if the software does the wrong thing.
2. Still dangerous, but slightly less so is either (a) when the software has
   direct control, but there is a delay between when it does the wrong thing and
   when it becomes dangerous; or (b) when the software is not directly in
   control, but a human must immediately react to software signals and perform
   an action to prevent danger.¹ E.g. the software commands a reactor
   shutdown when there are only seconds remaining until the reactor blows up.
3. Yet less dangerous is when the software is not in direct control, and there
   is time to verify its suggestion against independent methods to make sure the
   action recommended by the software is indeed appropriate.
4. The least dangerous is when software only has an auxiliary use and is not
   involved in controlling something serious.

I thought this was a neat way to look at things, and particularly salient now
that llms and computer vision have blown open new opportunities for injecting
software into processes in which software were previously subservient to humans.