A fake delegative system appears to transfer control to a human, but alerts the person too late or grants too little authority to prevent harm. Effective delegation requires timely awareness and genuine agency over the unfolding situation. section 9.2.2
Before treating human oversight as a safety mechanism, verify that the system:
- Alerts the human before the situation becomes unrecoverable. section 9.2.2
- Gives the human sufficient authority to choose an action that can avoid disaster. section 9.2.2
- Does not use a nominal failsafe to evade the evidence and requirements needed for trustworthy autonomy. section 9.2.2
Late intervention
A nurse who learns of life-support failure only after cardiac arrest has no meaningful opportunity to delegate or repair the error. Similarly, a control system that alerts a pilot after an aircraft enters an unrecoverable stall provides only the appearance of human control. section 9.2.2
Figure 9.3 The control flow for a delegative remote control rover [1]
Use the rover control-flow figure to inspect where delegated control and human intervention occur. section 9.2.2
