Safety Is Retrospective: Why AI Governance Should Regulate Risk, Not Trust

AI governance should be rebuilt around calibrated, independent measurement infrastructure. This paper maps six architectural failures from aviation (Air France Flight 447), oil and gas (Piper Alpha), nuclear oversight (NRC Reactor Oversight Process), and post-market drug safety onto current AI governance and proposes measurement-based alternatives. The six failures are: common-cause failure in redundant monitoring; silent degradation of operating assumptions; perverse reversal where correct actions trigger alarms; diagnostic information trapped inside systems; cumulative drift through individually tolerable steps; and known risk deferred without mandatory action. For each, the paper identifies the structural analogue in AI governance and proposes a measurement-based architectural response: validity state protocols, regulatory join keys, sequential conformal monitoring, and community intelligence.