What happens on the day you switch the AI off?
Europe's bank supervisors now ask whether AI can be switched off. The harder question is who can still do the work the morning after.

On 13 October, Reuters reported that the European Central Bank had given lenders “an end-October deadline to produce plans to counter AI-enabled cyber threats, including improved back-up and recovery”. Citing a person familiar with supervisors’ thinking, Reuters added that euro zone banks are also being asked to “map how AI is being used across their businesses and third-party providers” and to “ensure they can switch off and restart systems in a controlled manner”.
Switching off is the easy part. The morning after is the problem.
The UK’s air traffic service, NATS, found that out on Monday 28 August 2023, a bank holiday with high demand for flights. Its system received a flight plan it could not process. In less than 20 seconds, the main system and its backup both did what they were designed to do. They stopped, rather than send “potentially unsafe” information to an air traffic controller.
From then on, flight plans had to be entered by hand. Manual input “can only achieve a significantly reduced rate compared to automatic processing”, which at peak handles more than 800 flight plans an hour. So NATS limited traffic to what its people could type. The manual route ran through one terminal “feeding a single printer”, and data entry was slowed by “the manual translation required between formats”.
The engineer with specialist knowledge of the system was on call, as rostered on a bank holiday. When a full restart was needed, he drove in: “His journey to the office took longer than normally expected due to traffic congestion.” Automatic processing restarted at 14:27. The UK Civil Aviation Authority estimates that “over 700,000 passengers were impacted”.
- 08:32One flight plan stops the main and backup systems. Manual input begins.
- 09:06The on-call specialist is called and helps remotely.
- 10:12A full restart is needed: he has to come in.
- 11:00The first traffic limits take effect.
- 11:47He arrives, after a 1 h 35 min drive.
- 12:32Four hours of stored flight data run out.
- 14:27Automatic processing restarts.
- 18:03The last traffic limits are lifted.
Nobody was reckless. NATS found that “existing procedures were followed”. It also explained why there was no bigger manual team: keeping enough trained staff on standby to match the machine is “impractical and uneconomic”.
That is the real question behind a switch-off plan, for AI as much as any automation: on the day you switch the AI off, who can still do the work?
What the rules test, and what they miss
The supervisors’ request sits on top of law. The EU’s Digital Operational Resilience Act (DORA), which has applied since 17 January 2025, requires financial firms to test their continuity and recovery plans “at least yearly”. For all but the smallest, the tests must include “switchovers between the primary ICT infrastructure and the redundant capacity, backups and redundant facilities”. Those are switches between machines. A test that only moves work from one machine to another tells you nothing about the people who would take it over.
TSB Bank plc, a UK retail bank, shows what happens when the way back is gone. In April 2018 it moved its customers to a new platform that immediately ran into technical failures. The Financial Conduct Authority (FCA) and the Prudential Regulation Authority (PRA) later fined it £48.65 million in total. The PRA’s final notice is blunt: “A ‘roll-back’ plan was not possible for TSB in these circumstances.”
Takeaway: when you test a switch-off, test the people who would carry the work, not only the backup systems.
The irony of automation
In 1983, Lisanne Bainbridge, a psychologist at University College London, published a five-page paper called Ironies of Automation. It is still the clearest account I know of this problem.
Her first irony: the designer who tries to remove the operator “still leaves the operator to do the tasks which the designer cannot think how to automate”. The person keeps the hardest cases.
Her second: those cases arrive when the person is least ready. “Unfortunately, physical skills deteriorate when they are not used,” she wrote, so “a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one.” Knowledge fades the same way, because it “develops only through use and feedback about its effectiveness”.
She also saw the generational problem. Automated systems monitored by former manual operators “are riding on their skills, which later generations of operators cannot be expected to have.”
One line of hers is uncomfortable for anyone who builds automation, me included: “Perhaps the final irony is that it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training.”
Takeaway: the better your automation works, the more deliberately you must keep its people in practice.
When the machine hands back control
On 1 June 2009, Air France flight 447 was over the Atlantic when its airspeed readings became inconsistent, likely because ice crystals blocked the probes. The autopilot disconnected. France’s civil aviation accident investigator, the Bureau d’Enquêtes et d’Analyses (BEA), records that the pilot flying said “I have the controls”.
The failure “completely surprised the pilots”. Confusion and breaking crew cooperation then fed on each other “until the total loss of cognitive control of the situation”. The aircraft stalled and hit the sea just over four minutes after the autopilot let go. All 228 people on board died. Among the factors, the BEA lists: “The absence of any training, at high altitude, in manual aeroplane handling”.
In 2013, the US Federal Aviation Administration (FAA) warned airlines that “continuous use of autoflight systems could lead to degradation of the pilot’s ability to quickly recover the aircraft from an undesired state.” A joint FAA and industry working group found that pilots “sometimes rely too much on automated systems and may be reluctant to intervene”.
AI agents are not aircraft, but the pattern carries over. My working principle is that AI proposes and a person decides. That only holds if the person can still decide without the AI, on a bad day. And research on automation bias warns that it “cannot be prevented by training or instructions” (Parasuraman and Manzey, 2010).
Takeaway: a person who only approves the machine’s work is not yet a fallback; a person who regularly does it is.
The people who disappear with the layers
In my article on Conway’s law, my hypothesis was that organisations consolidate the work first, intending later to reduce the capacity needed to do it. If that hypothesis holds, the capacity that disappears is exactly the fallback: the people who knew the work by hand.
NATS shows what a fallback sized for normal days looks like. Its specialist engineers “gain and retain that expertise by working on those systems during normal office hours”; outside those hours, they are on call.
To be fair to NATS, its own analysis says the drive was not decisive: even without it, a fix through the supplier was unlikely before 11:23, too late to change the decision to limit traffic. A quieter finding matters more. There was “no single controlled document accessible to those that need it” with the supplier’s support details.
Bainbridge described the job of the people left behind to watch. It is “very boring but very responsible”, with no chance to build or keep the skill the responsibility needs.
Takeaway: before you consolidate a team, find out who would do its work on the night the system goes off.
Designing for people, not just systems
As an experienced architect and an entrepreneur, I treat the human in the loop as a component of the design. Like any component, it degrades without maintenance.
Practise on normal days. Bainbridge suggested hands-on control “for a short period in each shift”. The FAA asks airlines to give pilots “appropriate opportunities” to fly by hand. For AI, that means people regularly doing a share of the work, not only approving it. DORA already makes resilience training “compulsory modules” in staff training; make the yearly test include the hand-over.
Design degraded modes a person can run. Here in the Netherlands, Ahold Delhaize announced on 12 October that it was taking “some systems offline” after a cybersecurity incident, not an AI failure. Albert Heijn’s tills still applied national offers, while in some distribution centres work stopped completely, the company told the Dutch broadcaster NOS. As of 14 October, it was starting “a controlled re-launch of certain systems”.
Limit automatic retries. On India’s Unified Payments Interface (UPI), according to the Indo-Asian News Service (IANS), “The relentless transaction status checks by banks resulted in the massive UPI outage on April 12, leaving millions stranded.” The Reserve Bank of India’s guidance tells banks to “assume that disruptions will occur”.
Checklist
Switch-off drill: five questions about people
- Who takes over when it goes off, on every shift?
- When did they last do this work by hand?
- How much can they handle in an hour, and is that enough?
- Can the night shift find the runbook and the supplier's number?
- Who decides to switch back on, and with what information?
Takeaway: rehearse the switch-off like a fire drill, on a normal day, with the people who would run it.
What it costs, and who pays if you don’t
None of this is free. NATS is right that a standby team matching the machine’s speed would be “impractical and uneconomic”. A full manual twin of every AI-supported process would undo the reason for automating it. So match the fallback to the harm: a recommendation can simply stop; payments, care and supply cannot.
But be honest about who carries the risk when the fallback fails: the person on shift who was told to decide. The FAA and industry group put it plainly: pilots mitigate risks “on a frequent basis, and the aviation system is designed to rely on that mitigation”. Bainbridge, writing about a person asked to check a faster machine, was blunter: “The human monitor has been given an impossible task.”
If we put a person in the loop, we owe them the practice, the time and the information to do the job. Otherwise the loop is a place to put the blame, not a safeguard.
If your AI went dark tomorrow morning, who would still know how to do the work?

