Imagine an AI system that quietly clears reconciliation breaks it shouldn’t. The application stays available, every dashboard is green, and the problem only surfaces days later when a client queries their statement. Restoring the system wouldn’t help. The real question is whether anyone in the organisation still has the experience to work out what went wrong.
That scenario sits at the heart of something I’ve been thinking about since I wrote about the Confidence Gap at the start of the year: the difference between what organisations believe about their resilience posture and what they can actually substantiate through evidence. Successful testing, regulatory compliance, apparent stability and effective controls can all contribute to confidence without necessarily demonstrating an ability to withstand unfamiliar disruptive conditions.
In this article I want to explore how that gap can develop and widen over time, particularly as organisations make increasing use of AI.
Organisations are investing heavily in AI, and understandably so. The opportunities to improve efficiency, reduce costs and make better use of information are significant. But we need to consider something beyond how well the technology performs. What happens to an organisation’s ability to respond and recover as more of its activities become dependent on AI?
We’ve Been Here Before
In 1983, Lisanne Bainbridge published Ironies of Automation, highlighting a paradox that remains relevant more than four decades later. As organisations automate routine activities, the people responsible for intervening when something goes wrong may have fewer opportunities to develop and maintain the very skills they need.
What strikes me is that we’ve known about this for more than forty years, yet the problem hasn’t gone away. What’s changed is the nature of automation. AI now reaches into analysis and judgment, not just routine tasks.
Bainbridge examined the consequences for human operators. The wider question for operational resilience is what happens when similar effects begin accumulating across an organisation.
She wasn’t alone in examining these issues. Raja Parasuraman and his colleagues researched automation complacency and automation bias, looking at how people can become increasingly reliant on automated systems and less likely to recognise failures or challenge incorrect outputs. Jens Rasmussen also examined how organisations can gradually move towards the limits of what they can safely manage through ordinary pressures to improve efficiency and reduce costs.
What I find interesting about Rasmussen’s work is that nobody necessarily needs to make a bad decision. An organisation can gradually lose some of its safety margins through a series of decisions that, individually, make perfectly good sense.
AI makes this particularly relevant because it is increasingly being used for activities involving analysis, monitoring, decision support and other work previously undertaken by people. As organisations make greater use of these capabilities, they’re also changing how work is done, who performs it and what expertise they need to retain.
My question is whether we’re giving enough attention to how those changes could affect recoverability.
What Happens When AI Becomes the Way We Work?
AI introduces a different kind of organisational dependency that may be difficult to recognise through normal resilience assessments.
Traditionally, when we look at dependencies, we consider the infrastructure, applications, data, suppliers, people and facilities that support our business services. We want to understand what happens if those dependencies become unavailable and whether we can continue operating without them.
AI introduces another consideration. As organisations increasingly use AI to perform activities previously undertaken by people, they may gradually lose some of the knowledge and expertise needed to carry out those activities independently.
This means the organisation’s dependency on AI could increase over time, even if the technology itself hasn’t changed. The organisation may become less capable of operating without it because its people no longer have the same knowledge and experience.
Imagine an organisation introducing AI into a business-critical process. Over several years the AI models perform well, so human verification decreases, staffing changes and knowledge becomes concentrated in systems and a smaller number of specialists. Performance indicators show continuous improvement, while the independent expertise needed when something goes wrong may be quietly declining.
Here’s an example comparing two organisations.
Organisation A has relatively low automation. Humans perform and understand most decisions. Operations are slower, but staff regularly practise the underlying processes and retain direct experience of how the work is done.
Organisation B has much higher automation. AI performs most of the analysis and supports many of the decisions. Normal operations are faster, performance may be more consistent and employees spend less time undertaking activities that previously required significant human involvement. However, those employees may also have limited recent experience of operating independently.
Organisation B could significantly outperform Organisation A under normal conditions. It could also have excellent technical recovery arrangements. But if its AI models begin to produce unreliable results, it may have fewer people capable of recognising the problem and understanding what needs to be done to rectify the issue.
This doesn’t automatically mean Organisation B is less resilient. It may have developed effective controls, alternative arrangements and ways of maintaining the necessary expertise. The point is that normal operational performance tells us very little about how either organisation would cope when its usual way of working can no longer be trusted.
And crucially, Organisation B doesn’t need to reproduce every automated task manually. That might not be realistic or necessary. It needs to retain enough capability to prevent intolerable consequences and continue delivering essential business outcomes through different trusted arrangements.
This is where Minimum Viable Capability becomes relevant: the smallest trusted capability required to avoid intolerable harm. The test isn’t whether the organisation can return to its old way of working, but whether it can maintain the necessary business outcomes when its AI-enabled operating model can no longer be trusted.
Who Would Notice?
The real concern is the loss of independent judgment rather than simply losing the ability to carry out activities manually.
An organisation may have perfectly adequate alternative technology but still lack the expertise to decide whether the information it receives can be trusted. People who have spent years investigating problems, challenging results and dealing with unusual circumstances develop an understanding that isn’t always captured in procedures or documentation. It’s tacit knowledge.
The people best placed to recognise that organisational capability is deteriorating may be the same people whose roles are being changed or removed.
Experienced employees often understand far more about a process than what is written down. They know where problems tend to occur, which assumptions don’t always hold and which workarounds help keep things running when something unusual happens.
As AI increasingly undertakes these activities, those individuals may become less involved. Some may move into other roles, while others may leave the organisation entirely. Over time, the organisation could lose both the ability to perform certain activities independently and the knowledge needed to recognise that this capability is deteriorating.
How does an organisation identify that it’s losing an important capability when the people who understood its importance are no longer there? Operational performance may continue improving, providing little obvious reason to question whether anything has been lost.
When AI Can No Longer Be Trusted
Let me return to the reconciliation example. Matching transactions has been automated for years, but investigating the exceptions (the breaks that don’t match) has traditionally been done by experienced people who understand why differences arise and whether they matter. Imagine an organisation that now uses AI to investigate those breaks, suggest explanations and automatically clear the ones it judges to be resolved. Over time, the volume of human investigation falls and fewer people build that experience.
Now imagine the AI begins clearing certain breaks incorrectly, treating genuine discrepancies as explained timing differences.
The results appear reasonable. The application remains available, infrastructure is healthy and transactions continue to be processed. From a traditional technology perspective, nothing has failed. Days later, a client query about their statement reveals a discrepancy that the reconciliation should have caught.
A reconciliation exists to catch errors, so when the AI performing it gets things wrong, the organisation risks losing one of the very controls it relies upon to identify problems. If there aren’t sufficient independent checks, those errors could remain hidden until something else, like that client query, brings them to light.
The organisation now needs to understand what has happened, which reconciliations may have been affected and whether previous results can still be trusted. But does it retain the expertise needed to investigate the problem?
Someone who has spent years investigating reconciliation differences may recognise patterns that a less experienced employee could easily overlook. If AI has progressively taken over that work, the organisation may have fewer people with the practical experience needed to identify and resolve the issue.
This is particularly important because conventional disaster recovery may offer little assistance. Restoring the same application, model and data onto alternative infrastructure could simply reproduce the problem elsewhere. We might successfully recover the technology without resolving the problem affecting the business.
AI could improve recoverability for some types of disruption. It might help organisations identify infrastructure problems earlier, analyse incidents faster or coordinate recovery activities more effectively. But an integrity failure is different. If the information supporting business decisions becomes unreliable, and employees no longer have the experience needed to recognise or challenge it, the organisation may struggle even though the technology remains available.
One thing I’ve noticed over the years working in technology resilience is how easily we can confuse recovering the technology with recovering the business. I’ve seen recovery exercises where infrastructure comes back, applications are accessible and the test is considered successful. But that doesn’t necessarily tell us whether the business could operate in that environment for any meaningful period of time.
I wonder whether we’re creating a similar problem with AI.
We might demonstrate that the AI platform works, that its outputs meet expected standards and that the technology can be recovered. But does that demonstrate that the organisation can continue operating when the AI itself can no longer be trusted?
There’s another consideration here that I’ve been referring to as recoverability lag.
As organisations introduce AI, the way they operate can change quite significantly. Processes become automated, responsibilities change and activities that previously required human involvement are increasingly undertaken by technology. But our recovery arrangements may not be changing at the same pace.
We may still be testing recovery procedures designed around an operating model that no longer exists in quite the same form. The infrastructure can be restored, applications accessed and the recovery test recorded as successful. Meanwhile, the people, expertise and alternative arrangements those procedures assume will be available may have changed or disappeared.
For me, recoverability lag describes what happens when an organisation changes how it operates faster than it updates and tests the capabilities it needs to recover.
So How Do We Test Resilience to AI Dependencies?
One approach would be to introduce scenarios where AI continues operating but starts producing incorrect information. Using the reconciliation example, I’d want to understand how quickly the organisation could recognise the problem, who had the authority to suspend the automated process and whether it could identify which previous reconciliations were affected.
Then there’s the question of how the business continues operating. Do employees have access to trusted information? Do they retain sufficient expertise to investigate the problem? Can essential activities continue using independent sources and alternative processes, and for how long? We would also need to establish when the AI system was safe to trust again.
The point isn’t simply to demonstrate that AI can be switched off. It’s to demonstrate that the organisation can continue operating within acceptable limits when it can no longer rely on it.
Occasional manual testing may not be sufficient either. If employees can only operate independently for 20 minutes before workload, complexity or transaction volumes overwhelm them, that doesn’t provide much assurance that the business could continue operating during a prolonged disruption.
We don’t necessarily need to recreate every activity that AI performs, but we do need to understand which capabilities are essential, what alternatives exist and whether those arrangements actually work.
The Confidence Gap
I’m not against AI. Far from it. I think it has a lot to offer, including improvements to operational resilience itself.
But as organisations become more confident in AI, they’ll naturally look for ways to improve efficiency, reduce costs and change how work gets done. None of those decisions necessarily needs to be wrong, but taken together, they may gradually change what the organisation is capable of doing when things don’t go to plan. We can measure the efficiency gains. Are we also measuring what we might be losing along the way?
For me, that’s where the connection to the Confidence Gap sits. An organisation may have strong evidence that AI has improved its operational performance without having equivalent evidence that it can continue operating when the technology can no longer be trusted.
As organisations become increasingly capable because of AI, how do we ensure they don’t become progressively less capable without it?