Designing the AI future before It designs us
A few things stay with you after a good conversation. At AI Public Sector Week in March, I had the chance to host Giles Tully, ranked No.1 in the UK AI100 2025 and CEO of PinPoint Data Science, for a session called Designing the AI Future: Before It Designs Us. What stayed with me most was this: most organisations are not failing at AI strategy. They are failing at feedback.
That sounds simple. It is not.
We are good at starting things. Strategy documents, procurement processes, pilots, evaluations. What we are less good at is connecting the end of one cycle to the beginning of the next in a way that actually changes behaviour.
When that connection breaks, organisations do exactly what you would expect. People build local workarounds. Teams optimise for their own survival. Islands form. The collective outcome is worse than the sum of sensible individual decisions. Economists have a name for this kind of trap. What struck me was how precisely it describes what I see across public services, NHS trusts, and large private sector organisations trying to work through digital change.
The question is not whether AI can help. It is whether we are building the conditions for AI to help.
There is a principle worth holding onto: the purpose of a system is what it does, not what it was intended to do.
If your data governance policy repeatedly blocks the data flows that would improve patient outcomes, then blocking data flows is what that policy does, regardless of its stated intent. If your procurement process reliably selects vendors who are skilled at surviving long sales cycles rather than delivering value, then that is what your procurement process is optimising for.
This is uncomfortable because it removes the comfort of good intentions. But it is also clarifying. It tells you where to look.
The discussion I hosted kept returning to this diagnostic lens. Where does your system’s actual output diverge from its intended purpose? That gap is where feedback has broken down. And until you fix the feedback, AI is largely decorative.
Ashby’s Law of Requisite Variety is one of those ideas from systems theory that takes about thirty seconds to understand and a career to fully absorb. In short: you cannot control a complex system with a system that has less variety than the one you are trying to manage.
The NHS sees tens of millions of patients. Local context varies constantly. Technology and expectations shift rapidly. Leadership changes frequently. Yet many governance structures were built for a world that moved far more slowly, and they have not kept pace.
AI, used well, is not a solution to this. It is a mechanism for increasing the variety of your response. Different layers of AI, from predictive analytics to large language models, can handle different parts of the complexity problem. The issue is that most organisations bolt AI onto existing processes rather than embedding it into governance and feedback architecture.
AI does not reduce the complexity of the world. It can reduce the complexity of your response to it.
That is a meaningful distinction, and one I think gets lost in most AI conversations.
One of the sharpest observations from our discussion was about where feedback goes to die.
In many large organisations, clinicians raise issues. Tickets get processed. Context gets stripped. By the time the signal reaches someone with authority to act, it is low-fidelity noise. People eventually stop reporting because nothing changes. The loop exists on paper. It does not function in practice.
This is not a technology problem. It is a design problem. And it is fixable, but only if you treat the signal path as infrastructure that needs deliberate maintenance.
The question worth asking in your own organisation is: where does feedback lose either its urgency, its context, or its connection to a decision-maker? That is the point of redesign.
The discussion did not end with a grand call for systemic transformation. It ended somewhere more practical: redesign one feedback loop.
The most visible candidate is usually wherever duplication is highest. Duplicate work is a symptom of broken coordination. It signals that two parts of a system are not learning from each other.
A second target is the gap between strategic and operational teams. When both levels share similar goals but cannot close loops quickly enough, inertia accumulates. LLMs, used deliberately, can help here: summarising regulatory flows, flagging recurring issues, and keeping executive and operational teams better aligned without requiring constant translation.
The real-time signal that enables learning is whatever you can measure weekly that reflects actual system improvement. If you cannot measure it, you cannot reliably steer it.
What stays with me most is not the operational argument, though that is compelling enough. It is the broader point about who designs the systems we live inside.
Algorithms already govern enormous parts of public life. Laws get encoded. Government processes become operational rules. Corporations become slow-learning systems built around goals that none of us explicitly agreed to.
AI is accelerating all of this. The question is whether we are designing these systems consciously, with feedback loops that keep them accountable to human outcomes, or whether we are accepting defaults built for other purposes.
That is not a technology question. It is a governance question. And it is one that Digileaders are well-placed to take seriously.
Watch the full conversation here: https://aipsweek.digileaders.com/talks/designing-the-ai-future-before-it-designs-us/
Read More Data & Decision Making
