AI Assurance Shouldn’t Be a Final Gate

The government published a new AI Risk Management Toolkit last week.

What caught my attention wasn’t the list of AI risks.

It was the intended audience.

The toolkit isn’t aimed solely at AI specialists or governance teams. It is designed for multidisciplinary teams involved in designing, procuring, operating and delivering AI-enabled products.

I think that’s important.

There is a risk that, as AI adoption accelerates, organisations recreate a familiar governance pattern:

• Build something.
• Complete the paperwork.
• Take it to an assurance board.
• Get approval.
• Go live.

That model is already questionable for conventional digital services. For AI, it is particularly weak.

An AI system can behave differently as its data changes, users find unexpected ways to use it, suppliers update models or the environment around the service changes.

So assurance cannot simply answer:

“Was this safe when we approved it?”

It needs to keep answering:

“Is this still producing the outcome we intended?”

That changes where assurance sits in the architecture.

Risk assessment should start when the service is being designed, not when the model is ready for deployment.

Architects should be asking what decision or activity AI is supporting, what data it depends on, where human judgement remains necessary, what happens when the AI is wrong and how the organisation will know when performance starts to deteriorate.

The government’s AI Playbook already makes this lifecycle point explicitly: AI quality assurance should continue through testing, validation and monitoring once the service is operational.

There is also a supplier dimension.

Many public bodies will consume AI rather than build it.

That doesn’t remove the need for assurance.

If a supplier changes the underlying model, can we assess the impact?

Do we understand what data crosses the organisational boundary?

Can outputs be independently tested?

Can the organisation switch model or supplier without redesigning the whole service?

And, importantly, who owns the risk when something goes wrong?

Those are architecture and commercial questions, not just model-governance questions.

The same applies to data.

Government’s AI-ready data guidance stresses quality, governance, metadata, interoperability and clear stewardship.

That is significant because many apparent “AI problems” will actually begin further down the stack.

Poor source data.

Unclear ownership.

Weak integration.

Processes that were never designed properly.

Or a service whose outcome was never clearly defined in the first place.

Adding an AI model doesn’t fix those things. Sometimes it simply makes the consequences harder to see.

For architects, I think the useful principle is straightforward.

Don’t architecture the AI separately from the service.

Treat the model, data, human decision points, integrations, supplier dependencies, controls and operational monitoring as parts of the same system.

Then assurance becomes something useful.

Not a hurdle teams have to clear before go-live.

But a way of continuously answering whether the service remains safe, effective and worth operating.

Good AI governance isn’t about approving an AI system once.

It’s about maintaining confidence in the service for as long as AI remains part of it.