Most AI governance frameworks were written for models that answer questions. They address bias, explainability, data protection and model risk, all sensibly. Then they encounter software that takes action, and the assumptions underneath them stop holding.
The gap between governing a system that advises and governing one that acts is where the current regulatory risk sits, and it is wider than most policies acknowledge.
What changes when software acts
An advisory system produces output a human evaluates. The human remains the decision-maker, accountability is unambiguous, and a wrong output is caught before it has consequence.
An agentic system compresses that. The decision and the action happen together, frequently without a human in the path. Three things follow immediately.
Accountability becomes genuinely unclear. When an agent commits an error with financial or regulatory consequence, responsibility could sit with the person who approved the deployment, the team that configured the guardrails, the vendor of the model, or the business owner of the process. In most organisations this has never been decided, and it gets decided under pressure during an incident.
Prevention matters more than detection. With advisory systems, catching errors before action is the control. With agentic systems, the action has occurred. Controls must operate before the fact, which means permission design carries the load that human review used to.
Speed removes the natural brake. A human making a systematic error makes it at human pace, and someone usually notices. An agent makes the same error across the full queue before anyone opens a dashboard.
The regulatory picture
The specifics evolve, so treat what follows as orientation rather than legal advice, and take actual advice for your circumstances.
Under UK GDPR, the relevant provisions on automated decision-making apply where decisions are made without meaningful human involvement and produce legal or similarly significant effects. The word meaningful does substantial work: a human who approves a queue of recommendations at high speed, without practical ability to assess each, may not constitute meaningful involvement.
The EU AI Act introduces risk-tiered obligations, with heavier requirements around documentation, human oversight, logging and conformity assessment for higher-risk uses. UK-based firms serving EU customers are frequently in scope regardless of where they operate.
Sector regulators add their own layers. Financial services carries obligations around outsourcing, operational resilience and consumer outcomes that apply whether the process is run by a person or a system. Healthcare, legal services and insurance each have their own.
A useful test for whether human oversight is real: could the reviewer have reached a different conclusion, given the information and time available to them? If the practical answer is no, you have a control on paper rather than in operation, and that distinction matters to a regulator.
The controls that actually work
Permission scoping by default
Each agent receives access to the specific systems and operations its role requires, and nothing further. Read access separated from write. Write access separated from delete. This is ordinary security practice, and it is routinely skipped during AI pilots because broad access makes development faster.
Worth auditing explicitly: the union of permissions across a team of agents can quietly exceed what any individual employee holds.
Consequence-graded authority
Not every action needs the same treatment. Categorise by reversibility and impact. Reading data and drafting output can run autonomously. Actions that are difficult to reverse or externally visible, sending client communications, committing funds, altering records of account, warrant confirmation. Actions with regulatory consequence warrant a named human decision.
Logging built for reconstruction
The standard to design against is a specific one: could you reconstruct, from logs alone, exactly why the system did what it did on a given date, eighteen months later, for a regulator or a court? That means inputs, retrieved context, model and prompt versions, tool calls and their responses, the decision path, and any human interventions.
This is more than most teams instrument by default, and retrofitting it after an incident is both expensive and unconvincing.
Rate and budget limits
Caps on actions per period and spend per period bound the worst case. An agent in an unexpected loop should hit a ceiling and stop, not run until someone notices the invoice.
Kill switches that have been tested
A documented, tested mechanism to halt agent activity immediately, with a named owner who can invoke it without an approval chain. Untested kill switches have a poor record of working when first used.
Accountability structures
Technical controls are necessary and insufficient. Governance also requires answers to organisational questions, decided in advance and written down.
Who owns each deployed system operationally, by name. Who approves changes to its guardrails. What triggers escalation to senior management. What triggers a regulatory notification. Who is accountable when the system causes harm.
Organisations that handle incidents well have these documented before deployment. The ones that handle them badly are constructing the answer while the incident is live, which is visible to everyone involved, including regulators.
Auditability as a design constraint
Auditability is far cheaper designed in than added. Several choices affect it substantially.
Deterministic routing where possible. If the same input reliably produces the same path, behaviour can be tested and explained. Model-based routing everywhere makes both harder.
Structured intermediate outputs. Agents passing structured data rather than free text produce logs that can be queried rather than read.
Explicit uncertainty. Systems that record when they were unsure, and route accordingly, generate evidence that oversight was risk-based rather than uniform.
Version pinning. Knowing which model version produced a decision matters when explaining behaviour that has since changed underneath you.
Governance as an enabler
Governance is frequently framed as friction slowing deployment. In practice the opposite is common: organisations with clear frameworks deploy faster, because the questions have been answered once rather than relitigated for every project.
The constraint on agentic adoption in serious organisations is rarely capability. It is the inability to answer what happens when this goes wrong, and who is responsible. Answer that properly and the path clears.