How to Evaluate a Regulatory Change Agent Before You Trust It

To evaluate a regulatory change agent, test four things: how it maps a regulatory update to your obligations, how it reasons (the execution trace), where it hands decisions to a person, and whether it leaves an audit trail an examiner will accept. Score it on your own past rule changes, not a vendor demo. This guide gives you the four tests, a scorecard and a staged way to earn trust. It applies whether the agent is an AI agent, a software workflow or a third-party platform. We also show how Madison and Opencontroller by Lyzr measure up.

TL;DR

• The problem: a demo shows the agent at its best. Trust needs evidence from your own rules, your own policies and your own edge cases.
• Test 1, map: does it find what changed in official sources and trace it to your obligations, policies and controls?
• Test 2, trace: can you read every step, tool call and decision, and does it still pass after a change?
• Tests 3 and 4: a named person decides before anything goes live, and the log can answer an examiner without reconstruction.
• The rule: agents propose, people approve. Start in shadow mode, widen only as the evidence earns it, and re-test on every change.
• Our answer: Madison does the regulatory mapping, with cited proposals a named person approves. Opencontroller governs the agent: owner, approvals, audit trail.

What is a regulatory change agent?

A regulatory change agent is software that watches for new or amended rules, works out what they mean for a specific institution, and proposes the response: which obligations changed, which policies, procedures and controls they touch, and what should be updated. Unlike a feed that only flags a change, an agent carries the work through several steps and hands a person a finished proposal.

That makes it more useful and harder to trust. It reads source text, decides what applies, and may write to systems. Each of those is a point where it can be wrong without looking wrong. Evaluate it the way you would evaluate a new analyst who works at machine speed and never admits doubt unless it has been built to.

Why does a demo not prove an agent can be trusted?

A demo uses a rule the vendor chose, a policy library the vendor prepared and a question the vendor already answered. It cannot show how the agent behaves on your edge cases. Two patterns recur in regulatory technology:

• Flagging is not interpreting. Many tools detect that a document changed but leave the hardest analysis to the customer: what the new text requires and which of your controls it affects. One RegTech vendor’s analysis, reported by FinTech Global, makes this point about horizon scanning.
• Automation can be overstated. The same coverage describes a bank that found people doing the document processing behind a platform sold as AI-driven. The claim is vendor-sourced, so treat it as a prompt to ask, not as a statistic.

The fix is to move the proof from the vendor’s examples to your own. Run the agent on rule changes you have already worked through by hand, so you know the right answer. For a wider view of where banks are putting agents to work, see our overview of AI agents in banking.

What are the four tests for a regulatory change agent?

The four tests cover what the agent reads, how it reasons, who stays in charge and what it leaves behind.

Test 1: Does it map changes to your obligations, from official sources?

The agent has to do three things correctly before anything else matters: read the right sources, decide whether a change applies to you, and trace it to your own documents. Check scope and permissions first, because a change that alters data-handling rules may require action without touching the model at all.

The last row is where most of the value, and most of the risk, sits. Identifying that a rule changed is the easy part. Naming which clause of which of your policies it touches, and which controls depend on that clause, is the work an analyst spends days on. See how we frame it in Madison, Lyzr’s compliance layer for banks.

Test 2: Can you inspect the execution trace?

Do not score the final answer alone. By the time an answer appears, most of the agent’s work has already happened. Read the planning stage, the intermediate reasoning and every tool call, and check that someone could reconstruct the run later.

Score tool use in two parts: did it pick the right tool, and did it pass the right arguments, with no invented values. For planning, check that the proposed plan respects required steps and constraints, such as always checking the effective date, rather than matching one ideal sequence. Look for unnecessary tool calls, redundant steps and missed edge cases such as missing data or conflicting guidance. Run each scenario more than once, because an agent that passes once and fails the next time is not reliable.

Build a test set from your own history

A practical starting point is 30 to 50 real scenarios, a range that evaluation guides for compliance tools suggest. It is not a statistical standard, so grow the set with every failure you find. Mix difficulty, and include cases that should be rejected as well as accepted.

Score each run on a few clear metrics

Keep every confirmed failure in the set, so a fix cannot quietly regress. Re-run the whole set after any change to the model, prompt, rule sources or your own policy library.

Who decides, and when must the agent stop?

An agent that can propose is useful. An agent that can act on its own is a risk. Verify four controls before it touches a live policy:

1. Escalation triggers.

Hard thresholds force a human review of high-risk or ambiguous determinations. Ask to see them, and test that they fire on your edge cases.

2. A named owner.

An experienced compliance professional holds final authority to accept, modify or reject each assessment before any policy or system change goes live.

3. Read before write.

The agent reads regulatory text and your library, and proposes. Writing to a policy, a control or a ticket needs a separate, approved step.

4. Separation of duties.

The person who asks for a change should not be the only person who approves it. Check that the approval flow enforces this, not just that a policy says so.

What the agent may do, and what a person must do

Test 4: Does it leave an audit trail an examiner will accept?

Trust that cannot be shown later is not trust. The agent must produce immutable records of why each decision was made, which source text it referenced and how your internal controls map to the new rule. The test is simple: could it answer a detailed question about a past response without anyone reconstructing the story from memory?

This is what makes the work defensible. For the wider picture on the difference between using AI for compliance and governing that AI, read AI in compliance vs AI governance.

What does a scorecard look like?

Score each criterion from 0 to 2. Give 0 when there is no evidence, 1 when it was claimed or shown on the vendor’s data, and 2 when it was shown on your data with logs you can keep. A sensible rule is that a 0 on Test 3 or Test 4 is a stop, whatever the total. This is a suggested approach, not a regulatory standard.

How do you earn trust in stages?

Passing the tests once is a start. Trust is earned by widening what the agent may do only as the evidence builds. Four stages work well.

Step 1: Run in shadow modeLet the agent process real rule changes while analysts work as usual, and ignore its output. Compare its results with the human answer and score them on the metrics above.

Step 2: Show proposals, change nothingGive analysts the agent’s impact assessments as suggestions. Track how often they accept, amend or reject them, and why.

Step 3: Gate every proposalLet proposals flow into the real workflow, with a named person approving each one. Keep the trace and the approval together in the record.

Step 4: Sample the low-risk workOnly for categories with a strong record, accept low-risk items such as clear dismissals, and review a fixed sample. Keep high-risk and ambiguous items fully gated.

Step 5: Re-test on every changeRe-run the regression set whenever the model, prompt, rule sources or your policy library change. Register the agent and its owner so the re-test has someone accountable. See our AI agent inventory guide and agent lifecycle management.

What do regulators expect when you buy an agent?

There is no checklist written for regulatory change agents. Two points frame the expectation for US banks.

Agents fall to your own governance. SR 26-2, issued April 17, 2026, places generative and agentic AI outside its scope and tells banks to rely on their own risk management and governance practices. A bank therefore sets and documents its own evaluation standard for agents, which is what this guide supports.

Buying the agent is a third-party relationship. The June 2023 interagency guidance on third-party relationships, issued by the Federal Reserve, FDIC and OCC, describes a lifecycle of planning, due diligence and selection, contract negotiation, ongoing monitoring and termination. It scales diligence to the risk of the activity, and boards remain responsible for oversight. The guidance is non-binding, but examiners can look at the functions a third party performs for the bank.

Where do Opencontroller and Madison fit?

The two products answer different halves of the same problem. Madison is the compliance layer for banks. It does the regulatory work: it maps a rule change through obligations, policies and controls and shows where coverage has gaps. Opencontroller by Lyzr is the control plane that governs the agents doing that work. It registers each one with a named owner, enforces approvals and budgets, and records an audit trail, so the controls you tested stay in force after the contract is signed.

Use them together, or use Opencontroller to govern a regulatory change agent from any vendor. Below, we show how each answers the four tests.

How the two fit together

After the four tests, what Opencontroller adds

It complements, rather than replaces, your GRC and model risk tools. Agents hosted elsewhere register as external entries and are governed without being moved. For the registry concept in depth, read our AI agent registry guide.

How does Madison measure up on the four tests?

Madison is Lyzr’s agentic GRC product for US banks and credit unions. It holds a bank’s regulatory obligations, policies, procedures, controls and examination history as one connected model, and puts agents to work on it alongside your team. Regulatory change and coverage is available now. Here is how it answers each test, based on how it is built. Verify every row on your own rule changes, as you would for any vendor.

See both on your own rules

Bring your policy library and three past rule changes to a working session. We will run the four tests on Madison and show how Opencontroller keeps the agent accountable.

Book a demo

Explore Madison

Take the sprawl audit

Frequently asked questions

How do you evaluate an AI agent before trusting it?

Test it on your own data, not a demo. Check how it maps inputs to your obligations, read the full execution trace rather than only the final answer, confirm a named person approves anything that changes a policy or system, and verify it leaves an immutable audit trail. Then re-test whenever the model, prompt or sources change.

What is a regulatory change agent?

A regulatory change agent is software that monitors new or amended rules, works out what they mean for a specific institution, and proposes the response: which obligations changed and which policies, procedures and controls they affect. Unlike a feed that only flags a change, it carries the analysis through several steps for a person to review.

How do you test the accuracy of an AI regulatory change tool?

Build a test set from rule changes you have already handled by hand, so you know the right answer. Score obligation recall, precision, applicability accuracy, citation accuracy, mapping accuracy, repeatability and escalation correctness. Check a sample of citations against the exact source text, and keep every failure in the set.

How many test cases do you need to evaluate an AI agent?

There is no standard number. A practical starting point is 30 to 50 real scenarios, a range that evaluation guidance for compliance AI tools suggests, covering easy, medium, hard, edge, irrelevant and missing-data cases. Grow the set with every failure you find, and run each scenario more than once.

What is an execution trace and why does it matter?

An execution trace is the step-by-step record of what an agent did: its plan, each tool call and its arguments, the intermediate reasoning and the result. It matters because a correct-looking final answer can hide redundant calls, skipped checks or unflagged edge cases that only the trace reveals.

What should I ask a regulatory change management vendor?

Ask where it gets its rules and whether sources are official and named. Ask whether it extracts obligations or only flags changed documents, how it decides applicability, what a person must approve, and how much of the work is automated versus done by people. Ask to see a full trace and a decision log for one of your own rule changes.

Can an AI agent decide whether a new regulation applies to my bank?

It can propose an answer, but a named person should accept it. Applicability depends on your charter, size, products and states, and on whether a rule is proposed or final with an effective date. The agent should show its inputs and escalate ambiguous cases rather than decide them silently.

Do regulators require human review of AI regulatory change decisions?

There is no rule written specifically for regulatory change agents. SR 26-2 places generative and agentic AI outside its scope, so banks rely on their own governance. In practice, boards remain responsible for oversight, and a named compliance professional approving changes before they go live is the defensible standard.

How often should you re-evaluate a regulatory change agent?

Re-run the regression test set whenever the model, prompt, rule sources or your own policy library change, and sample live output on a fixed schedule. A single evaluation at purchase is not enough because agents and the rules they read both keep changing.

What are red flags in an AI compliance vendor demo?

Watch for claims of automation with people doing the work behind the scenes, tools that flag a changed document but do not interpret the obligation, answers with no citation to source text, no access to the execution trace, and no immutable log of who approved what.

How does Lyzr’s Madison handle regulatory change?

Madison is Lyzr’s agentic GRC product for US banks and credit unions. It holds a bank’s obligations, policies, procedures and controls as one connected model, shows analysts what a rule change touches, and proposes responses for a named person to accept, amend or reject. Every output traces back to the rule text it came from.

Is a regulatory change agent covered by third-party risk guidance?

If you buy it from a vendor, yes. The June 2023 interagency guidance on third-party relationships describes a lifecycle of planning, due diligence, contract negotiation, ongoing monitoring and termination, scaled to the risk of the activity. The bank and its board remain responsible for oversight of the function.

Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here