Trust as the currency of AI scaling. What separates demos from production in regulated sectors
- 3 days ago
- 16 min read
Over the past two years, organizations in Europe have spent vast sums on artificial intelligence, and presentation slides are full of promises of transformation. But when I talk to leaders who speak openly about their implementations—rather than in a boardroom presentation mode—it almost always sounds the same: “We have a few pilots, a cool demo, a couple of chatbots, but we still don’t see a tangible impact on the P&L.”
This isn’t a problem of model power. It isn’t a problem of technology availability. The real problem lies elsewhere—in trust. More specifically, in the fact that organizations are afraid to entrust AI systems with real-world processes on which the business depends. And they have very rational reasons for this.
In the era of agents—that is, systems that not only generate text but also take actions within a company’s systems—trust ceases to be a soft topic in the realm of PR or CSR. It becomes a hard prerequisite for scaling. If an organization doesn’t trust the system, then no matter how impressive the demo is, the agent will be confined to a sandbox. And an agent in a sandbox doesn’t generate value; it generates costs.
A paradox every CIO already knows: technology everywhere, value on slides
At a high level, the paradox looks simple: AI adoption is growing, technological maturity is growing, but the real value from implementations is concentrated in a relatively narrow group of companies. Most organizations are stuck in a perpetual “pilot” phase — something is being tested, something is undergoing a proof-of-concept, something is showcased as an innovation, but the company’s day-to-day operations continue at the same old pace.
This disconnect is particularly evident when it comes to agent systems. An agent capable of independently initiating a process, invoking a tool, or making a decision on the user’s behalf immediately hits a barrier that didn’t exist in the era of chat. A demo at a conference is impressive, but when it comes to integrating the same mechanism into credit processes, underwriting, claims handling, or back-office operations at a regulated institution, the same question arises—and it always does: “What if something goes wrong, and who is responsible?”
Without an answer to this question, the technology remains a novelty, no matter how brilliant its capabilities may be. Trust, therefore, is not a “nice-to-have.” It is the missing link between demonstrations of capability and lasting change in the way we work.
Why trust in AI is a hard business requirement, not a PR topic
Trust is a prerequisite for a very simple economic reason: an agent we don’t trust becomes a costly add-on. If a human has to check everything the agent has done, line by line, then at best the organization gains nothing, and at worst it adds work and an extra layer of tools to maintain. An agent under full manual supervision is the most expensive way to do what a team of people has been doing well so far.
Trust in this context operates on two levels that are easy to confuse but require completely different mechanisms:
The first level is the trust of the end user—the person who works with the agent on a daily basis. This is personal trust, built through interaction, transparency, and a history of small successes and corrected errors. This trust is largely determined by what I wrote about in the previous article in this series—the AI-native interaction design.
The second level is organizational trust—from management, risk functions, compliance, IT, and audit. This is institutional trust, built not through a demo, but through governance, control mechanisms, auditability, and a clear chain of responsibility. And it is this trust, the lack of which halts 90% of agent implementations in regulated sectors.
Without both of these layers, the agent remains a gimmick. And that is why companies that are serious about scaling AI start the conversation not with the question “what can this model do,” but with “how will we manage it, monitor it, and take responsibility for its decisions.” The order of these questions matters. Organizations that start with the model and add governance “later” almost always end up in perpetual pilot mode.
A new class of risks: not “AI says silly things,” but “AI does things we didn’t anticipate”
In the world of chatbots and classic general AI, the main concern was hallucinations—the system says something convincingly, but untruthfully. This was a semantic, localized risk, largely detectable by a human reading the output.
When we move to agents, the risk reaches a whole new level. The problem is no longer just what the system will say, but what it will do. An agent might invoke the wrong tool at the wrong time. It might make a series of small, seemingly harmless decisions that, in aggregate, have a significant impact on the customer, the regulator, or the bottom line. It may operate outside expected boundaries if the rules are unclear or too loose. It may also—and this is the most difficult case—behave correctly in ninety-nine scenarios, but catastrophically in the hundredth, which no one anticipated during the design phase.
Risks become process-related rather than semantic. This means that classic approaches to model validation—quality metrics, data tests, offline evaluations—are still necessary, but have long since ceased to be sufficient. We need to start looking at agent systems the same way we look at complex operational processes in banking, energy, and aviation: from the perspective of workflow paths, checkpoints, permissions, contingency scenarios, escalation procedures, and accountability.
More importantly—in regulated sectors, these requirements already exist; they just aren’t called “requirements for AI agents.” DORA requires financial institutions to ensure operational reproducibility of every material decision. The EU AI Act for high-risk systems (credit scoring, HR, critical infrastructure, public administration) mandates auditability, human oversight, and model lifecycle documentation. NIS2 requires strict access controls and the ability to respond to incidents. These are regulations that are coming into force or are already in effect, and which in practice define a minimum threshold of trust below which an agent in a regulated sector cannot operate—not because someone thinks so, but because the law says so.
The situation in which most large organizations in Europe find themselves today is therefore paradoxical: risk awareness is growing faster than the ability to systematically address them. They know there is a problem. They know the list of threats. But they do not yet have the mechanisms or competencies to manage them at scale. From a trust perspective, this is a volatile mix—because awareness without capability leads to decision paralysis, and decision paralysis leads to perpetual crisis management.
How to build trust through design, not just declarations
Awareness of risks is not trust. Nor is the lack of it a lack of trust. Trust in an agent system is built through design—through architectural, technical, and organizational decisions made before the agent touches production, and then consistently enforced throughout its entire lifecycle. These decisions are rarely flashy and almost never appear in a demo for the board. But they determine whether, after a year of using the agent in production, the organization is at ease or lives in constant fear of the day when something goes wrong.
From my implementations, I’ve identified three specific mechanisms that together form the minimum without which no declaration of trust is valid. Each of them is obvious when you think about it—and each is overlooked in most projects I see.
First: start with a process where the agent can make a mistake but cannot cause irreversible damage. This is the first decision an organization makes before the first pilot—and it almost always gets it wrong. The temptation to show a “wow effect” to management leads to the selection of spectacular, critical processes in which the agent must immediately prove its value. This is the exact opposite approach to designing for trust. The first process must have two characteristics: real stakes (so that the conclusions carry weight), but reversibility (so that a mistake costs a learning experience, not a catastrophe). Classifying documents before the actual credit decision, generating initial draft reports that are then verified by an analyst, and providing automated responses to repetitive customer inquiries—where a human reviews them before they’re sent. Here, the agent can make mistakes; the organization sees what’s happening, corrects it, and learns. After six months of this work, the team understands where the agent is strong, where it is weak, where it makes mistakes, and where it behaves unpredictably. Only then does it make sense to move it toward critical processes. Trust doesn’t grow from demonstrations. It grows from well-handled exceptions.
Second: test the agent on a digital twin before deploying it to production. This is where the concept of the digital twin, which I introduced earlier in this series in the context of legacy modernization, returns in a third, equally important role. Before production deployment, the agent should be run on a twin of the system where it will eventually operate: on a data replica with appropriate masking, in an environment that behaves like production but isn’t production. This is exactly where you test scenarios that engineers did not anticipate: strange combinations of inputs, edge cases, atypical sequences of tool calls, behavior under load, and reactions to conflicting information from various sources. The twin won’t catch everything. But it will catch significantly more than unit tests and offline evaluations on a static set of cases—because it preserves the system’s dynamics, not just its structure. This is the difference between asking “does the agent answer a set of 1,000 test questions correctly” (the classic approach) and asking “does the agent behave stably when running in parallel for a week on a live stream of events” (the twin approach). Every project today addresses the first question. Few address the second.
Third—and this is the hardest part: design the system so that the human supervisor does not degenerate into a mere figurehead. Here, a phenomenon comes into play that has its own name in the literature on aviation safety, radiology, and autonomous vehicles—automation bias. The mechanism is simple and ruthless: if an agent operates correctly 99 out of 100 times, after three months the supervising human stops carefully reading what the agent has proposed and begins to approve it mechanically. Not because they are lazy. Because their brain is rational and optimizes work based on historical error rates. And history tells us: this agent doesn’t make mistakes, so there’s no point in checking it. In this way, “human over the loop”—the concept on which most agent-based projects in regulated sectors rely today—becomes an organizational fiction. The human formally supervises. In reality, they approve everything the machine presents. And precisely when the agent finally makes a mistake—in that one-in-a-hundred, edge-case, unforeseen scenario—the error passes through the checks unimpeded.
This degeneration cannot be stopped by “better training” or a “culture of mindfulness.” These are wishful thinking, not solutions. It must be stopped through design, via specific mechanisms built into the system. A few that I use in my implementations: forced active justification of approval (the human doesn’t click “OK,” but writes a single sentence explaining why this is the right decision—and this record remains in the logs); periodic audits of random decisions, in which someone other than the supervisor verifies whether the agent was actually correct (and whether the supervisor actually checked it or just clicked through); rotation of decision-makers, so that the same person does not supervise the same agent for more than a few weeks; a second supervising agent whose task is to question the first agent’s decisions from a different perspective and escalate discrepancies. Each of these mechanisms has a cost— nd all together they add several dozen percent of organizational overhead to the agent’s operations. And each of them is cheaper than the day when automation bias fails in a way that makes headlines.
To summarize these three mechanisms: trust in an agent system is not a state that is achieved once and maintains itself. It is a consequence of design decisions that must be made consciously at three levels: strategic (which process you choose), technical (how you test it before production), and organizational (how you maintain human vigilance once the agent has been running for months). Without one of these three, the other two are not enough. With all three, you have a trust architecture that stands a chance of surviving contact with reality.
Trust maturity: where organizations have the biggest gaps
When we view the maturity of trust in AI as a continuum, the picture is surprisingly similar across different sectors and countries in the region.
We’re doing relatively well with the technical component. We know how to connect a model, build a prototype, integrate an API, and create a working demo. The market for development tools is mature, the documentation is good, and there’s no shortage of teams capable of putting it all together.
The situation is much worse when it comes to strategy, governance, and control mechanisms. Who is responsible for the agent’s decisions? What are the rules within which the agent operates autonomously? What does the escalation process look like when the agent encounters an edge case? How do we measure whether the agent is behaving as we promised the regulator and the board? What do we do when an agent behaves inappropriately in a given scenario—who detected it, when, and what were the consequences?
If these elements are missing, the organization intuitively stops short of full implementation. This often isn’t stated explicitly in decisions—I tend to hear phrases like “let’s run a pilot, we’ll see,” “for now, let’s keep a human as the bottleneck of control,” “let this be an internal tool that doesn’t touch the customer.” These are rational defensive reactions. The point is that without transitioning from this phase to a consciously defined model of responsibility, the agent will never become a primary actor in the process. And full integration into the process is a prerequisite for unlocking the value the company was counting on when purchasing the tool.
"Human over the loop": designing the human role in a world of agents
Discussions about AI have long been dominated by the "human-in-the-loop" metaphor—a person somewhere within the workflow, deciding at critical moments, approving every significant decision. This model made sense in the era of gen-AI as an assistant, when the human was effectively the direct recipient and verifier of the result.
With advanced agent-based systems, a different picture emerges: "human-in-the-loop." This refers to a situation where a team of agents is capable of executing the entire core end-to-end process, while a human evaluates the results, reviews decisions in edge or exceptional cases, and gradually shifts their role from executing steps to designing rules, monitoring deviations, and continuously improving the process.
This is a fundamental shift—both in system architecture and in organizational culture. Leaders who take this seriously start with a very simple question: where do we want humans to have the final say, and what does that practically mean operationally? Designing "human over the loop" requires courage in delegation—a conscious acknowledgment that an entire class of decisions can be automated, while maintaining appropriate oversight, logging, and auditability. This is the moment when trust ceases to be an abstraction and becomes a concrete decision to cede some authority to the system. It doesn’t come easily to every leader—and rightly so, because this decision shouldn’t be easy.
What organizations that are truly successful with agents do
If we look at companies that not only experiment with agents but actually derive value from them, several recurring patterns emerge.
They have clearly defined responsibility for AI—it isn’t blurred between IT, business, compliance, and the “innovation lab.” A specific person takes ownership of ensuring the agent operates within a framework that the organization accepts. And that person has sufficient authority to enforce that framework when the project starts to stray outside of it.
They treat trust as an investment, not a cost that slows down progress. They consciously allocate resources to control mechanisms, monitoring, and auditability. They understand that three months of extra work on the governance layer at the start of a project saves two years of endless piloting.
They don’t limit themselves to sandboxes, but they also don’t hand everything over to agents right away. They start with specific, well-defined processes in which they can precisely describe risks and success criteria. They choose processes where the stakes are real (for credibility) but reversible (for safety). And they learn from these before moving on.
They invest in people’s competencies—not only among engineers, but also among business and operational leaders who must be able to talk with agents and about agents. The latter is underestimated. A leader who does not understand the difference between a hallucination and a process error is unable to meaningfully address an incident when it occurs.
Importantly, in these organizations, trust in AI is increasingly seen not as a regulatory requirement but as a business lever. Without trust, an agent will never enter a critical process and thus will never realize its full potential. Whoever is first in the industry to build trust is the first to unlock scale.
A minimal trust program: four elements that can be launched quickly
For many organizations, the prospect of a “full Responsible AI program” sounds like a multi-year transformation project that no one wants to start because no one is sure they’ll survive to see it through. Meanwhile, a practical starting point can be much more concrete if we take the topic seriously and don’t get bogged down in producing documents that no one will read.
In the practical implementation of CDF—a methodology I’m developing for regulated sectors—the first four elements of the trust program that can be launched in a few weeks are:
Selecting a single process where the agent is expected to deliver real value. This isn’t about a flashy demo. It’s about a process where we’ll collect meaningful data on the system’s impact, risks, and behavior—broad enough that conclusions can be generalized, yet narrow enough to avoid derailing the entire implementation by the third month.
Defining the operational framework and accountability model. Where the agent can act autonomously, where human approval is required, what the escalation thresholds are, who is responsible for incidents and for the model’s decisions. This must be documented, approved by risk and compliance, and technically enforced—not just stated in a policy.
Monitoring and inspection mechanisms. How we continuously observe the agent’s behavior. What quality, security, and risk metrics we track. How quickly we detect anomalies. Who reviews these metrics and at what frequency. Without this layer, we don’t know if the agent is behaving as promised—even if it functioned correctly on the day of deployment.
A training program for the people who will be working with the agent. And here I want to pause for a moment, because this is where most organizations make the same mistake—and it’s also a topic I originally intended to write about in a separate article, but ultimately I see that it belongs right here.
The Leader in the World of Agents: What No Model Can Replace
When I talk to executives about how to prepare an organization to work with AI agents, the most common technical question is: “How do we train people on prompting?” That’s the wrong question. Not because prompting isn’t important—it is—but because it’s not prompting that distinguishes organizations that will build trust in agents from those that will get bogged down in pilot projects. What sets them apart is the leadership maturity of the people tasked with overseeing these systems.
In the traditional world, a leader won by having access to information and the ability to make decisions faster than the competition. In the world of agents, information is no longer an advantage—every employee today has access to tools that, just two years ago, were reserved for specialized analytics teams. This means that a leader no longer wins by having more knowledge. They begin to win by being able to define a clear direction and purpose for actions, establish values and boundaries (what we do not do, even if it would be “effective”), and create conditions in which people and agents can work together rather than get in each other’s way.
There are three human competencies that no current model can replace—and which, in a world of agents, become a fundamental advantage for the organization, not the individual.
First: aspiration and meaning-making. Models are great at continuing the patterns they see in data. They cannot independently say, “Let’s do something that doesn’t exist yet.” Ambition, vision, and the courage to set a goal that goes beyond existing trajectories—this remains the domain of humans. A leader reads people’s emotions and fears regarding change, can translate abstract technology into a concrete story for the organization (“what does this mean for us”), and invites people to co-own this aspiration, rather than communicating it from above. Without this, AI can help create a plan, but it won’t answer the question of why we’re implementing it in the first place.
Second: judgment and responsibility for values. AI can calculate risks, summarize regulations, and propose “rational” solutions. However, it bears no responsibility for the consequences—legal, social, ethical, or human. It will not stand before the team and say, “That was my decision.” A leader decides when a textbook-correct solution is unacceptable from the perspective of the organization’s values. They shoulder the burden of choices in situations where every scenario is painful in some way. They combine hard data with a soft understanding of context—relationships, reputation, and trust. In a world where systems will generate a multitude of rational-sounding recommendations, the courage to say “no” may be just as important as the ability to say “yes.”
Third: nonlinear creativity and tolerance for chaos. Models are optimized to generate “the most likely next thing”—great variations on what already exists. Leaders whose organizations succeed think differently: not “how to do the same thing twenty percent better,” but “how to do something that will yield a tenfold effect.” This requires connecting threads from different domains, adopting outside perspectives, and tolerating chaos in the early stages when nothing is finalized. AI can help explore the space of possibilities, but it is humans who decide which “strange” ideas are worth keeping alive, even if they seem unlikely at first.
When an organization builds trust in agents, it is really building trust in the people who manage them. An agent does not inspire trust on its own—it inspires trust as part of a system that features a competent leader, clear rules, predictable responses to incidents, and a culture that allows concerns to be raised without penalty. That is why investing in leadership development—not in the sense of yet another training program, but in the sense of rituals where a leader practices working with agents in real-world, rather than laboratory, conditions—is the fourth and perhaps most important element of a trust program.
In the long run, it is precisely these human competencies and the organizational culture in which they flourish that will shape the companies capable of leveraging agents as a real multiplier of possibilities.
A world in which systems are doing more and more of the work paradoxically requires stronger, more mature leadership. AI can write, calculate, and make recommendations. But it won’t decide what your organization wants to be, what risks it’s willing to take, how it treats the people who work for it, or where its moral boundaries lie.
These questions remain—and will long remain—exclusively in the hands of humans. And that is the good news for anyone wondering where their own value lies in this new configuration.
A Micro-Pattern from Practice
The quickest test of whether “human over the loop” still works in your organization or has already degenerated into a mere figurehead is this: look at the last hundred decisions the agent proposed and that a human approved. How many of them did a human reject or correct? If the number is close to zero—you have two possibilities: either your agent is perfect (unlikely), or your oversight no longer exists. The third possibility—that the agent is actually making 100 good decisions in a row—is so improbable that if you choose it as an explanation, you are most likely wrong.
This series breaks down the transformation of AI in regulated sectors into seven layers:
All texts were published on the product blogs allclouds.pl — genesis-ai.app/blog and savant-ai.app/blog. The entire series is a record of what I’ve learned from working in regulated sectors in Poland and CEE. Decisions that had to be made faster than caution allowed, mistakes that taught me more than successes, intuition honed in conversations with no script, and the will to build something that doesn’t yet exist.
Thank you for reading all seven—if any of these layers resonate with what’s happening in your organization today, it’s a good starting point for a concrete conversation. |
Comments