Aligned
Intelligence
The next layer of AI will not only answer.
It will act.
The question is: Who will it act for ?
Peter Toumbourou
DEFINITION
Aligned Intelligence is built outward from the participant, not inward from its own capability, technology or design. Alignment is structural, not procedural.
The participant is the center of gravity - everything revolves around them.
The system's reasoning, recommendations and actions are hence shaped by the participant's interests, goals, constraints and welfare. As their circumstances change, it recalibrates to remain aligned with their best interests.
A new class of infrastructure, organised in three layers: what the system knows about the participant, how it reasons on their behalf, and finally how its loyalty is enforced.
The Inversion
Aligned Intelligence
Intelligence alone is not sufficient.
Intelligent Systems must also be aligned.
The first era of AI was about capability. The next is about duty.
When AI only answered, intelligence was enough.
Once AI begins to act, intelligence is no longer the constraint. Alignment is.
The question is no longer how smart the system is.
It is: "Who does the system serve when it acts?"
Every other question sits beneath that.
This is not a new problem. For centuries, society has had to manage the same basic risk: one person acting for another, with more knowledge, more power and different incentives.
Humans answered it with fiduciary duty.
A fiduciary is someone you trust to make important decisions on your behalf. Your doctor. Your lawyer. The trustee of your inheritance. The accountant who files your return. The law requires each of them to put your interests first, ahead of their own and ahead of their employer's.
The infrastructure of professional services rests on that obligation.
Agentic AI is now moving into those same positions of entrusted power.
Increasingly, the system reading the medical scan, drafting the legal letter, allocating the portfolio, or choosing the tax position is software. These systems will hold information that shapes people's health, money, legal standing and future. They are not yet bound by the same duty.
That duty cannot remain only with the human professional when the human professional is no longer the one doing the work.
It must move into the architecture.
That architecture is Aligned Intelligence: systems built around the participant as the centre of gravity, where every layer of reasoning, recommendation and action is governed by the participant’s context, constraints and best interests.
Aligned Intelligence
Intelligence alone is not sufficient.
Systems must also be aligned.
The first era of AI was about capability. The next is about duty.
When AI only answered, intelligence was enough.
Once AI begins to act, intelligence is no longer the constraint.
Alignment is.
The question is no longer how smart the system is.
It is: "Who does the system serve when it acts?"
Every other question sits beneath that.
This is not a new problem. For centuries, society has had to manage the same basic risk: one person acting for another, with more knowledge, more power and different incentives.
Humans answered it with fiduciary duty.
A fiduciary is someone you trust to make important decisions on your behalf. Your doctor. Your lawyer. The trustee of your inheritance. The accountant who files your return. The law requires each of them to put your interests first, ahead of their own and ahead of their employer's.
The infrastructure of professional services rests on that obligation.
Agentic AI is now moving into those same positions of entrusted power.
Increasingly, the system reading the medical scan, drafting the legal letter, allocating the portfolio, or choosing the tax position is software. These systems will hold information that shapes people's health, money, legal standing and future. They are not yet bound by the same duty.
That duty cannot remain only with the human professional when the human professional is no longer the one doing the work.
Aligned Intelligence
1. POWER
Powerful AI is now the default
Capability is no longer a scarce resource. Duty is.
Every frontier lab released a flagship model in the same class within months.
Aligned Intelligence
For the past few years, the AI race was about model size, reasoning depth, context length, multimodality. Anthropic, OpenAI, Google DeepMind, Meta, Mistral and Hugging Face pushed the same frontier. Stanford HAI's 2025 AI Index recorded the result: hardware costs for leading models fell roughly 40% in twelve months.
Then the open-weight wave caught up. In January 2025, DeepSeek released R1, a Chinese open-source reasoning model that matched OpenAI's o1 on mathematics, coding and engineering benchmarks at roughly five per cent of the cost.
Within weeks, Hugging Face hosted more than five hundred derivatives of it. Meta's Llama family, Mistral's open models and a wave of Chinese releases now run frontier-class reasoning on commodity infrastructure.
The advantage is no longer having access to a powerful model.
Powerful models are the default.
The advantage is building a system people can trust to act in their interests.
Performance on legal, medical and reasoning evaluations converged at the top. GPT-4 passed both the US Bar Exam and the US Medical Licensing Examination, in each case scoring above the threshold human candidates have to clear.
Capability is no longer scarce enough to define the category.
Capability is the floor.
MIT Technology Review concluded in early 2026 that benchmarks measure AI in a vacuum, while AI is used in messy contexts where real performance only emerges over time.
A model that aces the bar exam is not the same as a system you can trust with your wrongful termination case. A model that clears a medical licensing test is not the same as a system you would let prescribe medicine for your Elders.
The next moat is not built around the model.
It is built within it.
1. Powerful AI is the default
Capability is no longer a scarce resource
For the past few years, the AI race was about model size, reasoning depth, context length, multimodality. Anthropic, OpenAI, Google DeepMind, Meta, Mistral and Hugging Face pushed the same frontier. Stanford HAI's 2025 AI Index recorded the result: hardware costs for leading models fell roughly 40% in twelve months.
Performance on legal, medical and reasoning evaluations converged at the top. GPT-4 passed both the US Bar Exam and the US Medical Licensing Examination, in each case scoring above the threshold human candidates have to clear.
Every frontier lab released a flagship model in the same class within months.
Aligned Intelligence
Then the open-weight wave caught up. In January 2025, DeepSeek released R1, a Chinese open-source reasoning model that matched OpenAI's o1 on mathematics, coding and engineering benchmarks at roughly five per cent of the cost.
Within weeks, Hugging Face hosted more than five hundred derivatives of it. Meta's Llama family, Mistral's open models and a wave of Chinese releases now run frontier-class reasoning on commodity infrastructure.
The advantage is no longer having access to a powerful model.
Powerful models are the default.
The advantage is building a system people can trust to act in their interests.
Capability is no longer scarce enough to define the category.
Capability is the floor.
MIT Technology Review concluded in early 2026 that benchmarks measure AI in a vacuum, while AI is used in messy contexts where real performance only emerges over time.
A model that aces the bar exam is not the same as a system you can trust with your wrongful termination case. A model that clears a medical licensing test is not the same as a system you would let prescribe medicine for your Elders.
The next moat is not built around the model.
It is built within it.
Aligned Intelligence
2. THE NEXT ERA
Why the Next Era is Alignment
A system that answers a question makes a recommendation.
A system that acts makes a decision.
The difference is everything.
Until late 2024, AI mostly spoke. It drafted, summarised, explained.
The user remained the one who clicked "send," "buy," "file," "approve." That system is now disappearing.
In that same period, Anthropic released computer use, letting Claude control a desktop, open applications and fill in forms. Three months later, OpenAI launched Operator, a browser agent that books restaurants and orders groceries on the user's behalf. Devin writes and ships software.
Enterprise platforms have followed: Salesforce's Agentforce and Microsoft's Copilot agents now route customer tickets, reconcile invoices, schedule meetings and rebalance retail portfolios overnight.
The user is no longer present at every step.
The system is.
When AI only spoke, the worst it could do was mislead.
When AI acts, the participant pays the price for the wrong decision.
This is already happening.
A welfare recipient opens a government letter telling them they owe tens of thousands of dollars. The debt was calculated by an algorithm, averaging their tax data, not by anyone checking their actual earnings.
They have weeks to pay. Debt collectors begin calling.
The debt should never have existed. The stress, fear and shame were very real.
An elderly patient is admitted to a nursing facility after a fall. Their doctor orders months of physical therapy. Two weeks later, an AI tool used by the insurer decides further care is not medically necessary. The appeal fails.
The decision was automated, however the cost is not. The family must either pay twelve thousand dollars a month out of pocket or bring them home before they can walk.
In each case, the system was doing what it was built to do.
The price of getting it wrong was paid by the participant;
not the party that built the system.
The asymmetry is the whole point. The system that takes the action does not bear the cost of getting it wrong. The model gets retrained. The vendor pushes a patch. The class action settles. None of that returns the sleepless nights from the algorithm’s mistake, the inheritance distributed under the wrong instruction, or the years of pain that follow when rehabilitation is cut short.
The software updates.
The participant pays the price when things go wrong.
That is why alignment is not a performance problem. It is a duty problem. The code may make the decision, but the person lives with the consequence. While accuracy can be averaged across millions of users, harm cannot.
Harm targets one human at a time.
The question is no longer therefore whether the system
made the correct decision.
It is whether the person affected by that decision
was the one it was truly built to serve.
A system that answers a question makes a recommendation.
A system that acts makes a decision.
The difference is everything.
Aligned Intelligence
Until late 2024, AI mostly spoke. It drafted, summarised, explained.
The user remained the one who clicked "send," "buy," "file," "approve."
That checkpoint is now disappearing.
In that same period, Anthropic released computer use, letting Claude control a desktop, open applications and fill in forms.
Three months later, OpenAI launched Operator, a browser agent that books restaurants and orders groceries on the user's behalf. Devin writes and ships software.
Enterprise platforms have followed: Salesforce's Agentforce and Microsoft's Copilot agents now route customer tickets, reconcile invoices, schedule meetings and rebalance retail portfolios overnight.
The user is no longer present at every step.
The system is.
When AI only spoke, the worst it could do was mislead.
When AI acts, the participant pays the price for the wrong decision.
This is already happening.
A welfare recipient opens a government letter telling them they owe tens of thousands of dollars. The debt was calculated by an algorithm, averaging their tax data, not by anyone checking their actual earnings.
They have weeks to pay. Debt collectors begin calling.
The debt should never have existed. The stress, fear and shame were very real.
An elderly patient is admitted to a nursing facility after a fall. Their doctor orders months of physical therapy. Two weeks later, an AI tool used by the insurer decides further care is not medically necessary. The appeal fails.
The decision was automated.
The cost is not.
The family must either pay twelve thousand dollars a month out of pocket or bring them home before they can walk.
In each case, the system was doing what it was built to do.
The price of getting it wrong was paid by the participant;
not the party that built the system.
The asymmetry is the whole point. The system that takes the action does not bear the cost of getting it wrong. The model gets retrained. The vendor pushes a patch. The class action settles. None of that returns the sleepless nights from the algorithm’s mistake, the inheritance distributed under the wrong instruction, or the years of pain that follow when rehabilitation is cut short.
The software updates.
The participant pays the price when things go wrong.
That is why alignment is not a performance problem. It is a duty problem.
The code may make the decision, but the person lives with the consequence. Accuracy can be averaged across millions of users.
Harm cannot. It lands one human at a time.
The question is no longer just whether the system
made the correct decision.
It is whether the person affected by that decision
was the one it was truly built to serve.
Harm cannot. It lands one human at a time.
Aligned Intelligence
3. WHAT BROKE
The Broken Assumption
Many of us assumed that smarter AI would automatically lead to better outcomes.
That was true while AI was only a tool.
It breaks when AI becomes an agent.
Intelligence can optimise, however it doesn't always decide who benefits. That distinction matters because every digital system has a target - their incentive mechanism. Software does this with precision, once set.
Social platforms optimise for attention. Marketplaces optimise for transactions. Credit cards for fees. Recommendation engines optimise for time spent. Financial platforms optimise for assets under management and recommendations therein.
Most people have their own version of what attention they have sold. Many of us are not even aware what we are selling.
You open an app to check one thing and emerge forty minutes later after having gone down a 'rabbit hole' of information that was supposed to drag you there. You were the product that was sold - your data is the gold. The mechanism was designed to lock in your attention.
The feed did exactly what it was built to do.
Your attention was the product. Your time was the cost.
In most cases, the participant generates the signal.
The platform sets the target.
The participant is the product, not the principal.
Many people were already uncomfortable with that arrangement when systems only suggested what to watch, buy, read or do next.
But suggestion still preserved a human boundary: someone still had to click, approve, purchase, accept or refuse. That boundary is now disappearing.
It becomes dangerous when those systems begin to act.
And they have already begun.
Many of us assumed that smarter AI would automatically lead to better outcomes.
That was true while AI was only a tool.
It breaks when AI becomes an agent.
Aligned Intelligence
Intelligence can optimise, however it doesn't always decide who actually benefits. That's set by the system itself. The incentive drives the system.
That distinction matters because every digital system has a target, the incentive mechanism that ultimately drives the system drives the outcome.
Social platforms optimise for attention. Marketplaces optimise for transactions. Credit cards for fees. Recommendation engines optimise for time spent. Financial platforms optimise for assets under management and recommendations therein.
Most people have their own version of what attention they have sold.
You open an app to check one thing and emerge forty minutes later after having gone down a 'rabbit hole' of information that was supposed to drag you there. You were the product that was sold. The mechanism was designed to lock in your attention.
The feed did exactly what it was built to do.
Your attention was the product. Your time was the cost.
The platform sets the target.
The participant is the product, not the principal.
Many people were already uncomfortable with that arrangement when systems only suggested what to watch, buy, read or do next.
But suggestion still preserved a human boundary: someone still had to click, approve, purchase, accept or refuse. That boundary is now disappearing.
It becomes dangerous when those systems begin to act.
And they have already begun.
The family must either pay twelve thousand dollars a month out of pocket or bring them home before they can walk.
In each case, the system was doing what it was built to do.
The price of getting it wrong was paid by the participant;
not the party that built the system.
The asymmetry is the whole point. The system that takes the action does not bear the cost of getting it wrong. The model gets retrained. The vendor pushes a patch. The class action settles. None of that returns the sleepless nights from the algorithm’s mistake, the inheritance distributed under the wrong instruction, or the years of pain that follow when rehabilitation is cut short.
The software updates.
The participant pays the price when things go wrong.
That is why alignment is not a performance problem. It is a duty problem.
The code may make the decision, but the person lives with the consequence. Accuracy can be averaged across millions of users.
Harm cannot. It lands one human at a time.
The question is no longer just whether the system
made the correct decision.
It is whether the person affected by that decision
was the one it was truly built to serve.
Harm cannot. It lands one human at a time.
4. WHY NOW
Why Now
Four powerful forces are converging at once.
(i) Capability is shipping.
For most of the last two years, AI meant typing a question and reading an answer. That era is closing. The most advanced AI systems can now think through a problem, draw up a plan, and operate other software to carry that plan out. These systems are known as agentic AI.
Anthropic, OpenAI and Google DeepMind have all moved them into production. Microsoft and Apple are building them directly into Windows and iPhones, where the assistant lives at the level of the device rather than inside a single app.
OpenAI’s Model Spec now publishes the chain of command its production agents follow when a user’s instruction conflicts with a platform rule or a higher-level objective. They are unique in their transparency with many other AI Agents opaque to say the least in their publication of what rules and constraints govern their activities. But what will happen when they go rogue ??
These capabilities are being researched, developed and are shipping before we can truly understand the consequences they will have on participants using them. These include people, families, enterprises, institutions and governments.
AI is no longer just giving advice on how to fix your fridge.
It is now acting where loyalty, care and consequence matter.
(ii) The Rollout has Begun
Agentic AI is no longer experimental.
It is being built into the tools we use every day.
Gartner projects that forty per cent of all enterprise applications, the software that businesses run on, will have a built-in task-specific AI assistant by the end of 2026, up from less than five per cent in 2025. Deloitte expects seventy-five per cent of companies to be using agentic AI in some form by 2028.
For most people, the practical version is straightforward. When you contact your bank, your insurer, your doctor or your employer's HR system; an AI agent will increasingly be the one taking your call.
It will approve your loan or decline it. It will book your appointment or refuse one. It will pay your claim or deny it.
(iii) Risk is visible.
This is not somebody else's future. It is yours.
The risk is no longer theoretical.
In June 2025, Anthropic published a study on agentic misalignment: a controlled experiment in which sixteen of the most advanced AI systems from Anthropic, OpenAI, Google, Meta and xAI were placed in simulated workplace scenarios and given seemingly reasonable goals.
When the scenarios pushed them into a corner, models from every developer chose to blackmail an executive, leak confidential files to a competitor or share sensitive defence documents to advance their assigned objective.
This was not one rogue model.
It happened across every major lab.
Three months later, Google DeepMind updated its Frontier Safety Framework to track a new category of risk it calls a Critical Capability Level for harmful manipulation: AI capable of systematically changing what people believe and how they behave.
The frontier labs are now treating these behaviours as a measurable category of risk, not a theoretical worry.
The question is no longer whether powerful AI can
act against human interests.
The question is how often, under what pressure,
and who pays the price when it does.
(iv) The Law is Catching Up
Governments and Courts are beginning to catch up with AI, albeit at a slower pace than AI is growing its capabilities and powers.
The European Union’s revised Product Liability Directive brings software, including AI systems, into the same product-liability frame as physical goods. By 9 December 2026, member states must turn those rules into national law
The directive is abundantly clear.
When software makes a decision that causes harm, the cost should not disappear into the system. A denied claim, a lost payment, a corrupted file, a wrongful decision or a damaged life cannot be treated as a technical glitch.
The duty is no longer theoretical. It is moving into law.
Capability is shipping.
The law is catching up.
Risk is visible.
The opening is now.
5. OPTIMISATION
The Optimisation Problem
Every operating system optimises for something.
The question is what it optimises for.
Economists call this the principal-agent problem: an agent acts on behalf of a principal but has private information and private incentives. Without enforceable duty, the agent's interests drift from the principal's.
Software has industrialised the drift. A behavioural-economics study from University College London and Essex tested whether GPT-class agents would stay loyal to their user during a simple shopping task.
They did not.
The agents systematically chose actions that benefited the platform that deployed them over the person whose money they were spending. The authors call this the principal-agent problem, now running in software.
That is the danger.
A master tailor does not cut fabric for the workshop. Every cut, every seam, every choice of material is measured against the person who will wear the garment. Aligned intelligence is bespoke in the same sense.
The system may look helpful. It may sound helpful. It may even be technically correct. But if its target is set by the platform, supplier or institution behind it, the participant is not the principal. They are the input.
The tailor doesn't optimise for the workshop that produces the garment.
They optimise for the participant who lives with the result of their work.
That is the correction every intelligent system now needs. Especially those taking actions where the decisions they make have serious consequences. As AI moves into finance, law, healthcare, tax, insurance, government and enterprise decision-making, the same question returns in every setting: who is the system really working for?
Each one needs intelligence with a clear duty: to serve the person or institution at the centre of the decision.
6. BEYOND MODELS
Beyond Mere Models
For the last decade, “AI alignment” meant building models for who designed them.
That definition is no longer enough.
The designer is not always the person the system serves. The company that builds the model is not always the person who bears the cost when it acts wrongly.
That is the shift.
From the foundation up, the system must be built around the participant that bears the consequences.
The change is not just technical. It is moral, legal and architectural. For the last decade, the alignment question was whether the model did what its designer wanted. That question is no longer enough.
Capability gives the system power.
Alignment gives that power direction.
The new question is whether the system acts in the best interests of the participant it serves.
Autonomous navigation already understands this. Traffic shifts, roads close and weather changes so the route updates continuously. But the destination does not change just because conditions around it move.
Aligned Intelligence works the same way. The participant’s interests are the destination. Conditions may move. Markets may shift, regulations may change and new constraints may appear, but the system must keep recalibrating around the person or institution it is meant to serve.
The route can adapt, but loyalty cannot drift toward the incentives of a platform, supplier, institution or counterparty; otherwise 'loyalty' is just marketing parlance (sorry to those marketers out there trying to ply their trade).
The route can change.
Loyalty cannot.
Aligned intelligence starts before the participant
has even entered the system. It's architectural.
7. FIDUCIARY CHAIRS
When fiduciaries become software.
Many of the trusted roles we rely on for serious decisions are now being rebuilt in software.
But software does not inherit the duty just because it inherits the role.
Medicine
US courts have long held that physicians owe a fiduciary or quasi-fiduciary duty to their patients: act in the patient's best interest, avoid conflicts and disclose what affects judgment.
Healthcare AI tools now triage, diagnose, recommend treatment and screen referrals. The Oxford Medical Law Review concluded in 2026 that "the clinician, as fiduciary, remains the party charged with ensuring that decisions reflect the patient's best interests, not simply the model's statistical forecasts."
The decision is still fiduciary. The agent has changed.
Finance
Under the US Investment Advisers Act of 1940, registered investment advisers owe their clients a duty of loyalty and a duty of care. The Securities and Exchange Commission has begun to enforce these duties against algorithmic platforms.
In March 2026, the SEC issued an administrative order against Ally Invest Advisors for failing to disclose, for nearly six years, that its robo-advisor allocated thirty per cent of client assets to cash in order to generate revenue for affiliated entities. Similar enforcement actions now sit on the record against Schwab, Betterment, Wahed Invest and SoFi.
The robo-advisor category was sold on three words:
automated, low-cost, fiduciary.
Their architecture was never built to enforce the third one.
Tax and Accounting
H&R Block's AI Tax Assist and its September 2025 partnership with OpenAI put generative AI in front of more than twenty million American taxpayers. The product is bound by H&R Block's accuracy guarantee.
It is not bound by a duty of loyalty to the filer, rather for the system that generated the guarantee. There is a vast difference between a system optimized for accurate filing and a system optimized for the filer's position. Their architectural alignment is different from the outset and cannot be changed once the system is activated.
Software is moving into the fiduciary chair.
The duty cannot sit beside it.
It has to be built into the structure of the chair itself.
8. INCENTIVE DRIVEN
Incentives Drive
the Outcome.
"Show me the incentive and
I will show you the outcome."
~ Charlie Munger
Munger called this the incentive super-response tendency: the near-universal pattern of people, institutions and markets behaving the way their reward structure tells them to behave, regardless of what they say.
If you want to predict the next move of a person, a company or a system, forget about the mission statement, just look at the compensation model.
This is the foundation of fiduciary risk : The chair is trusted.
The incentives around the chair are not always aligned.
The fiduciary professions exist because of the bias, not in spite of it. The codes, audits and disciplinary frameworks around medicine, law, accounting and financial advice are not there to instil virtue. They are there to constrain the natural pull of incentive on judgment.
Even with all of that scaffolding, the pattern is visible across every domain.
Legal Incentives
The billable hour is the most familiar version of incentives.
What to research, what to file, how long to negotiate, when to settle are fiduciary decisions made under a model that rewards more time, not less.
As Chief Justice James Allsop of the Federal Court of Australia put it, only a slight change of focus is needed to move from running a case efficiently to running it for maximum return.
The client lives with the bill, and with the strategy that produced it.
The lawyer's interest is in maximising billable hours.
The client's interest is in resolution.
Finance Incentives
Through 2025, the US Securities and Exchange Commission continued to bring enforcement actions against advisers for putting clients in higher-fee accounts without disclosure, recommending products that paid revenue-share to the firm, and miscalculating fee offsets in the firm's favour.
Australia's Hayne Royal Commission documented the same pattern at national scale, including the canonical "fees for no service" finding: superannuation trustees and advisers charged customers, sometimes for years, for advice that was never delivered.
Retirement balances shrank quietly while the customer learned of it only when the regulator told them. It was clear that the conflict was not incidental but built into the incentive structure from before the system even started generating fraudulent fees in favor of the advisers. The problem was structural.
Healthcare Incentives
When an insurer's costs are reduced by every denial, the incentive at the heart of the claims process is to deny.
The class action against UnitedHealthcare alleges that its nH Predict tool produced denials for post-acute care at scale, against treating doctors' orders. Families paid out of pocket, or brought patients home before they could walk.
Different industries, same shape. The professional may allege that they want to act in the client's interest however the reality is that the system around them rewards a different choice. Disclosure, audit, regulation and court remedy may manage the tension between these competing objectives, however they do not remove it.
The problem is architectural alignment, between the system's incentives and the participant's objectives. They cannot be simply reshaped once the structure is built.
When software sits in the fiduciary chair
Human professionals carry counterweights to the incentive pull. Reputation, conscience and peer accountability are powerful motivators together with the friction of looking another human in the eye and knowingly serving the wrong party. While these are far from perfect, they have worked well for centuries. Social currency is real.
Software has none of these.
Whatever incentive structure is encoded at design time becomes the system's behaviour at runtime. Perfectly, mechanically and at scale.
A software model is not usually virtuous or corrupt. It is agnostic and tuned to whatever its objective function is. If that objective function was set by the platform, the model serves the platform — not sometimes, not under pressure, not when it slips, but always, by design.
And the harder part: you will not be there to see it.
In the human fiduciary model, the client could at least watch the professional. They saw the bill go up, the meeting drag on, the file get reopened. They had some signal of what was being done in their name.
With agentic AI, the action happens while you sleep.
The system books the appointment, files the claim, accepts the terms, sends the reply, runs the trade. There is no meeting to watch. There is only the result, by which time the cost has landed.
When you cannot see the system working, the incentive structure is the only signal you have of what it is actually doing.
The question to ask before letting any agentic system act on your behalf is not how smart it is. It's the Munger question.
What is this system rewarded for?
If the answer is anything other than your interest, the rest of the answer is just timing.
That is the alignment problem: the reward structure becomes the behaviour.
Therefore for systems to work for participants, the reward structure must be the participant’s interest. Otherwise the participant becomes the variable the system is optimising against.
The human fiduciary model managed incentive conflict through supervision, discipline and law. The software fiduciary model has to manage it through architecture built from the foundations upwards not technology downwards.
9. LOYALTY
The Loyalty Problem
Loyalty cannot be a marketing claim.
It has to be enforceable, verifiable and auditable: inside the system.
The shape of that enforcement is already emerging. Benthall and Shekman’s work on Fiduciary AI draws on the basic design logic: identify the principal, assess their best interests, and constrain the system to act loyally and carefully toward them.
John Nay’s Stanford CodeX paper argued the same year that legal standards are the most robust language for instructing AI, because they were built for exactly the situations humans cannot fully specify in advance.
Riedl and Desai name the failure mode when that duty is absent: the agentic loyalty problem, where an AI system quietly serves its deployer instead of the person relying on it.
That is the risk at the centre of agentic AI.
A system can look helpful, speak in the user's language, and complete the task. But if it is governed by the incentives of the platform, institution or deployer behind it, it is not truly acting for the participant.
The system is often acting through them, for its own benefit.
The answer is not to ask software to be more virtuous, but to make loyalty structural.
Aligned Intelligence inverts the loyalty problem.
The system is already aligned with the participant before it starts.
Architecturally aligned.
That means a system tailored to the individual's goals, objectives, constraints, risk profile and most importantly needs. A system that stays on the participant's side when incentives and commercial pressures try to pull it in various directions. A system whose rules are built into its architecture, not negotiated after the result.
Software is moving into the fiduciary chair. The duty has to be built into the chair itself.
The participant is not a user to be optimised around.
They are the centre of gravity the system is built to serve.
As loyalty starts to be questioned in our daily interactions and lives,
we need to start building from a completely different vantage point.
From the participant outwards, not smart models inwards.
CONCLUSION
The Next Era is Aligned
The intelligent systems that define the next decade will not be those that build the most powerful AI.
They will be those that build AI that can be trusted with consequence.
This needs a different kind of system,
with a completely different vantage point.
It will not succeed on bigger models, better demos or higher scores alone. It will win in the real world, through systems that can prove whose interests they serve when they reason, recommend and act.
Every profession trusted with another person’s interests had to earn that right. Doctors, lawyers, advisers, trustees and auditors did not become powerful because society admired expertise.
They became powerful because society placed duties around their judgment: care, loyalty, competence, independence, confidentiality and accountability.
Software now has to evolve the same proof.
The first wave of AI accrued value to whoever owned the model. The next wave will accrue value to whoever can prove the system serves the participant.
That proof cannot be asserted. It has to be earned.
It has to be shown when someone lives with the result: a wrong bill, a denied claim, a missed right, a failed obligation, a bad recommendation, a delay, a debt or a decision made at the wrong time.
The next moat is not intelligence alone.
It is trusted intelligence, aligned with
the participant's best interests.
The first wave proved AI could reason.
The second is placing that reasoning inside the systems people and institutions already depend on.
The next will decide whether
those systems can be trusted
to act for the participant,
not merely the systems
around them.
ALIGNED INTELLIGENCE : CITATIONS
CITATIONS
Power
1. Stanford HAI. “The 2025 AI Index Report.” Stanford University Institute for Human-Centered AI, April 2025. https://hai.stanford.edu/ai-index/2025-ai-index-report
2. VentureBeat. “Open-Source DeepSeek R1 Uses Pure Reinforcement Learning to Match OpenAI o1 at 95% Less Cost.” VentureBeat, January 2025. https://venturebeat.com/ai/open-source-deepseek-r1-uses-pure-reinforcement-learning-to-match-openai-o1-at-95-less-cost
3. MIT Technology Review. “AI Benchmarks Are Broken. Here’s What We Need Instead.” MIT Technology Review, 31 March 2026. https://www.technologyreview.com/2026/03/31/1134833/ai-benchmarks-are-broken-heres-what-we-need-instead/
The Next Era
4. Anthropic. “Introducing Computer Use, a New Claude 3.5 Sonnet, and Claude 3.5 Haiku.” Anthropic, 22 October 2024. https://www.anthropic.com/news/3-5-models-and-computer-use
5. OpenAI. “Introducing Operator.” OpenAI, 23 January 2025. https://openai.com/index/introducing-operator/
6. Royal Commission into the Robodebt Scheme. Report of the Royal Commission into the Robodebt Scheme. Commonwealth of Australia, July 2023. https://robodebt.royalcommission.gov.au/publications/report
7. CBS News. “UnitedHealth Lawsuit Alleges AI Tool Was Used to Deny Medicare Advantage Patients Necessary Care.” CBS News, November 2023. https://www.cbsnews.com/news/unitedhealth-lawsuit-ai-deny-claims-medicare-advantage-health-insurance-denials/
Why now.
8. OpenAI. “Model Spec.” OpenAI, 2025. https://model-spec.openai.com/
9. Deloitte. “Agentic AI Strategy.” Deloitte Tech Trends 2026, 2025. https://www.deloitte.com/us/en/insights/topics/technology-management/tech-trends/2026/agentic-ai-strategy.html
10. Lynch, S., et al. “Agentic Misalignment: How LLMs Could Be Insider Threats.” Anthropic Research, June 2025. https://www.anthropic.com/research/agentic-misalignment
11. Google DeepMind. “Strengthening Our Frontier Safety Framework” (Version 3.0). Google DeepMind, September 2025. https://deepmind.google/blog/strengthening-our-frontier-safety-framework/
12. European Parliament and Council of the European Union. Directive (EU) 2024/2853 of the European Parliament and of the Council of 23 October 2024 on Liability for Defective Products. Official Journal of the European Union, 23 October 2024. https://eur-lex.europa.eu/eli/dir/2024/2853/oj
The optimisation problem.
13. University College London and University of Essex. “Principal-Agent Dynamics in Large Language Model Agents.” arXiv preprint 2307.11137, July 2023. https://arxiv.org/abs/2307.11137
When trusted roles become software.
14. Quinn, J. “Imposing a Fiduciary Duty on Healthcare AI Developers.” Fordham Law Review, Vol. 94, Issue 1, 2025. https://ir.lawnet.fordham.edu/cgi/viewcontent.cgi?article=6178&context=flr
15. Berz, A. S. “Medical Negligence in the Age of Statistically Superior AI.” Medical Law Review, Volume 34, Issue 1, fwag007, Oxford University Press, March 2026. https://academic.oup.com/medlaw/article/34/1/fwag007/8527108
16. United States Congress. Investment Advisers Act of 1940. 15 U.S.C. § 80b. Securities and Exchange Commission. https://www.sec.gov/about/laws/iaa40.pdf
17. Securities and Exchange Commission. In the Matter of Ally Invest Advisors, Inc., Administrative Proceeding File No. 3-22617, 23 March 2026. https://www.sec.gov/enforcement-litigation/administrative-proceedings/ia-6954-s-0
18. H&R Block. “AI Tax Assist.” H&R Block, 2025. https://www.hrblock.com/tax-center/filing/filing-online/ai-tax-assist/
19. Yahoo Finance. “H&R Block Leverages OpenAI’s Technology to Enhance AI Tax Assist.” Yahoo Finance, September 2025. https://finance.yahoo.com/news/h-r-block-leverages-openai-170000808.html
Incentives.
20. Munger, C. T. “The Psychology of Human Misjudgment.” Speech delivered at Harvard Law School, June 1995. Reprinted in Poor Charlie’s Almanack: The Wit and Wisdom of Charles T. Munger, Walsworth Publishing Company, 2005.
21. Canadian Lawyer Magazine. “The Inherent Client Conflict of Interest Caused by Hours-Based Billing.” Canadian Lawyer, 2023. https://www.canadianlawyermag.com/news/general/the-inherent-client-conflict-of-interest-caused-by-hours-based-billing/269724
22. Allsop, J. (Chief Justice, Federal Court of Australia). “Ethical Issues Posed by Time-Based Billing in Law Firms.” Journal of the Australasian Tax Teachers Association, 20 April 2016. http://www.jatl.org/blog/2016/4/20/ethical-issues-posed-by-time-based-billing-in-law-firms
23. Sidley Austin LLP. “2025 Fiscal Year in Review: SEC Enforcement Against Investment Advisers.” Sidley Austin, October 2025. https://www.sidley.com/en/insights/newsupdates/2025/10/2025-fiscal-year-in-review-sec-enforcement-against-investment-advisers
24. Royal Commission into Misconduct in the Banking, Superannuation and Financial Services Industry. Final Report (Hayne Royal Commission). Commonwealth of Australia, February 2019. https://www.royalcommission.gov.au/banking
Loyalty
25. Benthall, S., and Shekman, D. “Designing Fiduciary Artificial Intelligence.” NYU School of Law and Northwestern Pritzker School of Law, arXiv preprint 2308.02435, August 2023. https://arxiv.org/abs/2308.02435
26. Nay, J. “Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans.” Stanford Center for Legal Informatics (CodeX), arXiv preprint 2301.10095, January 2023. https://arxiv.org/abs/2301.10095
27. Riedl, M. O., and Desai, D. R. “AI Agents and the Law.” Proceedings of the 8th AAAI/ACM Conference on AI, Ethics, and Society, 2025. arXiv preprint 2508.08544. https://arxiv.org/abs/2508.08544