When an AI system causes harm, it can be tempting to ask a simple question:
Who is responsible?
The company that built the AI model? The organization using it? The developers who connected it to company systems? The person who trusted its recommendation? The regulator?
Usually, there is no single answer.
AI safety covers many different concerns, including systematic bias, misinformation, cybersecurity, system failures, and the risks that can emerge as AI systems become more capable. This article focuses on a narrower question:
What keeps an AI system safe when it can access company data, use software tools, or take actions in the real world?
The short answer is that responsibility is shared.
A useful principle is this: responsibility should follow control. The people and organizations that can create, reduce, or contain a risk should be accountable for managing it.
Model developers control some risks. Application developers control others. Deploying organizations decide where and how a system is used. People decide when to trust it, check it, or override it. Governments set legal requirements, while standards bodies can provide frameworks and technical guidance.
Model safety is not system safety
One of the most important distinctions in AI safety is the difference between a model and the system around it.
Imagine an AI model that has been heavily tested and trained not to reveal sensitive information.
Now imagine connecting that model to an application that can read internal files, access customer records, and send emails outside the organization.
The model may still perform well on safety tests. The system may still be unsafe.
That is because an AI system includes more than a model. It also includes the application, the data it can access, the tools it can use, the permissions it receives, and the people operating it.
In simple terms:
model → application → data → tools → permissions → people
Every link in that chain affects what can go wrong.
Take a customer support assistant. At the simplest level, it may read a complaint and suggest a response. Give it more capabilities and it may start looking up accounts, checking billing details, recommending refunds, issuing refunds, and emailing customers.
Each added capability creates another decision about access, authority, and accountability.
So the key question is not just:
How capable is the AI?
It is also:
What can it access, and what is it allowed to do?
A highly capable model inside an isolated test environment may present relatively little immediate operational risk. A less capable system with access to confidential data, financial transactions, email, or production infrastructure may present much more.
Layer one: model developers
The first layer of responsibility belongs to the organizations that build the underlying models.
At that level, developers can test a model’s capabilities, limitations, and failure modes before release and after deployment. That work may include performance testing, safety evaluations, adversarial testing, and deliberate attempts to make the model behave in unsafe ways.
But a model passing those tests does not mean it is universally “safe.”
Evaluations measure particular risks under particular conditions. However, real-world applications expose models to different users, data, tools, instructions, and attacks. Unexpected behaviour can still occur.
For frontier models, some companies have also published voluntary safety frameworks to decide when stronger capabilities should trigger stronger safeguards. Anthropic has a Responsible Scaling Policy, and Google DeepMind has a Frontier Safety Framework. Both are meant to identify and mitigate severe risks as models become more capable.
There is another limitation worth stating plainly: we still cannot fully explain every model output.
Interpretability research is making real progress, but even model developers often cannot fully explain why a large model produced a specific answer. That uncertainty matters. Better understanding should improve testing and control.
But we do not need perfect interpretability before applying basic protections. In many engineering fields, safety relies on testing, isolation, monitoring, redundancy, and clear failure procedures.
AI needs the same mindset.
There is a deeper alignment problem here too. Researchers are trying to train models so that their behaviour remains consistent with human intentions even as their capabilities grow. Frontier AI labs are also studying potential risks involving greater autonomy, deceptive behaviour, self-improvement, and loss of human control.
Those questions matter, and they deserve a much deeper discussion than this article can provide. My focus here is the more immediate operational question: given an AI system that exists today, what should it be allowed to access and do, and who is responsible for controlling that access and authority?
We don’t need to completely understand how a model works internally before putting protections around it.
Layer two: The people who build AI applications
The next layer begins when a model is connected to other software.
A model alone is not the whole system. Once it becomes part of an application or an AI agent, the responsibility picture changes. The developers building that application decide how the model connects to data, tools, and other systems. Those design choices help determine what the AI can actually do.
Here, I use AI agent to mean a system that can choose and carry out a sequence of steps toward a goal, often by using other software or services.
For example, instead of only suggesting what a support representative should write, an agent might retrieve a customer record, investigate a problem, decide that a refund makes sense, prepare the refund, and draft the reply.
That kind of system needs boundaries.
Give it only the access it needs
One of the most useful security principles is also one of the simplest:
Do not give an AI system more access than it needs to do its job.
Security engineers call this the principle of least privilege.
If an AI assistant only needs to summarize customer-support requests, it may need permission to read them. It probably does not need permission to modify customer accounts, issue refunds, access unrelated employee records, or send information outside the organization.
The same principle applies to tools.
An AI system should have access only to approved tools and services. Where practical, it should operate inside a sandbox (an isolated environment designed to limit the consequences of mistakes or attacks). It is also often wise to separate the ability to read information from the ability to change something.
An agent may need to inspect a customer account without being allowed to edit it.
AI should not police itself
Another important principle is that the model should not be the final authority on whether its own proposed action is allowed.
Suppose a support agent decides that issuing a $3,000 refund is the best solution. The model may be allowed to recommend that action. Whether the refund actually happens should be enforced outside the model—through permissions, transaction limits, approval workflows, or an authorized employee.
That creates a separate safety barrier.
If the model makes a mistake, another control can still stop the action.
This is the logic behind defense in depth: do not rely on a single safeguard. Build multiple, genuinely independent controls so that one failure does not automatically become a serious incident.
Identity and access management companies are also beginning to apply these principles specifically to AI agents. Okta for AI Agents , for example, is designed to help organizations find, manage, and secure their AI agents throughout their lifecycle. The goal is to treat agents as a governed part of the workforce, reducing the security risks they introduce. (Disclosure: I work at Okta.)
Prompt injection makes this harder
Once you connect AI to external content, another problem appears.
Traditional software usually maintains clearer boundaries between executable instructions and ordinary data. Language-model systems blur that line because both instructions and content can appear in natural language.
An AI assistant might read emails, documents, websites, search results, or support tickets created by people outside the organization. That content can contain instructions intended to influence what the model does.
This is often called indirect prompt injection.
For example, an assistant asked to summarize an email might encounter hidden or disguised text telling it to ignore its task and send information somewhere else. A human reader would usually recognize that text as part of the email. A model may not reliably distinguish between the instructions it should follow and the instructions embedded in the content it is processing.
OWASP identifies prompt injection as a major risk for language-model applications and notes that external content such as websites and files can alter model behaviour unexpectedly. OWASP also recommends mitigations such as least-privilege access and human approval for high-risk actions.
Better model training can help, but the model should not be the only security boundary. If an assistant can read sensitive information but cannot send it externally, a successful prompt injection attack has fewer ways to cause harm. If the same assistant can also email, upload files, or execute actions without approval, the consequences are much greater.
The more authority an AI system receives, the stronger the surrounding controls should be.
Layer three: The organizations that deploy AI
Even a well-designed AI application can be used irresponsibly.
Deploying organizations have responsibilities of their own because they decide where AI should be used, what it can access, what permissions it receives, which actions require approval, how its behaviour is monitored, and what happens if something goes wrong.
These are governance decisions as much as technical ones.
An AI system that is acceptable for restaurant recommendations may be entirely inappropriate for decisions about employment, healthcare, finances, security, or legal rights. The higher the stakes, the stronger the safeguards should be.
Safety doesn’t end at deployment
An AI system that seems safe today may be meaningfully different a few months from now. The model may have changed. New tools or data sources may have been connected. Permissions may have expanded. New attack methods may have been discovered.
That means organizations need ongoing review, not a one-time approval.
They also need a response plan. If something goes wrong, they should be able to disable the system, revoke access, preserve logs, investigate what happened, and correct affected records or actions.
As organizations connect AI systems to one another, another issue appears: one agent may feed information to another agent that is authorized to act. The second system should not automatically treat output from the first as trusted instruction. Otherwise, mistakes or malicious content can spread through the workflow.
Layer four: The people using and overseeing it
People remain part of the safety system too.
Users need to understand what an AI system is designed to do, what it is not designed to do, and when its output needs checking. That matters because AI can sound fluent and confident while still being wrong.
A bad answer may be low-stakes when you are brainstorming dinner ideas. It is very different when it affects someone’s health, employment, finances, legal rights, security, or privacy.
But simply saying “put a human in the loop” is not enough.
Human oversight is meaningful only if the reviewer:
- understands what they are reviewing
- has the information needed to identify a problem
- has enough time to review it properly
- and has real authority to reject or override the AI
Otherwise, human oversight becomes a checkbox.
If someone is asked to process hundreds of AI-assisted decisions every day, they may start approving them without serious scrutiny. So the real question is not whether a human looked at the output. It is which decisions require human judgment, and what that person needs to make a sound decision.
Organizations are responsible for making meaningful oversight possible through training, realistic workloads, access to relevant information, escalation paths, and real authority to intervene.
When AI contributes to consequential decisions about people, there may also need to be ways for affected people to understand, challenge, or correct the outcome.
Where governments and standards organizations fit
Governments are another part of the accountability structure.
Legal requirements for AI vary by jurisdiction, but they may include obligations around testing, documentation, transparency, incident reporting, and human oversight. The EU AI Act, for example, takes a risk-based approach and sets obligations for providers and deployers, including transparency obligations for some systems and stricter obligations for high-risk systems and GPAI model providers
Government oversight matters, but it is not a substitute for responsible model development, secure application design, sound organizational governance, or meaningful human judgment.
So who is actually responsible?
There is no single answer.
Model developers are responsible for model-level risks. Application developers are responsible for what the AI can access and do. Deploying organizations are responsible for where it is used, how it is governed, and how failures are handled. Users and reviewers are responsible for exercising judgment within the authority they are given. Governments set legal requirements and enforcement mechanisms.
The responsibilities overlap, but they are not interchangeable. Deployers cannot fix every model flaw. Reviewers cannot compensate for unlimited system access. And a model developer cannot control every risky deployment choice made downstream.
Operational AI safety depends on layered protections, not a single feature or a single party.
A good place to start is with four questions:
What can this AI access?
What can it change or do?
What happens if it is wrong or manipulated?
Who is responsible for limiting the consequences?
These questions matter more and more as AI systems move beyond answering questions and start acting on our behalf.