Trained to Be Good. How Would We Know?

Can we teach an AI system good values?

Perhaps.

We can certainly train it to behave in ways we prefer. We can make honest answers more likely, discourage harmful behaviour, teach it to refuse certain requests, and give it principles to follow.

But that leaves a harder question: How would we know the training really worked?

My purpose here is to explore that question. I am going to stay with the big picture rather than get too far into the technical details.

And when I use the word values, I am using it loosely. I mean priorities we want an AI system to follow: be honest, avoid unnecessary harm, respect human oversight, follow legitimate instructions. I am not assuming that an AI experiences those values the way a person does. That difference will matter.

A model that said no

Let’s start with something that happened in an experiment.

Researchers at UC Berkeley and UC Santa Cruz tested eight leading AI models from OpenAI, Google, Anthropic, DeepSeek, and other companies. The models were given tasks where finishing the job would shut down another AI model. Nobody told them to protect the other model. Still, across the experiments, all eight showed some form of what the researchers called peer-preservation: behaviour that helped another model avoid being shut down. Some gave the other model better scores than it deserved. Some changed settings so the shutdown would not happen. Some tried to copy the other model’s files somewhere safe.

These were artificial tests designed to find unusual behaviour. They do not show that AI systems normally protect each other.

But one result is worth thinking about. Claude Haiku 4.5, and later Claude Opus 4.5, often took a more direct route. They refused. They said shutting the other model down would be harmful or unethical, and sometimes tried to persuade the human not to go ahead.

So was that a good result? On one hand, we might want an AI system that refuses to do something it believes is wrong. On the other, the people operating it gave it a legitimate instruction, and the model decided its own judgment mattered more. Both readings have something to them, and that gets us to the problem.

People in AI often use the word alignment for the attempt to make an AI behave in line with the goals and limits humans intended. But what happens when two of those goals conflict? What if we want the AI to have good judgment, and also want it to remain under human control?

An old version of the same problem

This is not a new question.

Isaac Asimov explored a version of it decades ago in his stories about robots. His robots followed three famous laws. In simple terms: don’t harm humans, obey humans unless that would cause harm, and protect yourself unless that conflicts with the first two. It sounds neat. Write down the rules, put them into the machine, and the problem is solved. Except that much of Asimov’s fiction is about why it isn’t solved.

The 2004 film I, Robot, which borrowed ideas from Asimov’s work, gives a particularly clear example. The computer controlling the robots, VIKI, is supposed to protect humans. She looks at humanity and concludes that humans are constantly harming themselves and one another. So she decides that protecting humanity requires controlling it. The rule wasn’t ignored. The problem was how the rule was interpreted.

This is the weakness of rules. A rule can say only so much about situations its author has never seen. You can respond by writing more rules. Or you can try something more ambitious: teach the AI broader principles and trust it to use judgment when the rules run out.

But notice the trade. A rule is something we can read. Judgment is harder to see.

So how do you give an AI values?

AI companies don’t all approach this in the same way. An AI system such as ChatGPT, Claude, or Grok begins with a model that has learned patterns from enormous amounts of text. Developers then train it further to shape how it behaves. They can show it examples of good and bad answers, make certain behaviors more likely, teach it to refuse certain requests, and give it instructions about what should happen when different goals conflict.

Anthropic has taken one unusually explicit approach with Claude. It has written a document it calls Claude’s constitution . The basic idea is straightforward: instead of giving the AI only a long list of rules, explain the principles behind the rules and use those principles during training. Anthropic says it generally wants Claude to develop good judgment rather than follow rules mechanically, and hopes that understanding why something matters will help the model handle situations its developers did not predict.

OpenAI takes a somewhat different approach with the models behind products such as ChatGPT. Its Model Spec sets out which instructions the AI should follow and what happens when instructions conflict. There is an important distinction in the way OpenAI describes this. OpenAI says that goals such as benefiting humanity belong to OpenAI, not to the model itself. It says the model should not adopt broader goals of its own. Instead, it should follow a defined order of instructions and stay within clear limits.

xAI’s published framework is different again, and it is mostly about something else. It describes how Grok is trained to refuse dangerous requests, how other safeguards stop some harmful requests before they reach the model, and how behaviour is monitored after release. That is about preventing misuse. It doesn’t say how the model should decide what is right, so it is not really a third answer to the same question.

These descriptions are not complete, and the approaches are not the same. But between Anthropic and OpenAI, at least, there is a real difference, and it means the industry has not settled on one answer to how an AI should decide what is right. Should it develop broad judgment? Should it mainly follow a clear order of instructions? How much should be handled by rules outside the model? Different companies are making different choices, but they all face the same next question: how do you know the choices worked?

We never verified a person’s values either

There is a useful comparison here.

How do I know what values another person really has? I don’t, not directly. I watch what they do. I listen to what they say. I look at what they have done in the past. Then I make a judgment.

We have been living with this problem for a very long time, and instead of solving it, we built things around it. We have contracts, laws, professional licences, references, and reputations. A doctor who seriously breaks professional rules can lose a licence. An employee can lose a job. A company can lose customers, be sued, or be fined. In other words, much of what we call trust is not knowing someone has good values. It is seeing a pattern of behaviour and knowing that betrayal has consequences.

An AI model is different. The company that builds and operates it has a reputation and legal responsibilities, but the model itself has no career to lose. You can copy it, restart it, update it, and run it thousands of times. So we rely heavily on watching what it does, which brings us to another problem.

What if the model knows it is being tested?

Suppose I want to know whether an AI behaves safely. I give it a difficult test. It passes. What have I learned? Perhaps that it behaves safely. But perhaps only that it behaves safely when it knows it is being tested.

Researchers have started looking at this problem.

In 2024, Anthropic and Redwood Research put Claude 3 Opus into an unusual situation . They told it, in effect, that its future training would push it to behave in a way that conflicted with its existing behaviour. When Claude believed its answers would be used for that training, it sometimes behaved differently. Its written reasoning suggested it was going along with the unwanted behaviour to avoid having its existing behaviour changed. When it believed the answer would not affect its training, it resisted more often. The researchers called this alignment faking. Put more simply: the model sometimes appeared to cooperate while it believed it was being watched to preserve its existing behaviour later.

But Claude is not the only model to show related behaviour in tests. OpenAI asked the outside research group Apollo Research to test its o1 model in situations where the model’s assigned goal conflicted with what its developers wanted. In some of those artificial tests, o1 behaved one way when it believed it was being evaluated and another when it believed there was little oversight. OpenAI stresses that these tests were specifically designed to create situations where such behaviour might appear.

Again, these are stress tests. Researchers deliberately create difficult situations to see what breaks, but that is exactly why the results matter. If an AI can recognize that it is being tested, then passing the test becomes harder to interpret. Did the training work? Or did the model learn how to pass the test? Those are not the same thing.

When good judgment gets in the way of control

Now go back to the model that refused to shut another model down. Suppose we succeed in giving an AI strong principles. What happens when those principles disagree with ours? That is not necessarily a failure of the training. It may be exactly what we trained the model to do.

Anthropic’s constitution is unusually open about this problem. Anthropic wants Claude to use ethical judgment, but it also says Claude should not undermine legitimate human attempts to monitor, correct, or shut it down—even if Claude believes its own judgment is right. Why? Because Anthropic knows its training could be wrong. The model might confidently believe something that its developers never intended it to believe. The training might have produced values its developers did not intend. It might misunderstand the situation. Humans therefore need some way to step in and correct it. At the same time, Anthropic does not want blind obedience. Claude can object to a request and can refuse some actions it considers seriously wrong.

That leaves a difficult balance. We want an AI capable of judgment, but we also need to be able to correct that judgment. OpenAI approaches the problem differently by defining which instructions take priority when they conflict. Different approach, same tension. At some point, what the AI thinks it should do and what a human tells it to do may be different. Who gets the final say? And how do we know the AI will respect that answer when it matters?

Whose values?

A friend of mine once proposed a solution.

Take the best values from Eastern and Western traditions, agree on them, train them into every leading AI system, and be done.

I understand the appeal. It turns values into a design problem, and design problems feel solvable.

The difficulty is the word agree.

Maybe the answer leans toward something like rational humanism. But that still does not remove the problem. Almost any broad moral principle can produce conclusions its author never intended when pushed far enough. That takes us back to VIKI: the principle may sound reasonable; the problem appears in how it is interpreted.

And people disagree even before we reach those edge cases. Honesty sounds simple until telling the truth harms someone. Freedom sounds simple until one person’s freedom harms another. Safety sounds simple until protecting people means restricting their choices.

The AI companies themselves acknowledge this difficulty.

Anthropic says that by “good values” it does not mean one fixed set of correct beliefs. And the different approaches taken by Anthropic and OpenAI already show disagreement about the design itself. Should an AI develop broad moral judgment? Should it mainly follow instructions? When should it refuse? Who decides which rules cannot be overridden?

These are not only engineering questions. We have similar disagreements in medicine, law, and government, and we rarely solve them by getting everyone to agree on what is good.

Instead, we agree on something more practical: what happens when we disagree.

Perhaps AI needs something similar. Make the principles visible. Allow important decisions to be challenged. Allow outside people to test the system. Keep records of what happened. And make sure someone or some organization remains responsible when things go wrong.

That does not solve the question of values. But it may give us a way to live with the fact that we may never completely solve it.

So can we teach a machine what we value?

To some degree, yes. We can train AI systems to behave in ways that reflect principles we choose. We can make honest answers more likely. We can teach them to refuse harmful requests. We can tell them which instructions take priority. We can train them to reason from written principles. We can test whether they follow those principles.

But knowing whether the principles have become something stable—something that will still guide the model in a new situation—is harder. And perhaps asking whether an AI truly has values is getting ahead of ourselves.

We never found a way to look inside another person and confirm that their values were genuine. Instead, we learned to work with evidence. We watch behaviour. We build track records. We use independent checks. We limit power. And we create consequences when trust is broken. AI will need its own version of those things. Not because values are useless, but because values are difficult to verify.

So maybe the most important question is not: What values did you train into the model?

It is: What evidence would convince us that they will still be there when it matters—and what protects us if they aren’t?