Governing AI in Documentation Workflows

Most conversations about AI in documentation start with a list of things to automate: drafting, summarizing, release notes, translation. That list is easy to write, and it isn’t a strategy.

A strategy answers a different set of questions. What is the automation allowed to do? How does a reviewer know it was right? What happens when the agent does not have enough information? And who is accountable for what gets published?

Those are governance questions, and they are the ones documentation teams tend to skip.

There are also two sides to the problem. We need to govern AI that produces or checks documentation. But we also need to think about the AI systems that consume that documentation. An error published today might be repeated tomorrow by a support assistant, an in-product help system, or a developer copilot.

This post focuses on the first problem, with six guardrails for making AI-assisted documentation work more trustworthy. The examples come from two workflows I built, a research agent and a documentation drift checker. The case studies are linked at the end if you want the details.

These are not every guardrail a documentation organization might need, but I think the principles apply broadly.

Why documentation needs its own governance

Plenty of AI output needs oversight, so what is different about documentation?

Readers follow it literally. A wrong step in a procedure, or a button name that doesn’t exist, costs the reader time and costs the product trust. And one common failure of generative AI is especially dangerous here: filling gaps with confident, plausible text.

Documentation also has constraints that a generic AI policy may not address. UI labels must match the product exactly. A statement can be correct for one version and wrong for another. The same applies to configurations, permissions, and user roles. Terminology needs to be consistent. And in a structured authoring system, a change that is accurate but breaks a link or file structure is still a broken change.

So the goal is not simply to “use AI safely.” It is to make AI-assisted changes inspectable, traceable to appropriate evidence, and subject to controls that match the consequences of getting them wrong.

Human review matters, but it cannot carry that responsibility alone. The workflow needs to make errors harder to introduce, easier to detect, and possible to correct.

Six guardrails

1. Make every change scoped and separable

Real automation rarely produces one change. It produces thirty. So the question is not just how small a change is, but whether a reviewer can inspect each one and make a decision about it independently.

That takes four things:

Scope: Each change touches only what is affected. A fix to one label should not rewrite the paragraph around it.
Separability: One change per decision, so a reviewer can accept one and reject another instead of approving an inseparable bundle. This does not require manually inspecting every routine change, but each proposed change should remain individually reviewable.
Evidence: The reviewer can see what the change is based on and why it was proposed.
Bounded batches: A limit on how many changes arrive together, so reviewers can give them appropriate attention. Working through a documentation set file by file, rather than in one enormous pass, is a simple way to do it.

This is what a scoped change looks like (an illustrative example, not real product text):

- Select **Add Connection**, then choose a provider.
+ Select **Add Integration**, then choose a provider.

If an agent rewrites a whole file by default, one bad change can disappear among hundreds of harmless ones, and approval becomes a formality.

Sometimes a rewrite is the right call, such as when restructuring a topic. Treat it as a deliberate request, not the default behaviour. Break the work into section-level changes where practical, explain the reasons, and show the differences against the original.

Then validate the result with checks independent of the agent. Confirm that the file structure is intact, links still resolve, and the content passes whatever checks the authoring system requires.

The reviewer should see those results alongside the proposed change. A correct sentence should not quietly break the page around it.

The trade-off: Scoped, separable changes take more careful design up front than letting an agent rewrite freely. They also create more individual decisions to manage, which makes review policies important.

2. Require a source for every factual claim, and limit which sources count

A plausible answer is not necessarily a correct one. The agent needs evidence, and the workflow needs rules for what qualifies as evidence.

Three controls matter here.

First, define the approved sources. A design specification, engineering document, or product repository may be authoritative for a particular task. A discussion thread or an old ticket may be useful context, but not sufficient evidence of what actually shipped.

Second, restrict access where possible. Telling an agent to use only approved sources is not the same as preventing it from accessing unapproved ones. If the restriction exists only in the agent’s instructions, it is a request, not an enforcement mechanism. Tool permissions and retrieval boundaries need to enforce what the agent can access.

Third, make claims traceable. Every factual claim the agent produces should link to the relevant source so a reviewer can verify it.

That changes the reviewer’s job from “does this sound right?” to “does the evidence support this claim?”

In my research agent, every finding links to its source, and anything without one is flagged. That makes it easier to distinguish verified information from questions that still need answers.

But a citation is not proof of correctness.

The source might be outdated. It might describe a planned feature rather than the released behaviour. It might apply to the wrong product version, configuration, or user role. Two authoritative sources might even contradict each other.

And retrieved content is still input, not an instruction the agent should blindly obey. A document containing instructions directed at the agent should not be allowed to override its operating rules.

The important distinction is between controlling what the agent can access, deciding which sources are authoritative, and checking whether those sources actually support its output.

The trade-off: A narrower source set may produce fewer findings, and writers still need to verify important claims against authoritative evidence. Access controls, source maintenance, and verification add work, but they make the boundaries more dependable.

3. Make “I don’t know” an acceptable output

Give the agent a way to say it cannot answer.

Call it a GAP: anything it cannot verify, cannot find, or finds contradicted across the approved sources.

It might look like this (illustrative):

GAP: Maximum number of API keys per workspace

The design spec says 10.
The Jira ticket says 25.
No engineering spec confirms either.

Question for Engineering:
Which limit ships?

A gap is not necessarily a failure. Sometimes it is the most useful thing the agent can return.

It tells the writer exactly which question to ask an engineer, and it can surface missing information that might otherwise appear late in review.

The alternative is worse. An agent that is never allowed to say it doesn’t know is encouraged to guess, and guesses in documentation look like facts.

Scott Abel makes a related point about structured content: a document can be valid DITA and still omit a fact the reader needs, leaving an answer engine to infer it incorrectly.

There is also a useful distinction between missing and conflicting information. If no source exists, the writer needs to find the right owner. If sources disagree, someone needs to resolve which one is authoritative. Neither problem should be hidden by generating a confident answer.

The trade-off: The output may be less complete than a confident draft, and unresolved questions still require people to find answers. But the uncertainty is visible rather than buried in published content.

4. Define when the agent must stop

An agent that can report uncertainty also needs to know when uncertainty makes continuing unsafe or unreliable.

Decide in advance the conditions under which it must stop proposing changes.

The test is whether it has enough trustworthy information to judge the task fairly.

Consider a documentation drift checker comparing product documentation against the current UI. Suppose the agent can inspect only a small portion of the relevant interface. Several labels referenced in the documentation are missing from what it managed to retrieve.

If it continues, it might mistake missing evidence for outdated documentation and flag accurate pages as wrong.

That is a reason to stop.

But “not enough information” needs to be more than a vague instruction. Teams should define observable conditions for their workflows, such as incomplete product coverage, missing version information, unavailable authoritative sources, or contradictions that affect the proposed change.

A stopped run should also explain itself.

It can report what was inspected, what could not be verified, why processing stopped, and what is needed to continue. The agent stops making changes, but it does not have to discard everything useful it learned.

This is where GAP reporting and stop conditions work together. A gap identifies unresolved information. A stop condition determines when that uncertainty prevents the workflow from proceeding.

A guardrail that only works when everything is going well is not a guardrail.

The trade-off: Some runs end without proposed fixes. That can be frustrating, especially when the missing information appears minor. But a useful diagnostic report is better than a collection of confident findings built on incomplete evidence.

5. Put human approval where it carries weight

Approval at the end of a process invites rubber-stamping, the pattern I wrote about in Human in the Loop, but Is Anyone Looking?.

What keeps approval meaningful is not simply having a person in the workflow. It is what that person is expected to decide, the evidence they receive, and whether they have the authority to reject the result.

Scott Abel argues that polished AI output invites trust, and people stop checking it. The same risk applies when a subject-matter expert is asked to approve confident prose.

There are three decisions worth separating.

Reviewing a change means checking whether a proposed edit is supported by evidence and appropriate for the documentation.

Approving publication means authorizing the resulting content to become available to readers.

Auditing the workflow means periodically checking whether its controls continue to work, including for routine changes that receive less individual attention.

Those decisions do not always require the same level of human involvement.

For example, a workflow might propose thirty scoped changes. Automated checks run against all thirty. A reviewer individually verifies high-impact changes and inspects a defined sample of routine, lower-risk changes. A named owner then approves or rejects the batch under an established review policy.

That is an example of a possible review model, not a universal rule. For some workflows, every change warrants individual review. For others, risk-based sampling may be appropriate once the process has been tested.

The important thing is to decide this deliberately.

Accountability needs a place to live, too. If the changes arrive as pull requests owned by named writers, the team can record who approved what, when it was approved, and how it can be reverted.

But a pull request alone does not establish responsibility. The team still needs to decide who verifies factual correctness, who has publication authority, and who owns the automation when it behaves incorrectly.

That is the difference between “a human was in the loop” and “a specific person was accountable for the decision.”

The trade-off: Meaningful review limits how quickly a workflow can scale. Sampling can reduce the workload, but introduces the possibility that an incorrect change goes unexamined. The review policy needs to reflect that risk rather than treating every change as equally safe.

6. Route findings by confidence and risk, then measure the outcomes

Not every finding deserves the same attention.

In a drift-detection workflow, some discrepancies are straightforward. A current UI label differs from the documentation, and the evidence is clear. Others are ambiguous. The documentation might be wrong, or the product information might be incomplete.

A useful starting point is to classify findings by confidence:

A, high confidence: The evidence strongly supports a specific correction. The agent may prepare a scoped change and open a pull request for review.
B, medium confidence: The evidence suggests a possible problem but is insufficient for a reliable correction. A person triages the finding.
C, low confidence: The evidence is weak or ambiguous. The finding is retained rather than automatically turned into a proposed edit.

But confidence alone is not enough.

A low-confidence finding about a minor UI label is different from a low-confidence finding about a permission setting or security procedure.

The second may deserve investigation precisely because the consequences of getting it wrong are greater.

So the routing decision should consider both confidence in the evidence and the potential impact of the problem.

High-confidence, low-impact changes may follow a routine review path. High-impact changes warrant closer verification, even when the agent appears confident. Low-confidence, high-impact findings should be escalated rather than ignored. Low-confidence, low-impact findings may be deferred, but should remain available for sampling and later evaluation.

There is another complication: the agent’s confidence can itself be wrong.

An LLM saying it is 95% confident does not mean it is correct 95% of the time. Confidence should come from the evidence, not from the model’s own estimate of how sure it is. In my drift checker, a finding rests on something a reviewer can inspect, such as a matching label or a contextual fingerprint built from an element’s icon, location, visibility, and surrounding UI. The criteria for each tier need to be documented and tested against actual outcomes.

A governed workflow should also report on itself.

Reviewer decisions tell you how accurate the findings it reported were. They do not tell you what it missed. A system can raise very few false alarms while overlooking many genuine documentation problems.

To evaluate it properly, teams need more than the number of findings accepted or rejected. They need to understand what kinds of errors the agent misses, how much time reviewers spend, and whether its performance changes as the product or workflow evolves.

One practical approach is to maintain a small, independently verified set of documentation examples. Include known discrepancies, correct content, and ambiguous cases. Periodically test the workflow against that set, especially after changing the model, prompts, or source material.

Record why reviewers reject findings, too. Recurring reasons can reveal whether the problem is poor source data, incorrect version matching, weak comparison logic, or something else.

And don’t forget the findings that were deferred. Sample them periodically to check whether important problems are being overlooked.

Without measurement, trust is just an impression.

The trade-off: Risk-based routing and evaluation require more work than assigning a confidence label. Someone must maintain the criteria, review outcomes, and decide when the workflow needs adjustment.

What the guardrails look like together

Consider a drift checker that discovers a possible UI-label mismatch. The documentation says Add Connection, while the current product appears to use Add Integration.

The agent first checks whether it has evidence from the correct product and version. If the source is authoritative and the discrepancy is clear, it proposes a scoped change, preserving the surrounding procedure.

Automated checks then confirm that the file structure and links remain valid. The change enters the review process with its supporting evidence, and a named owner decides whether to approve it.

Now suppose the agent cannot determine which product version the UI evidence belongs to. Instead of making the change, it reports a GAP.

If version information is missing across much of the material being inspected, the stop condition may prevent the entire comparison from continuing.

And if the proposed change concerns a security-sensitive procedure rather than a button label, the workflow routes it for closer review regardless of how confident the agent sounds.

This is an illustrative example, but it shows why the guardrails belong together. Scope limits the change. Sources establish the evidence. GAPs expose uncertainty. Stop conditions prevent unreliable work from continuing. Human approval establishes accountability. Measurement tells the team whether the controls are working.

Real renames are less tidy. The one my checker caught in production shared no words with its old label, so name matching alone missed it. It was found because the element kept the same icon, location, and surrounding UI.

Where it went wrong, and where it still could

Guardrails are easier to trust when you know what they failed to prevent.

Here is what went sideways in my own workflows, and what I think could still go wrong.

What went wrong

Scope crept, and trust crept out. My research agent started as a research tool. Then it picked up drafting. Then review. Each addition looked small. Together, they meant I could no longer say what its guardrails covered, and a tool you can’t describe is a tool you can’t trust.

The lesson was to define responsibilities and boundaries explicitly. Research, drafting, and approval have different permissions and different failure modes. Adding a capability should require revisiting the controls, not simply extending the instructions.

Sometimes it just stops. When an agent can’t see enough, it ends the run without proposing changes. That’s the guardrail working, and it is still annoying when it happens. A useful explanation of what prevented the run from continuing makes the interruption easier to act on.

What could still go wrong

A perfectly cited wrong answer. A citation proves where a claim came from, not that it is true. If an approved specification is stale, the agent may reproduce its error faithfully.

Reviewer fatigue. A scoped diff is easy to approve, and a queue of them is easy to skim. Even a well-designed approval process can lose its value when reviewers stop checking the evidence.

False alarms. Each false positive spends a reviewer’s attention, and a tool that raises too many gets ignored. Recording why a finding was rejected makes recurring problems easier to find and fix.

A confident mistake. Confidence-based routing can fail if the system assigns high confidence to an incorrect finding or dismisses an important problem as unlikely. Risk assessment, independent checks, and periodic sampling help reduce that possibility.

A correct label on a wrong procedure. An agent can update a name that changed. It cannot always tell that the steps surrounding the name have changed too. Tom Johnson makes a related point about release notes: AI can tell you that something was deprecated, but not necessarily why.

Errors that compound. A wrong claim early in a chain can become the premise of everything after it. When Tom Johnson had AI improve a blog post over ten rounds, the early rounds helped and the later ones made it worse. Mistakes at the start of a chain tend to travel.

None of these problems disappears simply because a person approves the output.

They are reasons to combine human judgment with independent checks, clear limits, and evidence that the workflow actually performs as intended.

The other half of the strategy: documentation as AI input

Everything above governs AI that produces or checks documentation. But there is a second reason these guardrails matter: documentation is increasingly an input to other AI systems.

An agent that relies on approved sources may treat your documentation as one of them. Support assistants, in-product help, and developer copilots can retrieve documentation to answer questions or guide users through tasks.

If the documentation is stale, ambiguous, or unowned, those systems can repeat its errors at scale.

That means the quality of the source material matters just as much as the quality of the AI workflow using it.

Scott Abel calls one version of this failure retrieval contamination: real content applied to the wrong product, version, role, or situation.

A product with a legacy and a current release invites exactly that problem. Both versions may contain correct instructions, but retrieving the wrong one can produce an incorrect answer.

This is where documentation architecture becomes part of AI governance.

Clear ownership helps determine which content is authoritative. Version labels and metadata help retrieval systems distinguish between similar but incompatible instructions. Consistent terminology makes content easier to interpret. Structured, self-contained topics reduce the need to infer missing context.

And when information changes, the documentation needs a reliable way to stay current.

These are old documentation problems. What’s new is how far an error can travel.

An ambiguous procedure once confused the person reading it. Now it may also confuse an AI assistant that passes the same ambiguity on to hundreds of users.

That changes what documentation quality work is for.

Accuracy, ownership, versioning, structure, and terminology are no longer only concerns for the human reader. They are also controls on the reliability of systems built on top of the content.

A documentation team that governs AI-assisted changes but neglects the quality of its existing content has addressed only half the problem.

Where to start

Start with one workflow, not a program.

Microsoft’s guidance on AI rollouts notes that early workflows are easier to monitor closely, while oversight becomes harder as an initiative grows.

Pick one task and make it governable before expanding its responsibilities.

Then skip “what can we automate?” and answer five questions about that task.

These are a practical checklist drawn from the guardrails above, not an established industry framework.

1. What can the agent do on its own?

Define its permitted actions, scope, evidence requirements, and boundaries. Be explicit about what it can propose, what it can change, and what it cannot publish.

2. Which sources can it trust?

Identify authoritative sources, restrict access where appropriate, and define what happens when the evidence is missing, outdated, or contradictory.

3. What makes it stop?

Name the conditions that make continuing unreliable. Define what the agent reports when it cannot proceed.

4. Who reviews and approves the result?

Decide which changes require individual review, whether routine changes may be sampled, who authorizes publication, and what review volume the team can sustain.

5. How will we know if it is working?

Measure accepted and rejected findings, missed problems, reviewer effort, and performance over time. Set conditions that trigger a pause, such as repeated factual errors, rising false alarms, or evidence that reviewers are no longer checking the work.

That last question is especially important.

Microsoft’s guidance also recommends defining the signals that should trigger a pause, so teams can address problems before they escalate.

Try answering all five questions in plain sentences for one task.

If you cannot explain the boundaries, identify the evidence, name the accountable person, or say how failure will be detected, the workflow is not ready to scale.

The point is not to eliminate every possible error. No set of guardrails can do that.

It is to make the limits of automation explicit, keep uncertainty visible, and establish a way to detect and correct mistakes before they become routine.

A faster tool is useful. A tool whose work can be trusted, questioned, and corrected is far more valuable.