Automated Documentation Drift Detection

Overview

Product documentation can drift out of sync without anything breaking. A button gets renamed, a feature ships without documentation, or an older topic quietly becomes inaccurate. I designed and built a Claude Code workflow that detects these gaps and proposes targeted fixes for review, with every change under human control. In its initial audit, it scanned the product documentation and surfaced a substantial set of potential gaps for review. The next step is to turn it into an agent that runs on a schedule and routes findings to the appropriate writer.

My role

I owned the project from problem definition through implementation and rollout.

• Designed the detection model. Created two modes: Audit identifies potentially stale or missing documentation; Drift detects product changes such as renamed UI elements.
• Built the workflow. Implemented the skill, human approval points, source-coverage checks, minimal-edit behaviour, and post-change validation.
• Proved the approach. Tested it against a real production rename and refined the matching logic when simple name comparison wasn’t enough.
• Planned the next step. Scoped turning the skill into an agent that runs on a schedule, opens PRs for review, and produces a monthly digest.

Worked with

• Writers. Use and maintain the workflow across all products.
• Managers. Sponsored the pilot.

The challenge

Documentation can be technically valid and still be wrong.

If Engineering renames a button, the documentation continues to build successfully even though it now directs customers to something that no longer exists. Older content can age quietly, while new product capabilities can ship without corresponding documentation.

Traditional build and link checks catch structural problems. They don’t necessarily catch semantic drift between the product and the documentation.

I wanted a way to detect that drift without giving an AI system permission to rewrite documentation on its own.

Key decisions

1. Change only what is out of date. The workflow edits only the affected text rather than rewriting an entire topic. This keeps the proposed change easy to inspect and reduces the chance of unrelated content changing. Trade-off: The workflow requires more precise matching and editing logic.
2. Detect renames by what stays the same. A renamed UI element may share no words with its previous name. Instead of relying only on text matching, the workflow uses a contextual fingerprint based on characteristics that can survive a rename, such as its icon, location, visibility, and surrounding UI. This allowed it to identify a real rename that name matching alone missed. Trade-off: Product areas require additional configuration before this matching becomes reliable.
3. Fail closed when source coverage is incomplete. The workflow first checks whether the product source it can inspect appears sufficiently complete. If commonly documented UI labels are missing from that source, the run stops instead of assuming the documentation is stale. This prevents incomplete product data from generating misleading findings. Trade-off: Some runs stop without producing results.
4. Keep people in control. The workflow can detect, compare, and propose. It cannot publish. A writer remains responsible for approving the scope and reviewing the proposed documentation change before it is merged. Trade-off: The workflow is intentionally slower than a fully autonomous system, but editorial responsibility remains with the documentation team.

My approach

I started with one product area and one failure mode rather than trying to solve documentation drift across the entire product at once.

1. Discover. Identify a reliable source of product UI information.
2. Build. Create and test the detection workflow.
3. Prove. Detect and fix a real UI rename end to end.
4. Scale. Next, turn the validated workflow into an agent that runs on a schedule.

The solution

The workflow has two detection modes.

Audit mode compares product UI information with existing documentation to surface possible gaps in either direction: documented UI that may no longer exist, and product UI that may not be documented.

Drift mode compares two versions of the product UI. When it detects a likely rename, it searches the documentation for the previous label and proposes a minimal replacement.

The overall flow is: product source → detect change → locate affected documentation → propose minimal fix → writer review → validation → pull request. After a proposed edit, automated checks confirm that the document structure remains intact, links still resolve, and the file passes validation.

From skill to agent: the next step is to turn the skill into an agent that runs on a schedule. When it detects documentation drift, it will prepare a pull request for the appropriate writer to review, and a monthly digest will summarize activity and findings. Nothing publishes without human review.

Tools: Claude Code · DITA/XML · Git · GitHub · internal AI-agent platform (planned)

Impact

• Result: The project changed documentation drift from a problem that depended on someone noticing it into something that can be checked systematically.
• Coverage: The initial audit scanned the product documentation and surfaced a substantial set of potential gaps for investigation.
• Detection: The workflow identified a real production UI rename using contextual matching that simple name comparison had missed.
• Next step: Turn the manually initiated skill into an agent that runs on a schedule, with a monthly digest for ongoing visibility.
• Reviewability: Proposed fixes are intentionally small, allowing writers to see exactly what changed rather than reviewing an AI-generated rewrite of an entire topic.
• Adoption: All writers use the skill, and it covers every product the team documents, with each product area onboarded and configured individually.

What I learned

Modular workflows are easier to trust. I found that a skill is easier to test, understand, and maintain when it has one clear responsibility, so I handled related tasks with smaller skills that work together instead of one increasingly complex workflow.

More context isn’t always better. My first design made one skill responsible for too much. As its instructions and responsibilities grew, the workflow became harder to reason about and maintain. Breaking the problem into smaller components made the behaviour clearer and the system easier to evolve.

Scaling required product-specific design. Scaling this approach wasn’t a matter of turning it on everywhere. Each product has different repositories, architectures, and sources of truth, so I onboarded product areas deliberately, one at a time, with configuration and validation appropriate to each environment.