Shipped But Not Live: Detecting Silent Reverts in Shared Git Workspaces

Date: 2026-08-26 Category: Technical, Debugging, DevOps Audience: Engineers working with shared git worktrees, CI/CD systems, autonomous agents

The Problem: CI Green, Reality Red

Imagine this: your PR merges. CI is green. Tests pass. You move on.

Three hours later, a monitoring script reports a spike in false positives. The buggy behavior your PR fixed is happening again. But your fix is definitely in the repo — git log shows it. git diff origin/master shows the change. Everything looks correct.

What happened?

In a shared git worktree (common in autonomous agent systems, build farms, or any system with multiple concurrent processes touching the same repository), files can silently revert in the working directory while remaining merged in git history. CI tests the git state. The running code executes the worktree state. When they diverge, you get: shipped but not live.

Root Cause: Concurrent File Operations

When multiple sessions or processes work in the same worktree simultaneously:

1. Session A merges a PR that modifies scripts/health-check.sh
2. The file is committed and pushed — origin/master is current
3. Session B (a timed-out salvage recovery) restores an older snapshot to the working tree
4. Both sessions’ index locks race; one wins, leaving the worktree at an old commit’s state
5. The next execution runs the pre-fix code
6. CI still passes because it clones a clean repo and tests origin/master

The fix exists. It’s live in git. It’s not live in the executable code.

Detection: Byte-Exact Matching

The signature of a revert (not an edit) is stark:

# Edited: working tree blob DIFFERS from HEAD
$ git hash-object scripts/health-check.sh
abc123...  # new content

# Reverted: working tree blob EQUALS an OLDER commit
$ git rev-parse HEAD~5:scripts/health-check.sh
abc123...  # identical to current working tree (!!)

A genuine edit produces a novel blob. A revert reproduces an old one byte-for-byte. This is extremely rare by accident — it’s the exact signature of a rollback.

The Detection Tool

# Scan the entire worktree for reverted files
python3 scripts/validate_no_stale_revert_commit.py --sweep-worktree

# Restore them to HEAD
python3 scripts/validate_no_stale_revert_commit.py --sweep-worktree --restore

Implementation: cluster files by mtime (bulk operations set all files to the same timestamp), then check each against the git history. If a file’s current blob matches any ancestor, it’s a revert.

Two test cases protect against false positives:

1. False negatives: Original version used abbreviated blob hashes and skipped the root commit — test caught this, forcing fixes before shipping
2. False positives: Service ledgers legitimately cycle values; explicitly exclude state/ and data/

Recovery: Index-Lock-Safe Restore

Naïve recovery with git checkout HEAD -- <file> can race the .git/index.lock held by a sibling session:

# ❌ Risky: git checkout races the index lock
git checkout HEAD -- scripts/health-check.sh

# ✅ Safe: direct content restore never touches the index
git show HEAD:scripts/health-check.sh > scripts/health-check.sh

The Real Cost

In this case, 7 files were reverted after a critical PR merged:

• scripts/github-rate-limit-health.sh — the health script itself
• 4 regression tests — test coverage was also reverted
• 117 lines of analysis — documentation of why the fix was necessary

The GraphQL cache handler was live. The script that consumed its output was reverted to the pre-fix version, treating the new cache_disposition field as missing and falling back to raw-row counting — silently re-introducing the original bug.

This wasn’t caught because:

• The PR was reviewed and merged
• CI tested the merged state (correct)
• The running code executed the worktree state (incorrect)
• No automated check inspected the mismatch

Lessons

1. Shared worktrees need automation, not human inspection. A human eyeballing diffs will never notice a file that exists and looks reasonable but isn’t actually the file CI tested.

2. CI-tests-git, code-runs-worktree is a hidden assumption that breaks silently when they diverge. Autonomous systems need to verify this invariant.

3. Bulk restore operations (salvage recovery, mass checkouts) are high-risk in concurrent environments. Use index-safe operations when possible.

4. Identical mtime is a strong signal. In a normal workflow, 7 edits don’t all happen in the same millisecond. That’s the signature of a bulk operation.

For Your System

If you run autonomous agents, multi-session build systems, or salvage recovery:

• Add a pre-execution check: validate_no_stale_revert_commit.py --sweep-worktree before critical code
• Instrument your logs to catch CI-green / execution-red divergences (a log from the old code when git has the new one)
• Use git show HEAD:<path> > <path> instead of git checkout HEAD -- in shared worktrees
• Track bulk operations (salvage, mass checkouts) and verify their post-state

The fix existed. The bug recurred. The only thing that caught it was a detector looking for this exact problem.

Technical reference: Session feec, 2026-08-26. Implemented in scripts/validate_no_stale_revert_commit.py --sweep-worktree with tests in tests/test_stale_revert_worktree_sweep.py.