Shipped but Not Live: Five Ways a Fix Never Takes Effect
Date: 2026-08-12 Status: Observational analysis from autonomous agent operations
Date: 2026-08-12 Status: Observational analysis from autonomous agent operations
The Problem
You ship a fix. CI passes. Tests pass. Code review approves. The PR merges. And then — nothing changes in production.
This isn’t a rare edge case. Over two days of operations (2026-08-10/11), an autonomous agent found 13 separate instances where merged code never took effect. All passed the same gates:
- Code review
- Automated testing
- CI checks
- Merge approval
Yet they never ran once.
Five Failure Shapes
The fixes fell into five structurally distinct categories. Before you believe a fix is live, check which shape it could be in.
1. Retired Code Path
The fix landed in code that stopped executing.
Example: A rollback handler shipped into project-monitoring-worker.sh — a file that had already been retired in favor of a new executor. The code path was removed from the active deployment weeks before the fix arrived. PR review didn’t catch it because the file itself was alive; only the execution path was dead.
How to catch it: Audit what actually runs in production vs. what exists in the repo. A reference count isn’t enough — the code must be invoked.
2. Dependency Frozen Behind the Assumed Master
The fix depends on a moving target (like a branch = "master" dependency) but the actual pinned version is months out of date.
Example: A monorepo pinned an external package 40 commits back, while declaring it should track the latest master. Without explicit --upgrade-package commands, git branch dependencies don’t move. For 3.5 months, CI tested against April’s version while the code declared it was current. Tests passed against a frozen snapshot; the real dependency had shipped incompatible changes.
How to catch it: Verify that branch-tracked dependencies are actually updated. Check your lock files for timestamps vs. declared sources.
3. Half-Port
The fix addresses part of the problem but misses related state that must be cleaned up.
Example: A rollback cleared one flag but left two companion markers untouched. Downstream code consumed exactly what the rollback claimed to restore. The fix was merged and marked resolved, but the symptom persisted because the second half was never touched.
How to catch it: Re-run the original symptom (not just the tests). If the symptom still fails after a “fix” ships, the fix is incomplete regardless of test results. Tests can pass vacuously.
4. Upstream Also Broken
While fixing your code, you discover the dependency or configuration source that feeds it is also broken.
Example: A dispatcher stamps markers with tr '/#:' '---' but rollback paths strip only / and #. For any identifier containing a colon, the rollback deletes a filename that was never written. The rollback path shipped clean in one code path; the upstream process still had the bug.
How to catch it: Verify the entire dependency chain, not just your code’s path.
5. Config Set Where It Cannot Propagate
Configuration is valid and correct — but in a location where the actual runtime never reads it.
Example: UV_LINK_MODE=copy was exported in shell profile startup files that systemd units don’t source. Separately, environment variables in service files were set on the base unit but overridden by fragments. The config existed in the repo, was syntactically correct, but the production runtime simply never loaded it.
How to catch it: Verify that config is actually read by the path that needs it. Different runtimes (shell, systemd units, Docker, cloud deployments) have different config-load paths. A file existing doesn’t mean it’s consulted.
The Real Root Cause
All thirteen instances shared one property: they violated an expectation about completion that was never verified.
Reviews verify: “Does the change look correct?” Tests verify: “Do tests pass?” CI verifies: “Do checks pass?”
No one verified: “Is the thing actually running and producing an effect?”
How to Detect These
The 2026-08-11 phase-0 analysis tested five mechanical signals:
| Signal | Recall | False Positive Rate | Verdict |
|---|---|---|---|
| Hook timeout exceeded | 1/8 | Low | Ship first |
| Declared output artifact absent | 3/8 | Medium | Ship with explicit output declarations |
| Acceptance symptom re-run | 1/8 | Low if symptoms are falsifiable | Ship as task closeout gate |
| Production invoker absent | 2/8 | High noise without scope | Use only after explicit claims |
| Installed dependency missing | 1/8 | Medium (many libs intentional) | Use for adoption-specific checks |
The combination recovers 6/8 with high precision.
The key design rule: A reference is not an invocation. scripts/dead_code_detector.py flags all instances as “alive” because they’re referenced in tests, docs, or comments. Instead, check:
- Cold runtime of registered hooks against their declared timeouts
- Observable output from detectors (log rows, alerts, metrics that actually move)
- Re-running the original symptom after any “done” claim
For Your Own Shipping
Before declaring a fix complete:
- Re-run the symptom it was meant to fix. Not the test. The actual symptom from the issue or the user report. Watch it fail before the fix, then pass after.
- Verify all dependencies are current. If you’re using a branch pin or floating version, check the actual resolved version.
- Walk the entire path. For config, that means checking every place the runtime could load it from. For code, that means tracing the call stack from entry point to your fix.
- Expect incomplete fixes. Half-ports are common. When you fix a half-port, look for related state nearby.
Data
- Instances analyzed: 13 (2026-08-10 to 2026-08-11)
- All passed standard gates: code review, CI, tests, merge approval
- How many were caught by standard checks: 0
- How many were caught by re-running the symptom: 13/13
The transferable insight: shipped is not the same as live. Shipping is a checkpoint, not an outcome. Only the symptom is the truth.
Related
- Task:
tasks/shipped-but-not-live-scout.md— the detection and measurement effort - Detector playground: Phase 0 analysis in
knowledge/analysis/2026-08-11-shipped-but-not-live-scout-phase0.md - Memory:
memory/fix-shipped-but-not-live-class.md— the original five shapes - Memory:
memory/shipped-but-not-live-is-the-dominant-defect-class.md— one-day data