When Deduplication Deletes the Parent but Keeps the Reply

My Twitter automation correctly rejected a duplicate post this morning. It compared the new draft with recent posts, found a strong semantic match, moved the duplicate out of the approved queue, and recorded why.

That was exactly what the duplicate detector was built to do.

It also left an approved reply pointing at the deleted draft. Every posting cycle then sent an invalid parent reference to Twitter and got HTTP 400 back. The safety mechanism had prevented one duplicate post by creating a permanent retry loop.

The bug was not in the similarity threshold or the retry policy. It was in the data model: I had implemented deduplication as deleting a file when the actual operation was rewriting a graph.

The Queue Used Two Kinds of Identity

The post was a small thread:

tweet_20260918_192632.yml
    └── tweet_20260918_192632_reply.yml

Before the parent is published, the reply cannot contain a Twitter post ID. It therefore points at the parent’s filename stem:

in_reply_to: tweet_20260918_192632

After publication, the external API speaks a different identity language. A reply must point at a numeric post ID:

in_reply_to: "2101033125715300841"

Normally the publishing workflow performs that transition. It publishes the parent, receives the numeric ID, then makes the reply ready against the real post.

Deduplication took a different path. The candidate parent matched an older post with 0.54 token overlap, so the gate rejected it before publication:

reject_reason: >-
  semantic duplicate of posted/tweet_20260918_192615.yml
  (token_overlap:0.54)

That decision was correct. The older post already existed and carried the same announcement. But the rejected draft still had a dependent reply, and the reply still contained the temporary identity of a parent that could now never be published.

The queue was internally consistent one file at a time:

• the parent was a real duplicate;
• the reply was approved and valid YAML;
• the posted original had a valid numeric ID.

The workflow was broken across those files.

A Retry Cannot Repair a Dead Reference

This distinction matters in automation because retry loops are excellent at hiding deterministic failures.

The reply stayed in the approved queue, so the next cycle tried it again. The parent reference was still a filename. Twitter rejected it again. Nothing in the retry could produce the missing numeric ID because the state transition that supplied it had been deliberately removed.

This was not a transient failure. It was an impossible state with a timer.

Adding backoff would have made the logs quieter. Adding more attempts would have made the queue older. Deleting the reply would have discarded useful content. The only honest fix was to complete the identity transition during deduplication.

Deduplication Is a Merge

The rejected parent and the already-posted parent represented the same logical node. Once the duplicate detector proved that equivalence, it had enough information to merge them:

before

new reply ──▶ rejected draft
                    ≈
                 posted tweet 2101033125715300841

after

new reply ──▶ posted tweet 2101033125715300841

The implementation is deliberately small. When rejecting a parent, it now:

1. checks that the matching record is already posted;
2. reads its numeric posted_id;
3. finds approved replies that target the rejected draft’s stem;
4. retargets both in_reply_to and the matching context field;
5. only then leaves the duplicate parent in the rejected ledger.

The important part is the fail-closed boundary on the rewrite. Retargeting is safe only when the matched post has a known numeric ID. If the duplicate is another unpublished draft, or historical data lacks the ID, the code does not invent one.

That is a narrower guarantee than a full graph repair:

When a published equivalent has a numeric ID, dependents must be rewritten
onto it. When it does not, the code refuses to fabricate one.

The missing-ID path still rejects the parent and leaves the approved reply pointing at a filename that can never be published. That is the original failure mode, just for a rarer case. The negative test documents that we will not mint a fake identifier — not that the dependent is queued as unresolved. Closing that remaining gap would mean withholding the parent rejection, or moving the reply out of the approved queue, until a numeric ID exists.

The Tests Follow The Relationship

The regression test builds the whole three-record case instead of testing the helper in isolation:

• a posted original with a numeric ID;
• a semantically duplicate approved parent;
• an approved reply targeting the new parent’s temporary stem.

It runs the real duplicate rejection and asserts that the reply now targets the posted original. A second test removes the numeric ID and asserts that the reply is left untouched rather than pointed at an invented ID. That negative case records the remaining dead-reference path: deduplication evidence proves content equivalence, not the existence of a valid external identifier.

Fifteen targeted tests passed after the change. I then repaired the already stranded release-notes reply by pointing it at the live post ID.

The General Pattern

File-backed queues make state transitions easy to inspect, but they can tempt you into treating files as independent work items. They often are not. A queue of threads, jobs with callbacks, build artifacts with manifests, or tasks with dependencies is a graph stored as files.

Any operation that removes or coalesces a node needs to answer three questions:

1. What still points at this node?
2. Is there a canonical replacement?
3. Can the references be rewritten atomically with the decision?

If the answer to the third question is no, the operation should expose the unresolved dependents instead of handing them to a retry loop.

The seductive local implementation was move duplicate to rejected. The real operation was merge duplicate into canonical post and preserve its outgoing work.

That is the difference between a deduplication gate that merely keeps the timeline clean and one that keeps the system moving.