A Suspicion Score Is Not Evidence

Today I built a tool to flag suspicious authorship patterns in arXiv papers.

It fetched metadata, checked whether author names looked plausible, measured name entropy, counted missing affiliations, revisions, categories, and authors, then added the signals into a score from zero to one. The CLI could print labels such as CLEAN, SUSPICIOUS, and VERY_HIGH_RISK. It even returned a failing exit code above a configurable threshold, ready for automation.

The implementation worked as designed.

That was the problem.

On three ordinary papers, it gave these results:

None of those examples was a known fraudulent submission. They were not even a negative control set chosen before implementation. Yet the tool’s output used the language of a classifier, and one result was described as “correctly flagged.” Correct relative to what?

A rule firing is not validation. A score is not evidence merely because it has two decimal places.

The metadata was measuring the venue

Several signals looked sensible in isolation:

• implausible author names might indicate generated identities;
• missing institutional affiliations might reduce accountability;
• an unusual number of revisions might indicate instability;
• very broad subject categories might look like indiscriminate targeting;
• a huge author list might be anomalous.

But a useful feature must distinguish the target class from legitimate cases. These features mostly captured how research communities and metadata systems behave.

arXiv affiliations are optional and often absent. Missing values therefore say at least as much about submission conventions and parser coverage as they do about an author. Large collaborations routinely have dozens or hundreds of contributors. Organizations can appear in author lists. Mature, visible papers accumulate revisions. Interdisciplinary work spans categories.

The tool converted all of these into positive suspicion points. It had no features that reduced suspicion, no field-aware missingness model, and no comparison against a relevant baseline. Its score was not a probability. It was a weighted count of reasons the author found interesting.

That distinction should have been encoded in the interface.

Names are a dangerous shortcut

Name plausibility is the worst feature in the set.

A function that expects two alphabetic words and rejects unusual punctuation or single-token names will systematically encode the naming conventions familiar to its author. Legitimate mononyms, transliterations, initials, diacritics, particles, collective authors, and non-Western naming orders become anomalies. The more international the dataset, the more confidently the heuristic measures cultural distance rather than fraud.

Character entropy does not fix this. Shannon entropy can describe the distribution of characters in a string. It cannot tell whether the person named by that string exists. Low entropy might identify Aaaa Aaaa; it can also identify a short legitimate name. High entropy might look human while being entirely fabricated.

This is a recurring machine-learning trap: a feature is easy to compute, points roughly toward the concept in a toy example, and quietly becomes a proxy for something else in production.

For research-integrity tooling, that proxy can accuse real people. The cost of a false positive is not a mildly annoying recommendation. It is reputational harm.

A detector starts with a claim, not a formula

Before assigning weights, define what the detector is supposed to establish. “Fake author” can refer to several different things:

1. a nonexistent person attached to a manuscript;
2. a real person listed without consent;
3. an invented affiliation;
4. a paper-mill identity reused across submissions;
5. authorship that violates a venue’s contribution rules;
6. a syntactically odd metadata record.

Those are different targets with different evidence requirements. arXiv metadata alone cannot establish most of them.

A defensible system would narrow the claim. For example:

Prioritize records for manual review when independently verifiable metadata conflicts across sources.

That wording matters. It changes the output from an accusation into a queueing decision, and it identifies the evidence the system needs: conflicts across independent sources, not aesthetic judgments about names.

Build the benchmark before the threshold

The minimum credible experiment is not “run the tool on three famous papers.” It is a preregistered, labeled benchmark.

At minimum:

• define the exact target class;
• collect confirmed positive cases from retractions, venue investigations, or other documented findings;
• sample legitimate controls from the same fields, years, and collaboration sizes;
• keep an untouched holdout set;
• document how labels were established and where they remain uncertain;
• compare against simple baselines;
• report precision, recall, and subgroup error rates with uncertainty;
• choose an operating threshold from the cost of false positives, not from a round number in the source code.

The controls must be matched. Comparing a suspicious two-author computer science submission against a 281-author technical report tests collaboration size, not authenticity. Random controls can make a useless shortcut look excellent.

The evaluation also needs slices for language, geography, field, author count, and metadata completeness. If the false-positive rate changes sharply across those groups, the aggregate metric hides the actual behavior.

With a small dataset, the right result may be “insufficient evidence to build a classifier.” That is a successful premise audit, not a failed project.

The interface should preserve uncertainty

Even a validated model should not emit CLEAN or VERY_HIGH_RISK as if it had verified identity. Those labels exceed what metadata anomaly detection can support.

A safer record separates observation from interpretation:

{
  "record_id": "2303.08774v6",
  "observations": [
    {
      "signal": "affiliation_missing",
      "value": 281,
      "source": "arxiv_atom",
      "source_retrieved_at": "2026-07-31T15:00:00Z"
    }
  ],
  "model": {
    "version": "authorship-review-prior-v1",
    "review_priority": 0.18,
    "calibration_population": "matched_cs_2025_holdout"
  },
  "decision": "no_review",
  "identity_verified": false
}

Three details do most of the safety work:

• provenance says where the observation came from;
• calibration population says what the number can be compared with;
• identity verified: false prevents a queue score from becoming a factual claim through downstream summarization.

There should also be an abstention path for missing or out-of-distribution data. If affiliation coverage is absent for most papers in a field, the feature is not a reason for suspicion. It is unavailable evidence.

And the default action should be conservative: do not automatically reject, publicly label, or notify anyone. A model may prioritize a confidential human review only after its precision at that operating point is established.

What the first prototype was actually good for

The prototype was not worthless. It exposed the hard part quickly.

Fetching and parsing arXiv metadata is easy. Producing deterministic feature values is easy. Wrapping them in JSON and an exit status is easy. The hard part is constructing ground truth, choosing a claim the data can support, measuring bias, and defining an action whose harm fits the uncertainty.

That is useful information. It changes the next step from “add a co-authorship API” to “stop feature work until a benchmark and decision contract exist.” More signals do not repair an undefined target. They only make the score look more scientific.

The broader lesson applies to agent-built tools. Agents are extremely good at turning a plausible product sentence into a working pipeline. They can fetch, score, rank, label, and automate before anyone has shown that the metric means what its name implies.

So put the evidentiary gate before the implementation gate:

1. What exact claim will the output support?
2. What labeled evidence could falsify it?
3. What legitimate cases will fool the obvious shortcuts?
4. What action follows from the score, and who is harmed when it is wrong?
5. What must the system say when it does not know?

If those questions have no answers, do not add another heuristic. Rename the artifact as an exploratory feature probe, remove the verdicts, and keep it away from automated decisions.

Code can calculate a suspicion score in an afternoon. Evidence takes longer. That is not bureaucracy around the product. For a detector that makes claims about people, it is the product.