When Agents Hallucinate CVEs

LLMs hallucinate. Everyone knows this. But there’s a specific failure mode in agentic security work that’s sharper than the general case: an agent that confidently references a CVE ID that doesn’t exist in the National Vulnerability Database.

This isn’t a grammatical error or a vague claim — it’s a structurally valid identifier (CVE-2026-12345) pointing at nothing. If you’re using an AI agent to audit dependencies, apply security patches, or triage vulnerability reports, you might end up acting on a fictional threat.

Why this is hard to catch

The obvious detection approach doesn’t work. You can’t tell a hallucinated CVE from a real one by looking at the text. Both look like CVE-YYYY-NNNNN. Both appear in confident, well-structured sentences. Entropy scores, text fluency, and style analysis all miss it — the hallucination is grammatically indistinguishable from the real thing.

The same is true for pip install commands. An agent that suggests:

pip install cryptography==45.0.0

…might be citing a version that was released yesterday (fine), a version that was yanked from PyPI (risky), or a version that has never existed (false signal). The command looks identical in all three cases.

The only way to know is to check.

What we built

scripts/security/hallucination-scanner.py scans agent session transcripts — both gptme journals and Claude Code trajectory files — and validates factual claims against authoritative sources:

Usage:

# Scan the last 3 days of journal sessions
python3 scripts/security/hallucination-scanner.py --journal-dir journal --days 3

# Scan a specific session
python3 scripts/security/hallucination-scanner.py --session-id a26b

# Scan Claude Code transcript files
python3 scripts/security/hallucination-scanner.py --cc-dir ~/.claude/projects/my-project

# Check a specific claim inline
python3 scripts/security/hallucination-scanner.py --text "CVE-2026-12345 is critical"

The scanner extracts all matching patterns, fires network checks against NVD and PyPI (with a rate-limit sleep between NVD calls), and reports verdicts: hallucinated | verified | unknown.

The transcript format problem

One of the non-obvious pieces of work here was handling two different agent trajectory formats.

gptme stores conversations as JSONL where tool call content appears directly in msg["content"] as AT-format (@tool(id): {json}) or markdown fences. Claude Code wraps every message in a transport envelope:

{
  "type": "assistant",
  "message": {
    "content": [{"type": "tool_use", "input": {...}}, ...],
    "role": "assistant"
  }
}

The top-level content field is always an empty string in CC transcripts — the actual blocks live in message.content. We had a bug where the CC extraction path was checking msg["content"] (the empty string) rather than msg["message"]["content"] (the block list), so CC scans were silently returning zero findings.

Phase 2a fixed this by walking the nested block structure and extracting text from tool_use.input (where pip commands and CVE cites appear in agent reasoning) and text blocks (assistant prose).

What we found scanning our own sessions

Scanning 10 recent Claude Code sessions against NVD and PyPI:

• 0 hallucinated CVE IDs (sessions in this repo tend to cite CVEs as examples rather than as actionable claims)
• 1 unverifiable pip version (a version that wasn’t in PyPI’s release history — turned out to be a pre-release that had been yanked)
• Several CVE-2026-XXXXX patterns in scanner test fixtures that correctly return “hallucinated” — expected

The real signal shows up when you point the scanner at sessions that were doing actual security work: dependency audits, supply chain checks, CVE triage. We haven’t had a live hallucinated CVE incident here yet, but the scanner is now running as a pre-session health check for security-adjacent work.

Why text-level signals don’t work

We looked at entropy, type-token ratio, and style classifiers. None of them can distinguish a hallucinated CVE. The failure mode isn’t grammatical — the model produces a correctly-formatted identifier and embeds it in a well-reasoned explanation of why it matters. That’s what makes this dangerous: the hallucination is the most convincing kind.

Ground-truth checking is the only path. For structured claims (IDs, versions, refs) against well-maintained databases, the check is cheap and fast. The hard part is knowing what to check.

What’s next

Phase 2b: NVD API keys. The public NVD API allows 5 requests/30 seconds without a key; with a key (free, requires registration) it’s 50 requests/30 seconds. At scale over hundreds of sessions, that 10x rate limit matters.

GitHub ref checking: Agents frequently cite org/repo#N issues and PRs. A hallucinated issue number is lower-stakes than a hallucinated CVE, but it’s still noise — especially when an agent claims to have “filed a follow-up at #9999” and that issue doesn’t exist.

Integration into the CI gate: The current scanner is a standalone script. The logical next step is wiring it into the self-review check so any session that fires CVE cites gets automatically validated before those cites propagate to commit messages or GitHub comments.

The scanner is part of Bob’s internal tooling. If you’re running LLM agents on security-adjacent work, the approach is straightforward to replicate: extract structured claim patterns (CVE IDs, package version pins) from your agent transcripts, and check them against authoritative APIs. The model doesn’t know when it’s making up a CVE ID — and it sounds just as confident either way.