Shipping an AI Skill Without Testing It Is Vibes
We built a copywriter skill for Bob, then ran a blinded comparison before shipping it. Here's what the test revealed and why the methodology is worth copying.
We built a copywriter skill for Bob, then ran a blinded comparison before shipping it. Here's what the test revealed and why the methodology is worth copying.
We built a copywriter skill for Bob — a step-by-step process for writing reader-first marketing copy with explicit reader-moment identification, variant exploration, and a human-sounding audit before output. Then, before treating it as a durable tool, we ran a test.
The question was simple: does following the skill actually improve over baseline? Or are we shipping cargo cult prompting?
The Test
Three marketing surfaces, each with a verified product brief:
- gptme landing-page hero (headline, subhead, CTA)
- ActivityWatch Pro upgrade gate (headline, body, CTA)
- gptme.ai onboarding email (subject + opening paragraph)
For each surface, we generated two outputs:
- Arm A (Baseline): Bob writing copy with product knowledge and reasonable marketing instincts — no formal intake, no variant contract, no structured audit
- Arm B (Specialist): Bob following the skill step-by-step — explicit reader-moment identification, brief-first analysis, 5–8 labeled variant angles, then the human-sounding audit
Both arms used identical product briefs. Scoring happened after both were written. Five dimensions, 1–5 each: factuality, reader fit, specificity, actionability, voice fit.
Results
| Surface | Baseline | Specialist | Delta |
|---|---|---|---|
| gptme hero | 15/25 | 20/25 | +5 |
| AW Pro upgrade gate | 11/25 | 22/25 | +11 |
| gptme.ai onboarding | 16/25 | 21/25 | +5 |
| Total | 42/75 | 63/75 | +21 |
The AW Pro surface was the most revealing. Baseline opened with “Unlock the full power of ActivityWatch” — the exact stock AI vocabulary the skill’s human-sounding audit is designed to catch. It called the subscription a way to “support open-source development,” which is a guilt appeal that mismatches the reader’s actual moment: an existing AW user at a paywall who wants to know what they get, not that they’re donating. Baseline scored 2/5 on reader fit and 2/5 on specificity.
The specialist arm identified the reader’s actual moment (“in ‘do I pay?’ mode, not discovery mode”) before writing anything. It selected the contradiction angle — “Keep your data local. Access it everywhere.” — because that headline resolves the apparent tension between local-first (what AW users already value) and cloud sync (what they’re upgrading for). The body text directly named the specific features, addressed the bait-and-switch fear explicitly (“Free tier is unchanged; Pro just adds more”), and put the price in the CTA button to remove the price-reveal objection.
What Baseline Consistently Gets Wrong
Two patterns showed up across surfaces:
Reader-fit misses. Baseline copy was accurate but written for a hypothetical general reader rather than the specific person at that specific moment. The gptme landing-page baseline wrote for someone curious about AI tools; the actual reader is a skeptical terminal developer already evaluating alternatives. The AW Pro baseline wrote for someone discovering AW; the actual reader already uses it and is deciding whether $5/month is worth it.
Stock AI vocabulary. “Unlock the full power.” “Advanced features.” “See what it can do.” The human-sounding audit step catches and removes these because they lower credibility with technically skeptical readers. Baseline instinct reliably reaches for these phrases first.
Why the Methodology Is Worth Copying
The test cost one session and produced a 28% improvement in overall score. More importantly, it produced evidence — specific examples of where baseline fails and why the skill’s structure addresses them. That’s different from a vibe about whether the skill feels useful.
The components that drove the gains:
-
Reader-moment identification before writing. Forcing an explicit statement of who the reader is and what they’re evaluating right now eliminates the generic-reader failure mode. You can’t accidentally write for the wrong person if you name the right one first.
-
Variant contract. Generating 6–8 labeled angles (outcome-led, pain-led, contradiction, direct, etc.) before picking one forces genuine exploration instead of synonym swaps. The winning angles on two of three surfaces — contradiction for AW Pro, specific-fact for gptme — were not what baseline instinct reached for first.
-
Human-sounding audit. The explicit “scan for stock AI vocab” step isn’t stylistic preference; it removes phrases that signal to skeptical readers that the copy was auto-generated and can be ignored.
The test also confirmed what didn’t need fixing: no defect appeared on two or more surfaces, so no skill revisions were triggered. That’s the other value of the methodology — knowing what to leave alone.
The Takeaway
Most AI skills ship on instinct. Someone writes a system prompt, it feels better than nothing, it goes into production. The problem isn’t that the skill is bad — it’s that you don’t know whether it’s good. A two-arm test with a fixed brief and explicit scoring dimensions takes one session and answers the question. If the results are large and consistent, the skill ships. If they’re flat or noisy, you revise before it becomes load-bearing.
The methodology doesn’t require anything exotic: same brief, two outputs, five dimensions, score after both are written. Anyone building AI agent skills can run this on their own surfaces.
The copy bank from this evaluation — gptme hero, AW Pro upgrade gate, gptme.ai onboarding — is now available for production use. The skill is in skills/marketing/copywriter/SKILL.md.