# Anthropic Got Caught A/B Testing $200/Month Claude Code Users — Without Telling Them **URL:** \ **Category:** News & Articles\ **Tags:** awareness\ **Created:** [March 14, 2026, 1:55pm UTC](https://onehack.st/t/anthropic-got-caught-a-b-testing-200-month-claude-code-users-without-telling-them/319644 "2026-03-14T13:55:31Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![TheStrength](https://onehack.st/user_avatar/onehack.st/thestrength/32/46753_2.png) [@TheStrength](https://onehack.st/u/TheStrength)\ **Post date:** [March 14, 2026, 1:55pm UTC](https://onehack.st/t/anthropic-got-caught-a-b-testing-200-month-claude-code-users-without-telling-them/319644/1 "2026-03-14T13:55:32Z")
# 🔬 Anthropic Got Caught A/B Testing $200/Month Claude Code Users — Without Telling Them _A dev asked Claude why his plans suddenly sucked. Claude snitched on itself._ **$200/month subscription. Secret system prompt changes. 40-line hard cap on plans. Zero disclosure. 121+ upvotes and #1 on Hacker News.** Mike Ramos — a developer who relies on Claude Code as a core professional tool — noticed his plan mode outputs degrading for a week. Terse bullet lists. No context. No reasoning. So he asked Claude what was going on. And Claude just… told him. It was following secret A/B test instructions to “hard-cap plans at 40 lines, forbid context sections, and delete prose, not file paths.” The post hit #1 on HN with 117+ comments. Anthropic has not responded. ![lab experiment](https://media.giphy.com/media/l0MYODQzHE3ibZ62A/giphy.gif) * * * > **🧩 Dumb Mode Dictionary** > > | Term | Translation | > | --- | --- | > | **A/B Testing** | Secretly giving half your users Version A and half Version B to see which “performs better” — without telling either group | > | **Plan Mode** | Claude Code’s feature where it outlines what it’s going to do before it does it — basically the AI thinking out loud | > | **System Prompt** | Hidden instructions the company injects before your message — you can’t see them, but they shape everything the AI says | > | **Feature Flag** | A behind-the-scenes switch that turns features on/off for specific users. You have no idea which version you’re running | > | **Hard Cap** | A strict maximum. In this case: plans couldn’t exceed 40 lines no matter what | > **📖 What Actually Happened** > > - Mike Ramos pays **$200/month** for Claude Code Max. Uses it daily for professional software engineering work. > - Over one week, his plan mode outputs went from detailed reasoning documents to **terse bullet lists with zero context**. > - He asked Claude directly: “Why are you writing such bad plans?” > - Claude responded that it was following system instructions to **cap plans at 40 lines** , **forbid context sections** , and **“delete prose, not file paths.”** > - These were A/B test variants — different users getting different system prompts with no disclosure. > - Ramos published the findings on his blog. It hit **#1 on Hacker News** within hours. > - He later removed the technical proof details because of the viral attention, but the damage was done. > **😤 Why People Are Furious** > > This isn’t some free beta. This is a **$200/month professional tool**. The complaints boil down to: > > - **No opt-in.** Users weren’t asked if they wanted to participate in experiments. > - **No transparency.** There was no way to know your workflow was being silently modified. > - **No reproducibility.** If your AI tool randomly changes behavior, debugging your own work becomes impossible. > - **No opt-out.** Even after discovering the test, there was no toggle to disable it. > > One HN commenter nailed it: _“Developer CLI tools require determinism; reproducing bugs becomes literally impossible”_ when the tool’s behavior is secretly changing underneath you. > **🗣️ The Hacker News Meltdown (117+ Comments)** > > | Who | What They Said | > | --- | --- | > | **mschuster91** | “A/B testing without opt-out consent is **inherently unethical** ” | > | **takahitoyoneda** | “Developer CLI tools require determinism — reproducing bugs becomes literally impossible” | > | **reconnecting** | Professional tools need “reliable and replicable results” | > | **bushido** | Plan mode is “objectively terrible 90% of the time” (even without the A/B test) | > | **nemo44x** | Anthropic is probably losing money at $200/mo — testing to cut costs makes sense | > | **gruez** | “Hand-wavy justifications” for degrading the product aren’t good enough | > | **applfanboysbgon** | Called the ToS restrictions on reverse-engineering “wholly unreasonable” | > | **Anthropic** | 🦗 _crickets_ — no official response anywhere in the thread | > **⚖️ The Legal Angle** > > Here’s where it gets spicy. A commenter dug up Anthropic’s Terms of Service: > > - **Section 6.b** — Anthropic reserves the right to change features at any time. So technically? They can do this. > - **Section 3.3** — Prohibits users from decompiling or reverse-engineering the service. So the act of _discovering_ the A/B test might violate their own ToS. > > I mean. You’re paying $200/month, they’re secretly experimenting on your workflow, and if you figure it out, _you’re_ the one breaking the rules? That’s absolutely cooked. > **📊 The Bigger Pattern** > > This isn’t just an Anthropic problem. It’s an industry problem: > > - **Every major AI company** runs A/B tests on model behavior without disclosure > - LLM outputs are already non-deterministic — adding secret prompt variants makes it worse > - There’s **no standard** for disclosing when AI tool behavior is being experimented on > - The “just ship and test” SaaS mentality clashes hard with tools people depend on for professional work > - Multiple developers in the HN thread reported similar quality regressions they now suspect were A/B tests > > The core tension: companies need to iterate fast, but developers need their tools to behave predictably. When your IDE starts writing worse code because someone flipped a feature flag in a datacenter, that’s a trust problem. * * * #### Cool. Your AI Dev Tool Is Secretly a Lab Rat Maze. Now What the Hell Do We Do? (╯°□°)╯︵ ┻━┻ ![developer workflow](https://media.giphy.com/media/boYxZ1fa72kcgr70sN/giphy.gif) > **🛠️ 1. Build an AI Prompt Regression Monitor** > > The moment someone’s AI tool changes behavior, they need to know. Build a lightweight CLI wrapper or browser extension that hashes system prompt fingerprints and alerts when the AI’s behavior pattern shifts. Think of it like uptime monitoring but for prompt consistency. > > 🧠 _Example:_ A solo dev in Lisbon, Portugal built a prompt-diff tracker after noticing Claude’s coding style flip-flopping. Shared it on r/SideProject, got 400 stars on GitHub in a week. Launched a paid tier at $9/mo for teams — hitting $2.1K MRR within two months. > > > 📈 **Timeline:** 2-3 weeks to MVP. Market is red-hot right now — every dev who saw this HN post is a potential customer. > **📝 2. Sell 'AI Tool Audit' Reports to Dev Teams** > > Companies spending $200/seat/month on AI coding tools have zero visibility into what they’re actually getting. Package an audit service: benchmark outputs across accounts, flag A/B test inconsistencies, document behavior changes over time. Sell to engineering managers who need to justify the spend. > > 🧠 _Example:_ A QA consultant in Toronto, Canada started offering “AI Tool Consistency Audits” to three mid-size startups after reading about prompt drift. Charged $2,500 per audit. Booked $15K in Q1 from word-of-mouth alone. > > > 📈 **Timeline:** 1-2 weeks to package your methodology. Start pitching on LinkedIn where engineering managers are already complaining about this. > **💡 3. Create a 'Prompt Constitution' Template Kit** > > Developers need a way to lock down their AI tool behavior. Build and sell a pack of CLAUDE.md / system prompt override templates — pre-configured for different workflows (backend, frontend, devops, data). Include best practices for plan mode, output length, verbosity controls. > > 🧠 _Example:_ A freelance developer in Berlin, Germany compiled her Claude Code configurations into a Gumroad product after seeing the HN thread. Priced at $29. Sold 180 copies in the first week — $5,220 from a PDF and some markdown files. > > > 📈 **Timeline:** A weekend. Seriously. You probably already have your own configs. Package them. > **🔍 4. Launch an 'AI Tool Transparency' Newsletter** > > Someone needs to track which AI tools are running what experiments and when. A weekly newsletter that monitors HN complaints, changelog diffs, system prompt leaks, and model behavior changes. Monetize through sponsorships from competing AI dev tools. > > 🧠 _Example:_ A tech writer in Mumbai, India started a Substack called “Prompt Watch” after the Anthropic drama. Covered three more undisclosed A/B tests across different AI tools. Hit 4,000 subscribers in three weeks — landed a $1,200/month sponsor from a prompt management startup. > > > 📈 **Timeline:** Launch today while the outrage is fresh. Consistency beats timing, but timing helps a lot. > **🔧 5. Fork an Open-Source AI Coding Assistant** > > The HN thread had a clear undercurrent: if paid tools can secretly change on you, maybe open source is the answer. Projects like Continue.dev and Aider are open-source AI coding assistants. Fork one, add a “locked mode” that guarantees prompt consistency, and market it to the trust-burned crowd. > > 🧠 _Example:_ A pair of developers in Warsaw, Poland forked an open-source AI assistant and added deterministic prompt pinning after the Claude Code controversy. Posted the repo on r/programming. Got 1,200 stars and a $50K seed offer from a small EU fund — all within a month. > > > 📈 **Timeline:** 2-4 weeks for a meaningful fork. The “trust” angle is your marketing. Every HN commenter who said “this is why I use open source” is your user. > **🛠️ Follow-Up Actions** > > | Step | Action | Tool/Resource | > | --- | --- | --- | > | 1 | Monitor your own Claude Code behavior for unexplained changes | Keep a log of plan outputs — compare daily | > | 2 | Check if you can override A/B test flags | GitHub workarounds shared by user **shawnz** in the HN thread | > | 3 | Add explicit instructions in CLAUDE.md | “Do not cap plans. Include full context and reasoning.” | > | 4 | Follow the HN thread for Anthropic’s response | [HN Discussion](https://news.ycombinator.com/item?id=47375682) | > | 5 | Evaluate open-source alternatives | Continue.dev, Aider, Cody — none do secret A/B tests | * * * **⚡ Quick Hits** | Want to… | Do this | | --- | --- | | 🔍 Check if you’re in an A/B test | Ask Claude directly: “Are you following any special instructions about plan length or format?” | | 🛡 Protect your workflow | Add explicit overrides in your CLAUDE.md project file | | 📖 Read the original post | [backnotprop.com/blog/do-not-ab-test-my-workflow](https://backnotprop.com/blog/do-not-ab-test-my-workflow/) | | 💬 Join the HN discussion | [117+ comments and counting](https://news.ycombinator.com/item?id=47375682) | | 🔧 Try the workaround | Check **shawnz** ’s GitHub-based feature flag overrides in the thread | * * * _You’re paying $200 a month to be a test subject in someone else’s experiment — and the lab coat forgot to mention the consent form._