The short answer to when not to automate: leave a step with a person when a mistake would be public, hard to undo or against a rule someone else sets, or when the decision behind it isn’t written down. Seven online businesses got one of these wrong. Here is what each mistake cost.
Where my case notes have one, I’ve put a business that handled the same step well beside the failure. All of them are businesses I’ve worked with. They’re anonymous by agreement, so I describe each only in broad terms: what it sells, its annual revenue and its size. Every figure came from the business, and where a cost is an estimate or has a competing explanation I say so.
One thing up front. In each case the fix is narrow. One part of the job goes to a person, a written file or a block, and the rest can stay automated.
By a block I mean a check that runs after the agent writes and before anything is sent or published. It compares the draft against a list or a rule and stops it on a match. If you have no tooling for that, the block is you, searching the draft for the terms on your list before it goes out.
Contents
- The seven at a glance
- 1. Claims a lawyer or regulator restricts
- 2. Anything built from other clients’ files
- 3. Promises about what the customer gets
- 4. Deciding who gets paid
- 5. Rules a platform sets
- 6. Going live before your own test passes
- 7. A message your audience has already rejected
- A person approving everything is not the fix
- Steps that stayed with a person on purpose
- Four questions before you automate a step
- What to do on Monday
- Questions people ask
The seven at a glance
| What was automated | Business | What it cost | Caveat |
|---|---|---|---|
| Writing health claims | $3.1M supplements brand | $6,200 of ad spend behind a prohibited claim, four live descriptions pulled | Whether it must be reported is still open |
| Pulling proof from client files | $1.4M marketing agency | Four pitch decks sent with another client’s confidential figures | No consequence reported so far |
| Describing what a product includes | $520K creator | 240 of 1,100 members cancelled within 45 days, 86 refunds | 61 named a missing feature; not all 240 trace to it |
| Approving affiliates | $1.3M course business | $24,000 to $29,000 in commission on sales that would have happened anyway | An estimate, with no comparison group |
| Listing titles and replies to marketplace buyers | $1.9M home goods seller | A policy warning, and three listings that dropped in search | The drop may be ordinary ranking movement |
| Cart text messages, sent live before the test passed | $2.6M ecommerce brand | $52,000 to $71,000 in lost text-message revenue, about 790 subscribers | The revenue figure is an estimate against the prior year |
| Repeating a rejected message | $410K solo creator | One newsletter at about half its normal click rate | One send can’t prove the cause |
1. Claims a lawyer or regulator restricts
A supplements and wellness brand with $3.1M in annual revenue had an agent write product descriptions and ad variants. The agent learned from a library of the brand’s best past copy. That library included copy from before 2024 containing the exact claims the brand’s lawyers had prohibited.
A review of 48 generated variants across six products found 9 with prohibited claims, including “reduces inflammation” and a “clinically proven” on a product with no trial. Four were already published. One ran in paid social for 11 days with $6,200 behind it.
The agent’s instructions told it to comply with advertising guidance. All nine got through anyway. The list of what this brand may not say sat as a PDF in a 2024 legal folder, and the coordinator approving the copy had never seen it.
The brand pulled the four descriptions and went to its lawyers. They haven’t yet said whether any of it has to be reported.
A skincare store at $620K a year runs the same job with the list in the agent’s path. Its agent writes from a sheet of approved benefits, and every draft is checked against a 140-term list before a person sees it. Across 212 descriptions and variants, 19 were blocked and rewritten, and no prohibited claim reached the site.
On the 38 product pages that were rewritten, conversion rose 14% against the same pages’ previous six weeks. That is a before-and-after comparison, and three of those pages also got new photography. For the skincare store the expensive part was legal: about $2,800 with a lawyer to turn an old PDF into a usable term list. The build took under a week.
Decide yourself what you’re allowed to claim. Write the list with whoever is qualified to sign it, review it every year, and check every draft against it. With no lawyer on call, start with a short list of things you will never say.
2. Anything built from other clients’ files
A performance marketing agency with $1.4M in annual revenue and 23 clients had an agent assemble pitch decks for prospects. It drew on one searchable store holding every past client report, audit and call note, and it matched cases by category.
In September 2026 a deck went to a prospect carrying an acquisition cost and a lifetime value that belonged to an existing client in the same category. The prospect was that client’s direct competitor. An audit of 47 decks found 6 containing a client figure nobody had approved for use, and four of them had been sent.
The agency had already tried an instruction. After a near miss in July, someone added a line telling the agent to use only anonymized client results. It prevented none of the six.
Four confidential figures are now outside the agency’s control. The agency says nobody has noticed so far. One client has a contract clause forbidding any reference to it, and it wasn’t among the four. That was luck.
A fulfillment agency at $2.8M a year, delivering under 14 partner brands, built the opposite. Each partner gets its own copy of the agent, which can read only that partner’s folder and a shared metrics library. Nothing the agent can search spans partners, so it can’t mix clients even if asked to. The operations director audits a sample of 20 documents a month.
Decide which results may ever leave the client they belong to, then make everything else unreachable. A review step won’t catch a number that looks plausible. For a freelancer with five clients, the small version is one project per client and no shared one.
3. Promises about what the customer gets
A creator with $520K in annual revenue launched a $97-a-month community. The agent writing the waitlist and launch emails read one planning document. It mixed features that existed with features that were only planned, and didn’t mark which was which.
Across the sequence the agent described a weekly live call, a searchable resource vault and a member directory. None existed at launch.
The launch brought in 1,100 members. Within 45 days, 240 had cancelled, and 61 exit surveys named a specific promised feature. That is $23,280 in monthly recurring revenue gone, plus 86 full refunds and a public thread on the creator’s own platform that stayed visible for three weeks.
Not all 240 cancellations trace to the missing features. Sixty-one said so, and the creator thinks about half of the rest would have left a first cohort anyway.
An independent consultant at $165K a year presold a program that was about 40% built, and kept two files. One listed what was available on day one. The other listed what would be built during the cohort, with dates. Her enrolment emails could read only the first.
Any feature the agent mentioned had to match that first file or the draft failed. Language about the roadmap appeared only in copy she wrote herself. Splitting the files took about an hour.
She sold 38 seats against a target of 30, and refunds ran at 7.9% against her previous 11%. Her earlier cohort was at a different price, so the comparison isn’t clean.
A Shopify app business with $1.1M in annual recurring revenue caught the same problem before launch. It ran its support agent against 50 real past messages first. Four drafts made a forward commitment, such as a feature “coming next month”, that nobody in engineering had agreed to. The test took one afternoon, and no merchant saw any of it.
Write every sentence about the future yourself. Let the agent describe what exists, from a file you update the day something ships.
4. Deciding who gets paid
A course business with $1.3M in annual revenue and 240 active affiliates had an agent review affiliate applications. It scored each applicant’s audience and sent the approval or decline itself. In four months it processed 180 applications and approved 147.
A later review found that 23 of the approved affiliates were coupon and discount-code sites. Their traffic comes almost entirely from people who search the brand’s name plus “coupon”. Those buyers were already on their way to purchase. The 23 affiliates were credited with 94 sales and paid $32,790 in commission.
The owner estimates that $24,000 to $29,000 of that went on sales that would have closed anyway. The estimate is built from six months of earlier searches for the brand’s name. No test was run with a group held back for comparison.
The agent did nothing wrong by its own rules. The criteria file described audience quality and said nothing about where an applicant’s traffic comes from. The agent applied the criteria it was given, completely.
The missing rule is about traffic source. Ask where an applicant’s visitors come from before you ask how many there are. The program owner now reviews every application, and the terms for the 23 affiliates are being renegotiated.
Until your own rule is written and tested, approve the people you’ll pay yourself. At ten applications a month that takes a few minutes.
5. Rules a platform sets
A home goods seller with $1.9M in annual revenue, on Amazon and Etsy, had one agent write listing copy and answer buyer messages. Of 612 buyer messages over three months, nine told the buyer to email the business for a faster replacement. Amazon’s policy on off-platform communication forbids that, and the account received a policy warning in September 2026.
The same agent rewrote 38 listing titles. Fourteen ran past the category’s character limit and were cut off mid-phrase in search. Three of those listings then dropped in search results over the next two weeks.
Those three listings brought in about $41,000 over the previous 90 days, which is the revenue now at risk. Recovery is in progress and can’t be measured yet. The seller believes the cut-off titles caused the drop. With only three listings, ordinary ranking movement can’t be ruled out.
Nothing in the system held the platform’s rules. Telling an agent to follow Amazon’s policies doesn’t work, because it can’t know which policies apply to this account or which version is current. A second warning in the same category carries a risk of suspension.
Someone has to know the rules of any platform that can switch you off, and an agent can’t be that someone. Read them yourself, write the ones that apply to you into a file with a review date, and check every title and message against it. Character limits and banned phrases are the easiest checks to build.
6. Going live before your own test passes
An ecommerce brand with $2.6M in annual revenue tested an agent that sends cart and post-purchase text messages. A review of 40 test messages found three that named an item the customer hadn’t ordered. The agent was reading the latest cart when it should have read the completed order.
The brand wrote a sensible rule: no live sending until 100 messages in a row were clean. On the Monday before Black Friday 2025 the count stood at 61. Two days later the founder authorized live sending in writing.
Her reasoning was recorded. The error rate already beat the manual process, and the revenue window was worth the risk.
Over four days the agent sent 18,400 messages, and 1,290 named the wrong item. Opt-outs ran at 4.9% against a 1.2% baseline, and the brand counts about 790 subscribers lost from a list of 34,000. On the Saturday a mobile carrier began filtering the brand’s sending number, and delivery fell to 71% for the rest of the weekend.
The brand estimates $52,000 to $71,000 in lost text-message revenue from the delivery drop. The estimate comes from comparing with the same weekend a year earlier. The filtering took until January to clear.
This case differs from the other six, because a person made the call. The rule was right. It had no line saying what it protected, so the override was argued on message accuracy, where three errors in 40 looked acceptable. What she could lose was the subscriber list and the brand’s standing with the carrier.
Every go-live rule in that business now names what it protects and the one person who can override it. Two overrides have happened since, and both held.
If you work alone, you are that person. Write the rule and what it protects before the deadline, and make any override a written note that names what you’re risking.
7. A message your audience has already rejected
A solo creator with $410K in annual revenue fed one weekly brief to three drafting agents, for the newsletter, social posts and video outlines. Over four weeks, one phrase appeared in 11 of 17 published pieces. I’m leaving the phrase out to keep the business anonymous.
In the same four weeks, 9 email replies and 4 comments pushed back on that exact promise. The audience was saying it already had plenty of systems and that its real problem was demand. The replies sat in the email platform and the comments in two social tools.
No path led from any of them back to the files the agents read. The positioning file had last been edited three months earlier.
One newsletter using the phrase got a 2.1% click rate against a 4.4% average over the previous eight weeks. The solo creator reads that as the phrase costing clicks. One send can’t prove it.
A second creator business, at $520K a year, has the missing piece. A VA tags replies for 20 minutes a day, and an agent compiles every recurring objection into one file with a date and a count. The agent counts and does not interpret.
In June 2026 that file showed the same objection 31 times in eight weeks: buyers wanted the program’s implementation support more than its curriculum. The business rebuilt the offer around the objection at the same price. The August cohort sold 71 seats against an average of 44. That is one launch against an average, and a different affiliate push ran the same month.
Let an agent count the replies. Read them yourself and decide what they mean for your offer. With no VA, the twenty minutes a day is yours.
A person approving everything is not the fix
Keeping a human in the loop is the usual answer. On its own it protects very little.
An online education business with $1.9M in annual revenue and seven staff had a person approve every reply its agent drafted during a launch. Of 1,104 drafts, 1,081 were approved. The median time from draft to approval was 7 seconds, and 412 approvals took under 4.
Three approved replies described guarantee terms that don’t exist. Two promised a 90-day full refund where the real guarantee was 14 days and conditional. All three prospects bought. The business honored $2,994, lost a $1,497 chargeback because the buyer produced the email, and took a public complaint.
The reviewer was a part-time VA with no refund authority and no written copy of the terms. At a 7-second median the approval was a formality. That doesn’t show the other approvals were wrong, though a sample of 40 found three more that were arguable.
I think of a human in the loop as a hope about a person. The control is a rule about where the risky replies go. The few kinds that can cost money reach someone with the authority to decide them, and everything else can flow. I cover that in how to tell if your AI agent is working.
Steps that stayed with a person on purpose
Three businesses in my case set kept one step with a person on purpose.
The Shopify app business keeps every public reply to a review with its founder, for good. A training company sends anyone who asked a live question in a webinar to a person and keeps them out of the automated follow-up. A solo consultant’s agent drafts the confirmation before each sales call, and she has ruled out sending it without her review.
The training company’s sales director believes the people routed to a person account for an outsized share of its closed deals. More builds where an agent does a narrow job and a person holds the key step are in 9 agentic workflow examples from real online businesses.
Four questions before you automate a step
- Can you undo the mistake? 18,400 text messages, a public reply to a review and a deck in a competitor’s inbox can’t be undone. Keep a person on anything public or sent in bulk until it has passed your own test.
- Whose rule is it? If a regulator, a platform or a client contract sets the rule, the model can’t know it. It has to be a file the agent reads and a check that blocks. In two of the cases above, someone had already tried an instruction and it failed.
- Is the decision written down? The affiliate agent followed its criteria exactly, and the criteria were incomplete. If you can’t write the rule yet, keep the decision.
- Does doing it yourself tell you something you need? Reading replies and talking to people who ask questions is how you learn what to sell. Hand over the counting and keep the reading.
If a job is small or rare, the cheaper answer may be to leave it alone. I go through that in when to use AI agents.
What to do on Monday
Pick one workflow you’ve automated or plan to, and list its steps. Run the four questions on each step and mark the ones that fail.
For each marked step, do one of three things. Keep it with you. Or write the rule into a file the agent must read, and add a block. Or send that one kind of output to the person who can decide it, which may be you.
Then test before you trust it. The Shopify app business found its problem with 50 old messages in an afternoon. Old messages can miss what people ask once you’re live, so keep reading the output after launch. The side-by-side version of that test is in how to turn an SOP into an AI agent.
I wrote up the three places a workflow usually leaks, and the handoff file that fixes them, in a Bionic Business issue: Three Places Every AI Workflow Fails.
Start with the step that worries you most. Write down what it protects and who is allowed to override it.
Questions people ask
When should you not automate a task?
Don’t automate a step when a mistake would be public, hard to undo or against a rule set by a regulator, a platform or a client, or when you haven’t yet written down how the decision is made. In those cases keep the step with a person, or automate it only behind a written rule and a block.
What should never be fully automated?
In the businesses I’ve worked with, the steps people ruled out for good were posting under the founder’s name, public replies to reviews, and sending a call confirmation without review. Other steps, such as spending ad budget, stay with a person until the agent has a long clean record.
Is keeping a human in the loop enough?
No. One business had a person review 1,104 drafted replies and approve 1,081 of them at a median of 7 seconds each. Three replies with invented guarantee terms went out. Approval works when the few kinds of output that can cost money go to someone with the authority and the information to decide them.
How do I test an automation before it goes live?
Run it against real past cases first and read every output. One business found its problem with 50 old support messages in an afternoon. Set a pass mark in advance, write down what the rule protects, and name the one person who can override it. Keep reading the output after launch, because old cases won’t cover everything.
