Dry run it against last month
Your own history is the test set
Before an automation sends anything to a customer, point it at the last month of work your people already did and compare what it produces against what they actually sent. That comparison is the test. Nothing else you can do before launch tells you as much, because your own history is the only sample that has your customers, your file formats, and your exceptions in it.
The fleet behind this firm has logged 13,297 runs since it started keeping count on March 19, 2026, and 2,339 of those came in the last thirty days across 192 jobs. None of those jobs went onto a schedule without first being run by hand against work that had already happened. A job that fires at 3am has nobody standing next to it to catch the first bad output.
Read the mismatches
What a dry run gives you is a pile of pairs: what the machine produced, and what a person produced for that same input a few weeks ago. Read forty of those pairs and the mismatches sort into three kinds.
Wrong is the easy one. The machine pulled the wrong number off the invoice, addressed the wrong contact, quoted a lead time from the wrong column. You spot it on sight, and the cause is almost always an instruction nobody wrote down rather than a broken tool.
Different is harder, and more common. The machine's version is defensible and it is not what your office would have sent. Other wording, other order, an attachment your team stopped including last year. Some of these are worth correcting and some are the machine being fine while you are being particular. Reading them is the only way to tell which is which.
Nothing is the kind people miss. The machine produced no output at all for an input a person clearly handled. A blank row is quiet and easy to scroll past, and it is the exact failure that shows up two months later as a customer who never heard back. Count the blanks before you read anything else.
Four rules that keep a dry run honest
A dry run is only worth the afternoon if it is the real thing with the last step removed. Four rules keep it that way.
- Turn off sending in the code, not in the instructions. The switch that decides whether an email leaves the building belongs in the outermost layer, checked immediately before the send. Telling a model not to send is a preference. A flag is a fact.
- Feed it the ugly inputs. Use the actual month, including the phone photo of a scribbled work order and the forwarded thread with nine replies buried in it. A tidied-up sample proves nothing you needed to know.
- Write every result to a dated file that you open. If the output of the dry run only ever existed in a terminal window, the test did not happen.
- Score the first batch by hand. Forty items read carefully beats four hundred skimmed, and the reading is where you find the missing instruction.
Where a replay lies to you
A replay is a decent test and a bad guarantee. The gap between those is worth saying out loud before anyone gets confident.
Your history only contains what somebody handled. The lead that came in on Christmas Eve and got lost is not in the file, so the machine never gets tested against it. Neither is the task your team stopped doing in March without telling anyone.
History is seasonal, too. A month of invoices contains no year-end close, no annual renewal, no shutdown week. If the work has a yearly shape, replay a month from the busy part of the year instead of the convenient one.
And formats move. A supplier who sends spreadsheets in August may send PDFs in November, which is the ordinary way a working automation stops working. A replay proves the machine handles a normal month. It says nothing about the strange week, which is why the watching still has to exist afterward, the way why automations fail silently lays out.
What a clean run actually earns
A clean replay does not earn autonomy. It earns a live trial with a person on the send button. Same job, same schedule, real incoming work, output landing in a draft folder instead of a customer's inbox. A couple of weeks of that with no edits is the evidence that graduates a task past the approval gate.
If you are hiring someone to build this, ask what they intend to replay it against and how many items they will read before they call it working. An answer built on your own last month is a good sign. An answer about test cases they wrote themselves means the thing has only ever been tested against their idea of your business.
We run this step on every build, and it is usually where the last three or four unwritten rules of a job finally get written down. What that looks like start to finish is on the build page.
The audit answers this for your business
Two weeks, $2,500 flat ($1,000 for the first three clients), and you get the map of your own automatable work with dollars on it.