In the last seven days, Anthropic shipped new Claude models and Google cut the price of its fast video model by 30%. Next week, someone else will ship something. The week after, someone will ship again.
Meanwhile, the prompts your team runs every day were written for whatever model was current when someone wrote them.
The switch nobody tests
Here is how model changes usually happen inside a marketing team. Someone reads that the new model is better. They change a dropdown, or a vendor changes it for them. Work continues.
Two weeks later, someone notices the product descriptions got longer. The email subject lines lost their edge. The summarisation workflow started adding a cheerful sentence at the end that nobody asked for. Nobody connects it to the switch, because nobody recorded when the switch happened.
I have seen this exact sequence at clients more than once. In one case, a content workflow had quietly drifted for a month before anyone traced it back to an automatic model update by the tool vendor.
"Better" on a public benchmark means better on someone else's tasks. Your tasks are not on the benchmark.
The vendor is not hiding anything. A model that is better at reasoning can also be wordier, more cautious or more eager to format everything as bullet points. Those are trade-offs, and only you know which ones your workflows can absorb.
Why marketing teams are especially exposed
Engineering teams have a habit marketing teams lack. Before they change something important, they run tests. If the tests pass, they ship. If they fail, they know exactly what broke.
Marketing workflows rarely have that. The definition of good output lives in people's heads. The senior copywriter knows what on-brand sounds like.
The performance lead knows which ad angles work. When a model changes, those judgments are the only safety net, and they only catch problems when someone happens to look.
The other exposure is volume. A team running AI across hundreds of product pages or thousands of ad variations does not review every output. A small drift multiplied across that volume is a real brand problem before anyone sees it.
A regression set in one afternoon
The fix borrows directly from software. I wrote about the principle in evals are the new product spec. Here is the practical version for a marketing team.
Collect 30 to 50 real tasks from your actual workflows. Not invented examples, real ones: the product brief that always causes trouble, the tricky customer email, the ad brief for your hardest category. Include a few edge cases where the model should refuse or ask for more information.
For each task, write down what good looks like. Not a perfect answer, a standard. Under 80 words. Mentions the warranty.
Never promises delivery dates. Uses the customer's terminology, not ours. Two or three checkable criteria per task is enough.
Save the current model's outputs as the baseline. That is your reference point, the thing you are trying not to make worse.
Running the check
When a new model appears, run the same tasks through it. Compare against the criteria and the baseline. Some of the comparison can be automated with simple rules, word counts, required phrases, banned phrases. The rest needs a person who knows what good looks like.
That person is the most important part of the system. They decide whether a change is an improvement, a different style or a regression. Software companies are rediscovering how valuable that role is, the person who defines quality rather than produces volume. Marketing teams need the same role, explicitly named.
The decision rule should be agreed in advance. Switch if the new model matches or beats the baseline on most tasks and does not fail any critical ones. Stay if it fails anything critical, regardless of how impressive the demo looked.
At Madison AI, we run every model change through this kind of set before it touches a client workflow. It takes about fifteen minutes per model now that the set exists. Building it took an afternoon. The afternoon has paid for itself many times over.
The real cost is not the model price
Model prices keep falling, and that makes switching tempting. A 30% price cut looks like free money. But the cost of a bad switch is never on the model invoice. It shows up as rework, as brand inconsistency, as a campaign that underperformed for reasons nobody could explain.
The regression set also protects you from the opposite mistake: never switching because change feels risky. With a set in place, you can adopt better models faster, because you have evidence instead of anxiety. I made a related case in the model is not your moat. If the model is replaceable, your process for replacing it is part of your architecture.
Ask your team one question this week. If the model under your main workflow changed tomorrow, how would you know?
If the answer is "someone would probably notice," you do not have a process. You have hope.