Someone in your organization is about to pitch a plan to publish a thousand AI-generated pages. Before they do, they should understand how Google actually pays for crawling, because the plan usually dies there.
A recent Search Engine Journal piece laid out the crawl economics cleanly, and it explains a pattern I keep seeing. Scaled AI content works for about a quarter, then quietly disappears from the index.
Crawl Budget Does Not Scale With Your Ambition
Google does not automatically expand its crawl budget when you add hundreds or thousands of URLs. It allocates crawling based on perceived inventory, real demand, and domain popularity, because crawling costs it money and it manages that cost deliberately.
So the mental model of publish more, rank more is broken at the infrastructure level. You can create a million URLs. Google decides how many are worth its compute, and it decides using signals your content volume cannot fake.
New pages do get a brief window. Freshness signals give AI-generated content a temporary indexing boost, which is why the first month often looks like a win. That is the trap, not the proof.
Once novelty fades, pages have to earn their place with clicks, engagement, and user signals. Without them, the study describes de-indexation inside a 75 to 140 day window. The content did not get penalized so much as it failed to justify its own storage.
The freshness window is where most scaled-content post-mortems go wrong. A team ships five hundred pages, sees indexing and a little traffic in month one, and declares the program a success. Then month three arrives, the pages quietly drop, and nobody connects the collapse to a novelty boost that always had an expiry date. They optimized against a signal that was never going to last.
Google Is Pricing Information Gain, Not Word Count
The deeper point is what Google is actually buying. It rewards information gain, something new, original, or genuinely useful, not another restatement of what ten other pages already say.
Scaled AI content fails because it usually offers little unique value, no original reporting, and no distinct experience. Keyword placeholders, auto-translations, and reworded summaries of existing results are exactly what the systems are built to throttle. Google has also increased manual actions against this kind of mass-produced abuse.
Information gain is the concept worth internalizing here. Google is not only asking is this page relevant, it is asking does this page add anything the index does not already hold. A competent AI summary of the top ten results is, by definition, information the index already has. You spent compute to tell Google what it told you.
This is why first-party data has quietly become the strongest SEO asset most companies own and ignore. Your support tickets, your usage numbers, your pricing experiments, your customer survey, none of that exists anywhere else. A page built on it clears the information-gain bar automatically, because the model cannot reconstruct what only you measured.
I argued something adjacent in the ultimate guide is dead. The comprehensive page that covers everything at length is now the least defensible asset you can build, because the model already read those pages and blended them into one grey answer.
There is a supply problem underneath this too. A third of the new web is already AI-generated, a number I looked at in a third of the new internet is already AI. When everyone floods the same commodity content, the marginal page is worth less, and Google prices it accordingly.
The Move Is Fewer Pages That Earn Their Crawl
Stop thinking about SEO as a checklist you complete and start thinking about it as an economy you participate in. Google is a buyer with a budget. Every page you publish is asking it to spend, and it will only spend on pages that return demand.
Practically, that means fewer, denser pages built around a specific claim, a real number, or a first-party data point the model cannot find elsewhere. One page that states something worth quoting beats fifty that summarize the consensus.
Run the test on your own site. If you cannot point to the specific claim, number, or experience on a page that no competitor could publish, Google has no reason to keep spending its crawl budget on it. That is the whole filter in one question.
It also means watching engagement as a survival metric, not a vanity one. If a page cannot hold attention, it will not hold its index position past the freshness window. The click was always a signal, now it is life support.
Use AI to draft faster, to research, to structure. That is fine, I do it daily. Just do not confuse the speed of production with the value of the output, because Google measures the second and ignores the first.
The teams winning here are not publishing more. They are publishing less, and making each page earn the crawl. Volume was never the moat. It was just the cheapest thing to measure.