Most content owners think they have a switch. If an AI company should not read your pages, you add a line to robots.txt and the crawler goes away.
That switch only works if the crawler tells you who it is. The newest lawsuit from The New York Times is about a crawler that allegedly never did.
What the Times is alleging
A recent breakdown on Search Engine Journal walked through the complaint the Times filed against the makers of the Brave browser and Brave Search. Every point below is an allegation, not a finding, and Brave will have its response in court.
The Times alleges that Brave scraped and copied its journalism to build a search index, including content behind the paywall. It alleges that Brave sells access to that content through its Search API, used by companies including Anthropic, Perplexity and Mistral for training and for generating answers.
It also alleges that the Ask Brave AI chat summarises Times articles in a way that replaces a visit, that the browser helps users get around the paywall, and that Brave removes the Times's ads and substitutes its own. The complaint says cease-and-desist letters were sent in August 2025 and April 2026 and ignored.
The allegation I would pay most attention to is quieter than the rest. The Times says Brave does not identify its crawler and gives publishers no way to block it.
The privacy brand and the supply chain
The interesting tension is reputational. Brave built its brand as the privacy-first alternative to Google, with an independent index and no tracking across the web. All of that is true, and it is why many people trust it.
The complaint describes a different business model on the other side of the product. Instead of monetising users, it allegedly monetises the work of publishers, by selling their content to AI companies and serving answers that compete with the original pages.
I am not interested in deciding who is right here. I am interested in what it reveals about how content actually moves into AI systems. A model provider can have clean crawler policies and still end up answering with content that arrived through a search API it bought from someone else.
That is a supply chain. Most marketing teams have never mapped it, because until recently there was no reason to.
Robots.txt is a request, not a lock
I made this point in disallow was never a privacy setting, and this case makes it concrete. Robots.txt is a convention. It works when the crawler identifies itself honestly and chooses to respect it.
If a crawler presents itself as an ordinary browser, there is no user agent to block. Your server logs show traffic that looks like a normal visitor, and the content leaves the building without a record of where it went.
For a large publisher, that becomes a lawsuit. For a mid-sized company, it usually becomes nothing at all, because nobody is watching. Your pricing pages, your research reports and your gated content can end up inside an AI answer, and you will only find out when a prospect quotes it back to you.
What content owners should do now
I would not panic, and I would not try to block every bot on the internet. Some AI visibility is commercially valuable, and the same crawlers that copy content also put brands in answers.
But I would separate content into three groups. Public content you want machines to read and repeat, such as product information, service pages and thought leadership. Content you want summarised but not reproduced, such as long reports. And content that should never leave your control, such as paid research, client materials and anything behind a login.
The first group should be easy for machines to read. The third group needs real access control, meaning authentication and server-side checks, not a polite line in a text file. A paywall that loads the full article in the HTML and hides it with a script is not a paywall to a scraper.
Then watch the logs. Look for traffic patterns that behave like crawlers but claim to be browsers. Rate limits and bot management tools at the CDN level catch more than robots.txt ever will.
What API buyers should ask
The second-order effect is for companies on the other side, the ones building products on top of third-party search and data APIs. If you buy search results to ground an AI assistant, you are inheriting whatever the provider did to collect them.
That used to be a technical question. It is now a contract question. Before signing, ask the provider how the index is built, whether its crawler identifies itself, and whether it honours publisher opt-outs. Ask for indemnification if the answers turn out to be wrong.
When we build AI systems for companies, data provenance now sits next to latency and price on the evaluation sheet. A cheaper API that creates legal exposure is not cheaper. It just moves the cost to a later quarter, the way unlicensed images moved costs to design teams a decade ago.
I wrote earlier that training data has lawyers now. This case extends that to retrieval. The content an AI uses to answer a question at runtime is going to get the same scrutiny as the content it was trained on.
The web ran for thirty years on the assumption that visitors say who they are. AI crawling is testing what happens when they stop.