Publishers can limit how AI firms use their pages. Read the terms you accept, keep dated copies, and set robots.txt and server rules for AI crawlers. Then check your logs to confirm the rules are being followed.
Gather what you need before starting
This guide is for website owners, editors and small publishers who want more say over whether their content is used to train AI systems. It comes from a recent change reported by The Verge. After criticism, Meta is revising its commercial terms again. It is removing wording that seemed to let the company use content from websites, including news sites, to develop and improve its AI. The Verge reports that this wording no longer appears on the commercial terms page. It is not publicly known whether the clause was ever used to collect content, or how many sites it could have covered.
The episode shows two separate ways your content can end up in a training dataset. One is the contracts you sign with platforms. The other is the crawlers that visit your site. You need to deal with both.
Before you begin, gather the following:
- A list of every platform service your organisation has signed up to, such as advertising accounts, analytics tags, social media business pages and commerce integrations.
- Login access to those accounts, so you can find the version of the terms you accepted.
- Access to your website’s root directory or content management system, so you can edit the robots.txt file.
- Access to server or content delivery network (CDN) logs, or someone who can pull them for you.
- Somewhere to store documents with dates, such as a shared drive.
List the terms you have accepted
Begin with contracts, not code. Many publishers accept business or commercial terms when they install a tracking pixel, connect a domain to an advertising account or verify ownership of a website. These terms are separate from the consumer terms that individual users accept. They can include clauses about content found on the domains you link to the service.
For each platform, find the commercial, business tools or partner terms, plus any data processing addendum. Write down which domains you registered with each service. A clause covering “content from such domains” only matters if your domain is among them.
Read the data-use clauses closely
Search each document for words such as “content”, “domains”, “URLs”, “develop”, “improve”, “train”, “machine learning” and “artificial intelligence”. Phrases such as “provide, develop and improve our products” may look routine. But if they appear next to a reference to content on your domains, they can be read broadly.
For each clause, note three things: what content it covers, what the platform may do with it, and whether there is any opt-out. If the wording is unclear, ask the platform in writing what it means, and keep the reply. If your organisation has legal counsel, this is the point to bring them in.
Archive a dated copy of every version
Platform terms change, sometimes more than once in a short period. The Meta case involved a revision followed by a further correction. If you only have the current page, you cannot show what you agreed to earlier.
Save each set of terms as a PDF or full-page capture. Name the file with the platform name and the date you downloaded it. Do this again whenever a platform tells you its terms have changed. A simple record is enough: date, document, clause and your summary. That gives you something to work from if a dispute ever comes up.
Audit your robots.txt file
The second route is crawling. Open yoursite.com/robots.txt in a browser and read what it says. Many sites still use a template written before AI crawlers existed. These often allow every bot except a few named ones.
Robots.txt works by user agent, the name a crawler uses to identify itself. Several AI developers publish separate user agents for training-related crawling. Examples include crawlers documented by OpenAI and Common Crawl, a Google token for AI use, and agents that Meta describes in its developer documentation. These names change and new ones appear. Check each company’s official crawler documentation for the current names rather than relying on lists from third parties.
Add rules for the crawlers you want to exclude
For each crawler you want to block, add a section to robots.txt. It names the user agent and disallows the paths you want to protect. Use a forward slash to cover the whole site. You can block training crawlers while still allowing ordinary search crawlers. Search traffic and AI training are usually handled by different agents.
Make one change at a time and keep a copy of the earlier file. One misplaced line can block search engines from your whole site, which causes a serious and immediate drop in traffic.
You can also add a clear statement to your site’s own terms of use saying that automated collection for AI training is not permitted without a licence. This does not stop a crawler on its own. It does record your position in public.
Verify compliance in your server logs
Robots.txt is a request, not a lock. Compliance is voluntary. Some reputable crawlers respect it, and others may not identify themselves accurately.
After changing the file, look at your server or CDN logs over the following weeks. Search for the user agents you blocked and check whether they still request pages other than robots.txt itself. Also look for heavy, systematic requests from unfamiliar agents. If a crawler ignores your rules, your CDN or firewall can usually block it by user agent or by published IP range. Some CDNs offer ready-made settings for managing AI bots.
Avoid the mistakes people actually make
- Treating robots.txt as legal protection. It sets out your wishes and many companies honour it. It does not override a contract you have signed, and it does not physically prevent access.
- Ignoring the contract route. Blocking a crawler achieves little if a commercial agreement you accepted grants broader rights to content on your domain.
- Copying old blocklists. Lists of AI user agents go out of date quickly. Some entries are wrong or refer to agents that no longer exist.
- Blocking search crawlers by accident. A wildcard rule intended for AI bots can remove your site from search results.
- Not keeping records. Without dated copies of terms and logs, you cannot show what changed or when.
- Assuming one platform’s correction applies to all. A change at one company tells you nothing about the terms of others.
Know when this approach is the wrong choice
This approach suits publishers who want to limit AI training use without giving up search visibility. It is the wrong choice if your organisation wants its content to appear in AI-generated answers, for example to reach readers through chat assistants. Blanket blocking may reduce that exposure.
It also falls short if you plan to license your archive to AI developers for payment. In that case you need a negotiated agreement, and robots.txt rules become one part of a wider commercial strategy. Content behind a paywall or login also needs access controls, not crawler directives. And if you believe your content has already been collected in breach of an agreement, technical steps will not settle it. You need legal advice.
Frequently asked questions
Did Meta’s terms let it scrape news websites for AI?
The Verge reports that Meta’s commercial terms contained wording suggesting the company could use content from certain domains, web pages or URLs to develop and improve its products, including AI. After criticism, Meta removed that wording from the commercial terms page. It is not publicly known whether the clause was ever applied to collect content, or which sites it could have affected in practice.
Does robots.txt stop AI companies from training on my content?
Robots.txt asks crawlers not to access certain pages, and many established AI developers say their crawlers respect it. It cannot enforce anything. A crawler can ignore it or misidentify itself, and it does not cover content gathered before you added the rules. For firmer control, combine it with firewall or CDN blocking and clear terms of use on your site.
Will blocking AI crawlers hurt my search rankings?
Blocking AI training crawlers should not affect traditional search rankings if you target only the relevant user agents. Search engines generally use separate agents or tokens for indexing and for AI purposes. The risk comes from broad wildcard rules that block search crawlers by mistake. Test any change carefully and keep a copy of your previous robots.txt file.
How do I find which AI crawlers are visiting my site?
Check your server or CDN access logs for user agent strings. Search for names published by AI developers and look for heavy, systematic requests from unfamiliar agents. Many CDNs provide bot analytics dashboards that sort traffic by crawler type. Compare what you find with each company’s official crawler documentation, because user agent names change over time.
Sources and further reading
- The Verge: reporting on Meta’s removal of commercial terms wording about using website content for AI development.
- Meta developer and business documentation: official descriptions of its crawlers and its commercial terms.
- The Robots Exclusion Protocol standard published by the Internet Engineering Task Force: how robots.txt is meant to work.
- Crawler documentation from AI developers and content delivery network providers: current user agent names and bot management options.
Surfaced from the rss:verge signal “platform AI terms revision”. AI-assisted draft, editorially reviewed.
