Operators of very large websites increasingly describe automated crawling as a persistent load and cost problem rather than an occasional nuisance. The debate is about how sites distinguish wanted from unwanted traffic, and at what cost to ordinary readers.
Key takeaways
- Site operators running very large numbers of pages report that automated crawling has become a sustained infrastructure burden rather than a background occurrence.
- Much of the difficulty comes from telling apart search crawlers, AI training collectors, archival tools, scrapers and ordinary readers, because all of them arrive as HTTP requests.
- The robots exclusion protocol is a voluntary convention, so it constrains only crawlers that choose to honour it and offers no technical enforcement.
- Common countermeasures — rate limiting, blocking address ranges, requiring proof-of-work or presenting challenges — reduce automated load but also affect some genuine visitors.
- The scale and identity of scraping traffic on any specific site cannot be verified from outside, because the underlying server logs are private.
What is actually happening on large websites
A website with a very large number of distinct pages presents a particular problem: the number of unique URLs available to be fetched is far greater than any human audience will ever read. Databases of records, archives, catalogues, wikis, code repositories and reference works all generate this shape. Each page may be cheap to serve individually, but an automated client that walks through the whole set imposes a load pattern no human audience would produce.
Operators of such sites describe a recurring pattern. Automated clients arrive, request pages systematically, and continue regardless of whether the content changes. Some identify themselves clearly in the user-agent string. Some declare a name that cannot be verified. Some present themselves as ordinary browsers. Requests may come from a single network range, from cloud hosting providers, or from very large pools of residential addresses that are difficult to distinguish from home broadband users.
The result is that a site’s costs — bandwidth, database queries, cache misses, the compute needed to render pages — can be driven substantially by traffic that generates no readers. For a site funded by advertising, subscriptions or donations, this creates a direct mismatch between the cost of operating and the revenue the operation produces.
Why this is being discussed now
Discussion of crawling load has intensified alongside the growth of systems that consume large quantities of web text. Training data collection, retrieval systems that fetch pages on demand, and assistants that read pages on behalf of users all generate machine requests to websites. This has been a widely reported concern across the technology press and among infrastructure providers, though the precise volumes attributable to any particular category of client are not publicly established.
At the same time, the tools available to operators have become a subject of active development. Bot-management products, challenge systems, proof-of-work gateways and various forms of network-level filtering have all attracted attention. Accounts written by individual site operators — describing what they tried, what worked and what broke — circulate widely because they offer concrete operational detail that vendor material and policy discussion tend not to provide.
The background a newcomer needs
Web crawling is not new and is not inherently hostile. Search engines index the web by crawling it, and for decades most sites treated crawler traffic as a benefit, because indexing produced visitors. Archival projects crawl to preserve material. Academic and commercial research crawls the web for analysis. The infrastructure of the open web assumes that pages can be fetched programmatically.
The main convention governing this is the robots exclusion protocol, usually expressed in a file named robots.txt at the root of a site. It lets an operator state which parts of a site automated clients should not fetch. It is a request, not a technical control: a client that ignores it faces no barrier. Enforcement, where it exists, comes from separate mechanisms — blocking, rate limiting, authentication — or from legal action, which is slow, jurisdictionally complex and generally impractical for a small operator.
There is no reliable technical way to verify what a client intends to do with a page once it has been served. A user-agent string is self-declared and can be set to any value. IP address ownership can be checked against published ranges, and some major crawlers publish verification methods, but clients routed through residential proxy networks do not present a distinguishable network fingerprint. Behavioural signals — request rate, access patterns, whether a client fetches associated assets — are informative but imperfect, and they can be imitated.
Who is affected and how
The operators most exposed are those running content-heavy sites without the revenue of a large platform: open-data projects, community wikis, independent archives, documentation hosts, and hobby projects that happen to expose many pages. Their infrastructure is often modest, and a sustained crawl can be indistinguishable from a denial-of-service event in its effects, even where no such intent exists.
Readers are affected indirectly. Countermeasures deployed to reduce automated load also touch human visitors. Interstitial challenges add delay. Blocking entire network ranges can cut off legitimate users who share those ranges, including people using VPNs, mobile carrier gateways or institutional networks. Restrictions can degrade accessibility for people using assistive technology or older browsers. Requiring accounts to view content reduces the openness that made the material valuable.
Crawler operators are affected too, including those acting in good faith. As defences generalise, well-behaved crawlers can be caught by measures aimed at badly-behaved ones. Archival and research projects have described difficulty accessing material that was previously open.
Where informed people disagree
There is no consensus on where the fault lies. One position holds that crawling is a normal cost of publishing on a public network, that sites choosing to serve pages to anyone have accepted the consequences, and that the correct response is engineering — caching, static generation, rate limiting — rather than exclusion.
A contrary position holds that systematically extracting a site’s entire contents is qualitatively different from indexing it, particularly where the extraction supports a commercial product that does not send readers back. On this view the reciprocal arrangement that made crawling acceptable — traffic in exchange for indexing — has broken down.
There is further disagreement on remedies. Some favour technical measures, arguing they are the only mechanism a site can actually enforce. Others argue that escalating defences degrade the open web for everyone and mainly disadvantage smaller crawlers who lack the resources to work around them. Proposals for machine-readable declarations of permitted use, for payment or licensing arrangements, and for legal or regulatory intervention all have advocates, and none has been established as a general solution.
The practical implications
For an operator, the practical question is which lever to pull first, and each has a cost. Caching and static generation reduce the per-request expense without deciding who is allowed in. Rate limiting curbs the heaviest clients but requires tuning to avoid catching shared network addresses. Blocking by address range or autonomous system is blunt and produces collateral effects. Challenge systems and proof-of-work gateways impose a cost on every visitor, including humans. Authentication is effective and closes the site.
None of these is a settled answer, and the choice depends on what the site is for. An archive whose purpose is broad public access will weigh the trade-offs differently from a commercial database. Operators also face an ongoing maintenance burden: rules require adjustment as client behaviour changes, and time spent on this is time not spent on the site itself.
What to watch next
Several developments would change the picture. Wider adoption of machine-readable standards that distinguish crawling purposes — indexing, archiving, training, live retrieval — would give operators a way to express preferences more precisely than robots.txt allows, though such a standard would still depend on voluntary compliance. Clearer legal positions on large-scale extraction, in any major jurisdiction, would alter operators’ calculations.
Also worth watching is whether infrastructure providers make filtering available by default at low cost, which would shift the burden away from individual operators, and whether that concentration of gatekeeping raises its own concerns. Finally, the extent to which sites move behind registration or paywalls is an observable signal of how the trade-off is being resolved in practice.
Frequently asked questions
What is web scraping and how is it different from crawling?
Crawling means systematically fetching pages by following links, typically to build an index. Scraping means extracting specific data from the pages retrieved. The two overlap heavily in practice and the terms are often used interchangeably. The technical activity — issuing HTTP requests and reading responses — is the same. The distinction people usually intend is one of purpose and of what happens to the content afterwards, not of method.
Does robots.txt actually stop scrapers?
Only those that choose to obey it. The robots exclusion protocol is a voluntary convention: a file stating which paths automated clients should avoid. Major search engines generally honour it. A client that ignores it encounters no technical barrier, and there is no built-in mechanism to detect or penalise non-compliance. Operators who need enforcement must use separate controls such as rate limiting, network blocking or authentication.
Why is scraping expensive for a website owner?
Each request consumes bandwidth, and pages generated from a database also consume compute and database capacity. A crawler walking through a very large site touches many pages that are rarely requested by humans, so caches do not help and each request costs the full amount. Because this traffic produces no readers, subscribers or advertising views, the cost is not offset by any corresponding revenue.
Can a site tell the difference between a bot and a person?
Not reliably. A client declares its own identity through the user-agent string, which can be set to any value. Network address checks help for crawlers that publish their ranges, but not for traffic routed through residential proxies. Behavioural signals such as request rate and access patterns are useful but imperfect, and can be imitated. Detection therefore involves probabilistic judgement and produces both false positives and false negatives.
Do anti-bot measures hurt ordinary visitors?
They can. Challenge pages add delay and sometimes fail for people using older browsers, VPNs or assistive technology. Blocking whole network ranges can cut off legitimate users sharing those addresses, including mobile and institutional networks. Requiring an account excludes casual readers entirely. Operators generally treat this as a trade-off to be tuned rather than eliminated, accepting some impact on human visitors in exchange for reduced automated load.
Is scraping a public website legal?
The position varies by jurisdiction and by the specific facts, including what is scraped, how, and what is done with the results. Copyright, contract terms, database rights and computer-misuse laws may all be relevant, and courts in different countries have reached different conclusions. There is no single settled answer, and this article does not offer legal advice. Operators considering legal action typically find it slow and costly.
Sources and further reading
- Technical discussion forums where site operators publish first-hand accounts of infrastructure load and mitigation attempts.
- Published documentation of the robots exclusion protocol and related web standards work.
- Reporting by established technology publications on automated data collection and its effect on website operators.
- Documentation from content delivery and infrastructure providers describing bot-management and rate-limiting techniques.
Surfaced from the hackernews signal “website scraping load problems”. AI-assisted draft, editorially reviewed.
