A widely shared developer write-up describes how long it took to move Anubis, an open-source proof-of-work gate that filters automated web traffic, from a JavaScript challenge to a WebAssembly one. The delay is the story.
Key takeaways
- Anubis is an open-source reverse-proxy tool that makes a visitor’s browser complete a small computational puzzle before a request is passed through to the protected website.
- The discussion trending on Hacker News concerns how long it took to ship a WebAssembly implementation of that puzzle, rather than any newly disclosed vulnerability or breach.
- WebAssembly is a compact binary format that browsers execute in a sandbox, and it can run tight hashing loops faster than equivalent hand-written JavaScript.
- Software that sits in front of every single request has an asymmetric failure cost, because a bug that blocks a minority of browsers locks those users out of the site entirely.
- Security engineers disagree over whether proof-of-work walls meaningfully deter well-resourced automated crawlers or simply shift cost onto ordinary visitors with slow devices.
What is actually happening
The trending item is a technical account, circulated on Hacker News, of a long-running piece of engineering work in Anubis: replacing or supplementing the JavaScript routine that computes its proof-of-work challenge with a WebAssembly module. The headline framing — that it took roughly a year — is what drew comment, because the change sounds, on paper, like a contained optimisation. Compile a hashing loop to WebAssembly, serve the module, measure the speed-up.
The write-up’s argument, as reflected in the discussion around it, is that the surrounding constraints dominated the work rather than the algorithm itself. Anubis is a gate. Every request to a protected site passes through it, and any client that cannot execute the challenge correctly is, from the user’s point of view, simply denied the website. That turns a routine performance change into an exercise in compatibility, packaging and failure handling across an unbounded population of browsers, extensions, proxies and privacy tools.
The precise timeline, the internal reasons for individual delays and the final performance figures are matters for the project’s own repository and release notes; they are not independently established here.
Why it is in the news now
Two things put this in front of a general audience. The first is that proof-of-work walls have become far more visible on the open web over the past couple of years, as operators of code forges, documentation sites and community wikis have looked for ways to blunt the load imposed by aggressive automated crawling, including crawling associated with the collection of training data. Readers who have encountered an interstitial page telling them their browser is being checked have a direct stake in how that mechanism behaves.
The second is that the piece speaks to a familiar frustration in software engineering: the gap between how long a change looks like it should take and how long it actually takes when it ships into hostile, heterogeneous conditions. That gap generalises well beyond one project, which is usually what pushes a narrow technical post into a broad comment thread.
The background a newcomer needs
Proof of work asks a client to spend a measurable amount of computation before it is granted something. The client repeatedly hashes a value with a small varying component until the output meets a difficulty condition — a result that is expensive to find and trivial for the server to verify. Bitcoin popularised the idea, but its use as an anti-abuse measure predates that and was originally proposed as a way of imposing a cost on bulk email.
Applied to the web, the logic is economic rather than cryptographic. A human reading a handful of pages barely notices a fraction of a second of hashing. A crawler fetching millions of pages pays that cost millions of times, which is intended to make indiscriminate scraping less attractive than targeted, rate-limited fetching.
WebAssembly is the other half of the story. It is a standardised, portable binary instruction format that modern browsers execute inside the same sandbox as JavaScript, and it is generally well suited to numeric, loop-heavy work of exactly the kind a hashing routine performs. Shipping it, though, means shipping a binary artefact: it must be built reproducibly, served with the correct content type, allowed by the site’s content security policy, cached sensibly and given a working fallback for clients that refuse or fail to load it.
Who is affected and how
Three groups are affected in different ways. Site operators are choosing whether to place a challenge in front of their content at all, and they carry the consequences when it misfires: lost readers, broken feeds, and support requests they cannot easily reproduce.
Ordinary visitors are affected most directly. A faster challenge implementation means less waiting, and the difference matters most on old phones and low-powered laptops — precisely the devices for which a JavaScript hashing loop is slowest. Visitors using text-mode browsers, older devices, hardened privacy configurations or assistive technology sit at the sharp end of any compatibility gap, because for them a failed challenge is indistinguishable from an outage.
The third group is everything automated but legitimate: search engine crawlers, archiving projects, accessibility tools, link checkers, feed readers and API clients. These do not always execute JavaScript or WebAssembly, and whether they are allowed through depends on configuration choices made by each operator rather than on any property of the challenge itself.
Where informed people disagree
The central disagreement is about effectiveness. Critics point out that proof of work is a tax, not a lock: an operator with substantial computing resources can pay it, and the cost of doing so may be negligible compared with the value of the data collected. On this reading, the wall reliably inconveniences the least-equipped visitors while barely troubling the best-funded crawlers.
Defenders argue the goal is narrower — to remove the cheapest, highest-volume, least discriminating traffic, and to make the remaining automated access legible and rate-limitable. Marginal deterrence, in that framing, is still a useful outcome for a small operator facing a bandwidth bill.
There is a parallel argument about the web’s direction. Requiring script execution to read a page conflicts with the principle that documents should be retrievable by simple clients, and the energy spent on billions of discarded hashes is a real, if small, cost. Alternatives — conventional rate limiting, cryptographic attestation issued by a small number of large intermediaries, or negotiated commercial crawling agreements — each carry their own centralisation or accessibility objections. No approach is uncontested.
The practical implications
For anyone maintaining infrastructure, the transferable lesson concerns risk placement rather than WebAssembly. A component that all traffic must traverse has no low-stakes changes. The safe path involves feature detection rather than browser sniffing, a working fallback along the original code path, gradual rollout, and enough telemetry to notice that a subset of clients has begun failing — while keeping in mind that a locked-out user often cannot report the problem through the site that locked them out.
For site operators weighing a challenge wall, the implication is that adopting one is an ongoing commitment. Challenge pages interact with caching layers, content security policies, single sign-on flows, feeds and API endpoints, and those interactions need revisiting whenever any of the parts change. For readers, the practical point is more modest: an unexplained interstitial is now a normal part of browsing, and a faster implementation shortens rather than removes it.
What to watch next
Worth watching is whether measured speed-ups from WebAssembly translate into lower difficulty settings, since the aim is to hold the cost roughly constant rather than to hash more. Also relevant is how challenge systems handle non-browser clients as more legitimate traffic becomes automated, and whether allow-listing conventions or signed-agent schemes gain traction as an alternative to blanket challenges.
Beyond that, the direction of travel for anti-scraping on the open web remains unsettled: whether it consolidates around a handful of large intermediaries, or stays a patchwork of self-hosted tools maintained by small teams. The trade-offs in that choice — resilience, cost, and who gets to decide which clients count as real — are the substance behind an otherwise narrow engineering post.
Frequently asked questions
What is Anubis?
Anubis is an open-source tool that sits in front of a website and requires a visiting client to complete a small computational challenge before the request is forwarded to the protected server. It is typically deployed by operators trying to reduce the load imposed by high-volume automated crawling. It is self-hosted software rather than a commercial service, and its configuration — including which clients are exempted — is decided by each operator.
Why would a website make my browser do calculations?
The intent is to impose a small, near-unnoticeable cost on each request so that fetching one page is cheap but fetching millions becomes expensive. It is an economic deterrent aimed at indiscriminate automated crawling rather than a test of identity. The calculation runs in your browser, takes a fraction of a second on typical modern hardware, and is discarded once the server verifies the result.
Does WebAssembly make these challenges faster?
WebAssembly is generally well suited to tight numeric loops of the kind a hashing routine uses, and browsers execute it in a sandbox alongside JavaScript. In principle that means the same challenge completes sooner, which matters most on slower devices. The size of any real-world improvement depends on the specific implementation and hardware, and is not something that can be stated as a single figure.
Why would a small change take a year to ship?
Because the change sits on the critical path for every request. A component that all traffic must pass through cannot fail gracefully in the way an optional feature can: a client that mishandles the new code is locked out of the site entirely. Verifying behaviour across browsers, versions, devices, privacy configurations and intermediary proxies — and building a reliable fallback — accounts for most of the effort.
Do proof-of-work walls actually stop scrapers?
Opinion is divided. They raise the cost of high-volume, indiscriminate fetching, which can deter the cheapest crawling, but a well-resourced operator can absorb the cost. Critics argue the burden lands disproportionately on visitors with slow devices while doing little against determined collectors. Supporters treat it as one layer among several, useful for shedding low-effort traffic rather than as a comprehensive defence.
Will a challenge wall break feed readers or accessibility tools?
It can. Clients that do not execute scripts — including some feed readers, archiving crawlers, command-line tools and certain assistive setups — may be unable to complete a challenge. Most such systems allow operators to exempt specific paths, user agents or networks, but that depends entirely on configuration. Whether a given site handles these cases well is a decision made by that site’s operator, not by the software alone.
Sources and further reading
- The project’s own public repository and release documentation, for the authoritative account of what shipped and when.
- Hacker News, where the write-up was submitted and discussed, useful for the range of practitioner opinion rather than as a factual record.
- The W3C and MDN Web Docs specifications and reference material on WebAssembly, for how the format is executed and delivered in browsers.
- General technical press coverage of automated crawling and anti-scraping measures on the open web, for context on why operators have adopted challenge systems.
Surfaced from the hackernews signal “open-source anti-scraping tool update”. AI-assisted draft, editorially reviewed.
