An update on the scraper situation [LWN.net]

Welcome to LWN.net The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!

By Jonathan Corbet
July 10, 2026

Our article “Fighting the AI scraper bot
scourge”, published in early 2025, discussed the problem of widespread
scraping of web sites in search of training data for large language models
and related projects. This activity overwhelms sites with traffic. Over a
year after that article is published, the problem is still growing. The
hammering of sites by shadowy actors has reached new heights, and the open
web is becoming increasingly difficult to maintain. Where is this traffic
coming from, and what can be done about it?

Residential proxies

As was described last year, scraper attacks come from a huge number of
sources across the net. It is not unusual to see coordinated requests from
millions of unique IP addresses over the course of a few hours, each of
which hits the site at most two or three times. Attacker-controlled data,
such as the user-agent field, is entirely fictional; each hit is meant to
look like just another human with a web browser. There are ways to tell
the difference — the bots usually do not fetch images or CSS, for example —
but, by the time that determination is made, the address in question will
not be used again. Blocking the address at that point is just a waste of
time.

This traffic comes predominantly from residential and mobile networks,
directed by central command-and-control nodes. Software is installed on
ordinary systems that takes orders from a control node, fetches web pages
on demand, and forwards the resulting data back to the controller. Much of
the time, this activity occurs without the knowledge or consent of the
owner of the device in question. The term “residential proxies” is used to
describe systems that are used in this way.

There are a few different (on the surface, at least) types of operator
running residential-proxy networks to attack web sites. One type is purely
criminal, running scrapers on systems that have been compromised with some
sort of malware. At the beginning of the year, Google acted
to take down a bot network called IPIDEA and provided a lot of
information about how these operations work. The shutdown of IPIDEA
correlated with a significant reduction in scraper traffic here at LWN;
things were relatively peaceful for a few months. That period of peace has
since come to an end, though.

More recently, media-streaming devices have been identified
as a major carrier of malicious scraping software. Sometimes the devices
are compromised at the source; other times, they are just poorly secured
and easily compromised after the fact.

The second sort of operator works more overtly, pretending to a degree of
legitimacy and offering “ethically sourced” IP addresses. A company called
Bright Data is one of the most prominent of these; it happily advertises
its prowess at getting around web-site access controls and traffic limits.
Bright Data offers a “free” VPN service; all that is needed is for the user
to give Bright Data the ability to route traffic through the user’s device
— to become a part of the company’s residential-proxy network, in other
words. Every phone or other device that makes use of this VPN becomes yet
another endpoint that will be used to attack web sites.

There are many other examples of this type of operator out there; often
they offer a library that app developers can link into their offerings and
be paid for hijacking their users’ network connections. One of them even
sent us a query about running an ad for its SDK on LWN; that was, it
suffices to say, a short conversation. In general, these companies range
from those that aspire toward some appearance of legitimacy, advertising
“GDPR compliance” for example, to others that are just overtly sleazy.

While these residential-proxy networks are used for web-site scraping, it
is worth emphasizing that these operators have the ability to run code that
accesses resources on whatever networks millions of devices happen to be
connected to. To assume that this type of access would only be used for
scraping would be naive at best.

Then, of course, there are the high-profile companies developing models as
their core business. These companies do their own scraping; the traffic
that can be easily attributed to them is clearly identified in the
user-agent field and, as a general rule, observes measures like
robots.txt. They, too, will scrape an entire site, repeatedly,
seemingly on the theory that articles written in 2003 might somehow have
changed in the last day, but they do not generate overwhelming amounts of
traffic from millions of systems and are not the biggest problem.

What isn’t clear is who is using the residential proxies; somebody
is paying them to run these attacks on web sites. There is no
evidence (that I am aware of) that the frontier-model companies are using
those networks. If were to turn out that they are doing so, though,
the increase in global astonishment would barely register. Those companies
are feeding their models somehow, they are not forthcoming about how they
get their training data, and they have not distinguished themselves with
their level of respect toward content creators — or toward anybody who
might have concerns about their operations.

For every public model, though, there must be a vast number of undercover
models. Many companies are surely trying to build their own; after all, we
are reliably informed that AI is going to take over the world and the
companies that come out on top of that race will be worth untold amounts of
money. There must be shadowy government agencies in many countries working
on their own models and groping for training data wherever they can find
it. Large-scale criminal organizations (to the extent that they are
distinct from governments) probably also want to have their own models.
These tools are seen as weapons, and there is an arms race underway. The
Internet as a whole is caught in the crossfire.

Defending the open Internet

In response to all of this, web-site operators have been scrambling to
defend their sites while minimizing the effect on their actual users. Anubis, which attempts to fend off scrapers by
requiring a proof of work, is now widespread. Other sites use commercial
services, which sometimes make themselves known with a “prove you are
human” button. Or sites force users to pick out squares containing
streetlights (but only those with LED bulbs), place puzzle pieces, or hum a
song while holding down the space bar. Many site features have been placed
behind login gates or paywalls. Some sites attempt to actively poison the
data sent to scrapers with tools like iocaine.

Both the need to set up and maintain these mechanisms, and the requirement
that users cope with them to access a web site, constitute a heavy tax
placed on the world as a whole by scrapers and those who pay them.

Recently, LWN was subjected what was, by far, the heaviest scraper attack
yet. Thanks to the defenses that have been implemented, the site bore the
traffic well enough that most actual readers probably did not even notice.
There have been requests to describe the measures we have taken to defend
the site; for obvious reasons we do not wish to discuss them in any detail.
It is an arms race at this level too.

What we can say is that we have tried to minimize the impact on real
readers as much as possible. We have not gone with tools like Anubis,
partly because it causes annoying delays for those trying to get to the
site, but also partly because it seems inevitable that the scrapers will
eventually find their way around it. Indeed, there are some indications
that is already happening. A proof-of-work requirement is not a huge
obstacle when you have millions of other people’s machines to do the work
on.

There is also a desire to not impede the operation of legitimate search
engines, the Internet Archive, and other such groups. Some sites may add
explicit allowlists to, for example, give the dominant search engine access
to the site. Such measures have the effect of further entrenching a
monopoly that already serves us poorly and should be avoided. We have,
thus far, succeeded in that.

We have aggressively optimized parts of the site, and found ways to
minimize expensive operations during times when the site is under attack.
Anonymous readers may occasionally encounter one of those measures;
logged-in users will not. Amusingly, the response time when the site is
under attack is often better than during the calm times, when the defensive
measures are dormant. We have learned better than to think that the
problem is solved, though; consideration must be given to our next steps
once the current measures are no longer effective.

On July 2, Google announced
that it had, in coordination with the US Federal Bureau of Investigation
and others, taken down a residential-proxy network called “NetNut”. For
the time being, that action would, indeed, seem to have succeeded in
reducing the level of scraper attacks somewhat. Experience shows, though,
that this welcome peace will only last so long. Google takes pains to
point out that its Play Store will now check for NetNut-infected apps, but
all of the major vendors are silent on the topic of why it is so easy to
put apps with residential-proxy functionality into their app stores.

It would be good to find a more lasting solution before the entire Internet
is driven behind defensive walls, and the open network that inspired so
much creativity is lost. The industry that is driving these attacks seems
entirely at ease with turning independent web sites into smoking craters
after having pillaged their contents — an attitude that extends to the
planet and its economies as well. Some of us, though, object to that idea
and will fight against it. Someday, with luck, the world as a whole will
decide to hold the companies behind large language models and related
technologies to a minimal ethical standard. Until then, though, this
behavior will continue, and we will have no choice but to defend ourselves
against it.

The LWN site is currently under high scraper load, so comment
display has been suppressed for anonymous users. If you are a
human, you may read the comments by clicking the button below:

Note: you can avoid this step in the future by logging
into your LWN account.