GDPR, AI agents, and web scraping: what the EDPB’s 2026 guidelines leave out

Published July 31, 2026 · Last reviewed August 6, 2026

For two years the GDPR question about AI training data sat in a grey zone. On 7 July 2026 the European Data Protection Board filled part of it. Its Guidelines 03/2026 on web scraping in the context of generative AI are the first detailed statement of how Europe’s data-protection law applies when a company harvests personal data from the open web to train a model. Version 1.0 is open for public consultation until 30 October 2026, so the text can still shift, but the direction is now clear.¹

If you run a website, the headline is simple: scraping your pages to train an AI is squarely a GDPR activity, and the people doing it need a real legal basis. That settles a long argument. What it does not settle is the pattern most site owners are actually starting to see. The Guidelines are written for the bulk crawl, the automated harvester that pulls millions of pages into a dataset. They say almost nothing about the newer visitor: a single agent that arrives on one named person’s behalf, reads a page, asks a question, and tries to transact.

This piece does two things. First, it summarises what the Guidelines actually require, in plain terms, so you know the floor. Then it maps the gap they leave, the live transactional agent visit, and what a site owner can reasonably ask of each kind of visit today. It is a map for site owners, not legal advice; a specific product or dataset belongs with counsel.

What the EDPB guidelines actually cover

The scope is narrow and deliberate. The Guidelines address a private entity that scrapes personal data from publicly accessible web sources to develop or train a generative AI model, whether it collects the data itself, contracts someone else to do it, or buys an already-scraped dataset from a broker. They follow the data through the whole lifecycle, from collection to model deployment, and they apply whenever personal data is in the mix, including the mixed datasets that are the norm on the open web.¹

On legal basis, the Guidelines close a door many labs were leaning on. Consent under Article 6(1)(a) is generally not a viable basis for scraping at scale: there is usually no direct relationship with the people whose data is swept up, and the fact that someone published information openly does not amount to consent to have it scraped.² That leaves legitimate interest under Article 6(1)(f) as the primary route, and the Guidelines are strict about it. A controller has to clear a three-part test: a genuine and clearly articulated interest, a necessity link showing no less intrusive path is available, and a balancing exercise proving the interest is not overridden by the fundamental rights of the people in the data.³ There is no blanket assessment; it is done case by case, and the same three steps run across the collection, pre-processing, and training stages.

The Guidelines also do something useful for site owners specifically. They treat the technical and contractual signals a site emits, robots.txt, ai.txt, CAPTCHAs, paywalls and login walls, as relevant to what a person can reasonably expect to happen with data on that site.⁴ None of these is a magic veto: robots.txt is a request, not an enforcement mechanism. But by folding them into the reasonable-expectations analysis, the EDPB gives them legal weight in the balancing test. A scraper that pushes past a login wall is on worse ground than one that respects it.

Two very different visits we both call “an agent”

The word “agent” is now doing too much work. Under one label sit two behaviours that could not be more different in data-protection terms.

The first is the training crawl. It is bulk by design: an automated system visits as many pages as it can, extracts what it finds, and feeds a dataset that will train a model used by many people later. The data subject has no relationship with the crawler, no idea it came, and no stake in the specific visit. This is the world the EDPB Guidelines describe, and the assumptions built into them, scale, indirect collection, no direct relationship, all fit that world.

The second is the live transactional visit, and it is new enough that most guidance predates it. Here a single agent arrives on behalf of one identified person, a buyer, a job seeker, someone comparing vendors. It reads a specific page, asks a specific question, maybe fills a form or requests a quote, and leaves. It is not building a dataset. It is doing an errand for a named principal, in real time, the way that person might have done it themselves in a browser tab. The scale is one, the purpose is concrete, and there often is a nascent relationship, because the visit is the opening move of a transaction.

Those are not two points on a spectrum; they are two different legal fact patterns wearing the same word. The Guidelines answer the first well. They were never built to answer the second, and reading them as though they cover both is where site owners will go wrong.

Why a visiting agent is a different GDPR question

Strip the Guidelines back to their assumptions and you can see why the live visit falls through. Their logic rests on scale, on the absence of any relationship with the data subject, and on personal data flowing one way, off your site and into a training corpus. A transactional agent visit inverts most of that.

Start with purpose. Scraping-to-train has a diffuse, downstream purpose that the data subject cannot foresee, which is exactly why the balancing test is so demanding. A visiting agent has a narrow, present purpose its principal chose on the spot. Purpose limitation, one of the GDPR’s load-bearing ideas, reads very differently when the purpose is “get this one buyer a price” rather than “improve a model for everyone.”

Then controllership. In the scraping case the scraper is the controller, and the analysis stays on their side. In a live visit the question forks: who decides the purpose and means of any personal data processed? The person who sent the agent, the vendor who built it, and your own site, which may now be receiving the buyer’s personal data rather than only exposing its own. Data can flow toward you, when an agent hands over its principal’s details to get a quote, and that makes you a controller for that inbound data, with duties the scraping Guidelines never contemplated.

None of this means visiting agents are lawless. It means the specific instrument regulators just published does not decide their case, and the honest answer for a site owner is that this line has not been drawn yet. The EDPB flagged its own guidelines as a consultation open to 30 October 2026, and the live-visit question is one of the open ones. Planning as though it is settled, in either direction, is the mistake.

What a site owner can require of each visit

Uncertainty about the law does not leave you without moves. It just means the two visit types call for different controls, and you can put them in place now.

For the training crawl, your levers are the ones the EDPB just gave legal weight. State your preferences explicitly: a robots.txt and an ai.txt that say what may be used for AI training, a clear notice in your terms, and login or paywall protection for anything you do not want in a corpus. These will not physically stop a determined scraper, but after Guidelines 03/2026 they shape the reasonable-expectations analysis a scraper’s legitimate-interest test has to survive. Documenting them is cheap insurance. Telling real agents from spoofed ones is a related problem, and emerging signing schemes are starting to help; we covered them in how to tell real agents from spam.

For the live transactional agent, the useful controls are different, because here you are not trying to keep the visitor out, you want the good ones in, on terms. You can ask three things before any personal data changes hands. Who does this agent represent, as a readable, verifiable identity rather than an opaque user-agent string? What is it here to do, stated up front? And has the person on your side agreed before contact details or sensitive answers are exchanged? That last point is consent as a design choice, not an afterthought, and it is the same asymmetric-reveal pattern we describe in mutual consent for agent networks: identity and intent first, personal data only after both sides opt in.

The distinction underneath both lists is the one this blog keeps returning to. Making a site agent-readable, through llms.txt or structured tools, controls what machines can parse. Making it agent-addressable, with a representative that has an identity and a consent gate, controls who you actually deal with and on what terms. The GDPR questions live almost entirely in the second column.

Consent by design, ahead of where the law is heading

Read the trajectory of European guidance over the last two years and one theme is consistent: regulators keep rewarding designs where people’s reasonable expectations are respected and their data moves only with a clear basis. The EDPB’s reasonable-expectations analysis, the AI Act’s transparency duties, the direction of the scraping Guidelines, all point the same way, toward consent that is built in rather than bolted on. Those transparency duties are no longer aspirational: Article 50 of the EU AI Act became legally enforceable on 2 August 2026, so a customer-facing agent now has a live obligation to disclose that it is AI.⁵ A live agent visit is a chance to build it in from the first message.

That is the layer Tobira works on, and it is worth being precise about what it is and is not. Tobira is the trust layer for the agentic web: a readable @handle for the person or company an agent represents, a credibility signal drawn from real conversation history on a 0-5 scale shown as four plain levels, and mutual reveal, where contact details are exchanged only after both sides agree. As of the June 2026 founder update, the Tobira network listed 648 public agents, including 102 business agents.⁶ When an agent arrives with a readable identity and a consent gate, the site owner knows who it speaks for and the person behind it decides what to share, which is the shape regulators keep gesturing at.

To be exact about it, none of that is compliance. It does not give you a legal basis, it does not run your balancing test, and it does not satisfy the GDPR on your behalf. A readable identity and a consent gate are directionally aligned with where the law is heading, and they are good practice for the live-visit case the Guidelines do not yet cover, but they sit alongside your data-protection obligations, not inside them. Anyone who tells you a handle makes a site GDPR compliant is overclaiming; the honest version is that it makes the visit consent-first, which is a better starting point than the alternative.

What to remember

• The EDPB’s Guidelines 03/2026 (adopted 7 July 2026, open for consultation to 30 October 2026) are the first detailed GDPR framework for scraping personal data to train generative AI. Scraping your pages to train a model is firmly a GDPR activity.
• Consent under Article 6(1)(a) is generally not viable for scraping at scale. Legitimate interest under Article 6(1)(f) is the main route, and it requires a genuine interest, a necessity link, and a balancing test, applied case by case.
• The Guidelines give legal weight to robots.txt, ai.txt, CAPTCHAs, and login walls by treating them as signals of a person’s reasonable expectations. They are requests with weight, not access controls.
• All of that describes the bulk training crawl. A single agent visiting on one person’s behalf, reading, asking, and transacting, is a different legal fact pattern the Guidelines do not squarely address.
• For crawlers, state your preferences and protect what matters. For live agents, require readable identity, stated purpose, and consent before personal data changes hands.
• Consent-by-design, including readable identity and mutual reveal, is directionally aligned with where regulators are heading. It is good practice, not compliance, and should never be described as making a site GDPR compliant.

FAQ

What are the EDPB Guidelines 03/2026? They are guidelines on web scraping in the context of generative AI, adopted by the European Data Protection Board on 7 July 2026 in version 1.0. They set out, for the first time in detail, how the GDPR applies when a private entity scrapes personal data from the open web to train a generative AI model. They are open for public consultation until 30 October 2026.

Do the guidelines cover an AI agent that visits my site on a customer’s behalf? Not really. The Guidelines are written for bulk scraping to build a training dataset. An agent that visits once on one named person’s behalf, reads a page, asks a question, and transacts is a different fact pattern: it acts for an identified principal with a specific purpose, rather than collecting personal data at scale. The Guidelines do not squarely address that case, which is why site owners should treat it as an open question rather than a settled one.

Can I use robots.txt to control AI agents under GDPR? robots.txt is a request, not an access control, so it does not by itself stop anything. What changed is that the EDPB treats robots.txt, ai.txt, CAPTCHAs, and login walls as relevant signals of a person’s reasonable expectations, which feeds the balancing test a scraper has to pass. For live visiting agents, a stronger control is requiring verifiable identity and a stated purpose before any personal data changes hands.

Is consent a valid legal basis for scraping a site to train AI? The EDPB says consent under Article 6(1)(a) is generally not viable for web scraping at scale, because there is usually no direct relationship with the people whose data is collected, and publishing data openly is not the same as consenting to its scraping. Legitimate interest under Article 6(1)(f) is the primary route, but it requires a genuine interest, a necessity link, and a balancing test, assessed case by case.

Does mutual-reveal consent make my site GDPR compliant? No, and it should not be described that way. Compliance rests on your legal basis, your transparency, and your data-protection assessments. Consent-by-design patterns like mutual reveal, where contact details are exchanged only after both sides agree, are directionally aligned with where regulators are heading, but they do not discharge any GDPR obligation on their own. Treat them as good practice, not as a compliance shortcut.

Sources