Thought for the week: Web scraping for generative AI is subject to the GDPR

On 7 July, the European Data Protection Board approved "Guidelines on web scraping in the context of generative AI." Perhaps not surprisingly, the EDPB considers that web scraping to support generative AI model development falls within the scope of the EU General Data Protection Regulation to the extent that it involves identified or identifiable data about natural persons. 

Although one would expect the EDPB to generally take that view, one might be surprised at the level of detail it expects a company to address in order to engage in such activities. Among other provisions, the guidelines cite a variety of specific obligations under GDPR Articles 5, 6, 9, 10, 14 and 89. 

Key data protection points in the guidance

The EDPB envisions that businesses will address data protection throughout the full web-scraping life cycle including initial design, data collection and processing, and secure destruction/anonymization. Several of the key points in the guidelines are as follows.

Understanding the company's role

Companies need to identify their data protection role in the context of the web scraping activities. The various options include that the company might be a controller of the underlying personal data, i.e., decides purposes and/or means of processing, or a data processor acting on behalf of controller, or a joint controller — jointly deciding purposes and/or means with another entity. 

While this definitional role might seem like a sleepy technical issue, it's quite important. If a company engages a vendor to do web scraping on its behalf, it needs to know the respective roles of each of the companies. If the vendor is identified as a processor in the contract and actually conducts the scraping and related processing on the company's behalf, the company — as the controller — is responsible for complying with the GDPR requirements described in the guidance. The vendor's direct GDPR regulatory duties might be more narrow and less comforting than otherwise thought.

Data minimization

The EDPB specifies that the controller must engage in a thorough data minimization assessment starting before any data collection begins. Key steps include designing precise data collection criteria, applying filters to avoid the collection of certain categories of data, such as sensitive data and excluding certain categories of websites, e.g., those directed to minors, or those opposed to scraping through CAPTCHA and other controls. Such data minimization also includes steps after collection and through the full life cycle of data, such as syntax-based mechanisms to filter out personal data, replacement of some or all real data with synthetic data, anonymization or pseudonymization of personal data, and the like.

Transparency

The EDPB recognizes that, depending on the context, it may be impossible or require disproportionate effort for a controller to provide a data protection information notice to all affected data subjects. The EDPB notes, however, that the controller must still publish a privacy policy and/or information notice on its website regarding its web scraping activities and adhere to additional appropriate measures and safeguards to protect the data subjects' interests. 

Legal basis for processing/safeguards

The EDPB recognizes that most companies will take the position that legal basis for processing is based on a sufficient legitimate interest of the controller. The EDPB emphasizes the controller needs to carefully document a legitimate interest assessment for this purpose that documents the controller's interest, the necessity of the processing and a balancing test that evaluates the opposing data protection interests of the data subject. The balancing test should take into account additional safeguards, such as measures to protect against "memorization" and "regurgitation" by the AI model. The EDPB specifies there are no blanket legitimate interest assessments for this purpose, as it must be performed on a case-by-case basis.

Special categories of data

The EDPB recognizes that, even if a controller does not intend to do so, it may "residually" engage in the collection and processing of sensitive personal data in the context of web scraping activities, e.g., social media posts that reveal political opinions or health information. The collection and processing of such sensitive personal data is generally prohibited, absent the data subjects' explicit consent — which is not feasible here — or other narrow grounds.

The EDPB expects the controller to incorporate controls across the full life cycle to address these sensitive data risks, from the definition of precise criteria before the collection, to the deletion of sensitive data from datasets immediately after collection, to steps during and after the development and deployment of the AI model, e.g., to updated or reinforced output filters, restrictions on prompts or functionalities, and the like.

Observations on what this means for companies in practice

The guidelines are in a consultation period until 30 Oct. 2026. Companies still have a chance to influence the final shape and scope of the guidance from the authorities on this important topic. It seems unlikely we could see major architectural changes in the final version of the guidance, although there could be some small, but important, modifications/clarifications. Assuming the final version is comparable to this draft of the guidelines, there are a few key takeaways to consider regarding how the EU data protection authorities — collectively — view this topic.

Web scraping in the context of generative AI model development is fully regulated under the GDPR

The EU data protection authorities fully expect controllers to comply with a wide range of GDPR requirements when engaging in web scraping to support generative AI model development. Controllers should not expect the EU data protection authorities would consider that such activity involves "publicly available" data that is outside the scope of regulation.

This guidance is not only relevant for the major developers of LLMs

This guidance is relevant for the major large language model developers, but also for all companies that perform web scraping directly or indirectly via vendors or others in order to fine tune AI models as part of their overall business strategy. Notably, elements of the guidance would also apply to web scraping for other purposes, such as to assess consumer sentiment or engage in other marketing activities.

Careful assessments and documentation are critical

Controllers engaged in web scraping activities would be well-served by a careful examination of the guidelines and the development of thoughtful legitimate interest assessment and other assessments throughout the data life cycle. Some of the most challenging aspects may relate to establishing sufficient controls for sensitive data and also assuring suitable controls against memorization and regurgitation. Companies should expect that, in response to any complaints from data subjects or a self-generated inquiry from an EU data protection authority, one of the first asks will be to provide copies of any legitimate interest assessment and other documentation.  

Extraterritorial web scraping likely to be fall under the territorial scope of the GDPR

Although the guidance does not speak directly to extraterritorial application, companies should expect that the EU data protection authorities would consider ex-EU web scraping that targets sites in the EU to be within the territorial scope of GDPR, particularly if the company has any subsidiary, affiliate or other operation in the EU, on the basis of an "in the context of an establishment" argument under Article 3(1).

On the last point, an entirely different set of challenges would relate to a scenario where the personal data processing activities in relation to web scraping would occur in whole or part outside the EU, as this could be interpreted to give rise to application of the cross-border data transfer restrictions in GDPR Articles 44-49. Plenty to unpack there, but we'll save that one for another time.