States worldwide are turning to technology to make the welfare state more efficient and mitigate welfare fraud. In the Netherlands, the state used a digital welfare fraud detection system called Systeem Risico Indicatie. The SyRI was a system that used personal data from different sources and uncovered fraud. In 2020, a Dutch court decided the SyRI legislation was unlawful because it did not comply with the right to privacy under the European Convention of Human Rights. This is among the first time a court invalidated a welfare fraud detection system for breaching the right to privacy. We analyze the judgment and its implications in a full paper; below are some of our main points.
The SyRI system and the court case
In 2018, six nongovernmental organizations formed a coalition to sue the Netherlands over SyRI. These were human and civil rights-related NGOs such as the Nederlands Juristen Comité voor de Mensenrechten, Platform Bescherming Burgerrechten, Privacy First and Stichting KDVP, along with the Netherlands Trade Union Confederation and De Landelijke Cliëntenraad, an NGO that represent clients at the national level for labor and income issues. Lastly, two famous Dutch authors joined the coalition: Tommy Wieringa and Maxim Februari.
The U.N. Special Rapporteur on Extreme Poverty and Human Rights Philip Alston submitted an amicus brief to the court in the SyRI case. The letter discussed the SyRI system in the context of the digital welfare state. It is unusual in Dutch court cases that an international organization or a U.N. official writes an amicus brief.
The judgment
In 2020, a Dutch court decided the SyRI legislation was unlawful because it did not comply with the right to privacy under the European Convention of Human Rights. According to the court, the SyRI system did not strike a fair balance between fraud detection and privacy. The court's most important reasons were the SyRI system was too opaque, it collected too much data and the purposes for collecting the data were not clear and specific enough.
Regarding transparency, the court mentioned the uncertainty regarding what SyRI was. There were no indications SyRI used deep learning or data mining. However, the court noted the SyRI legislation made it possible for the state to use such applications. Furthermore, the risk model and its indicators were not public. Additionally, the legislation contained no duty to inform people their data had been processed.
The court stated "the SyRI legislation in no way provides information on … which objective factual data can justifiably lead to the conclusion that there is an increased risk." Moreover, the court noted there was a risk SyRI was biased against people in lower-income neighborhoods. Because the working of SyRI was not revealed by the state, it could not be verified if SyRI used a discriminatory algorithm.
The court then assessed the data minimization and purpose limitation principles. The SyRI legislation was obscure about how much data could be collected and for which purpose the data was collected. The legislation was therefore not specific enough about the amount of data collected and the purpose for the collection of the data. The court's conclusion was that the right to privacy, as protected by the European Convention of Human Rights, had been violated.
Soon after the judgment, the Dutch government said it would not appeal the ruling. Therefore the judgment is final.
We show in the paper the immediate effects of the judgment are limited. The judgment does not say much about fraud detection and automated decision-making in general. A court might approve a similar system if the government ensures more transparency.
Still, the SyRI judgment is important. The judgment reminds policymakers that fraud detection must happen in a way that respects data protection principles and human rights, such as the right to privacy. The judgment also confirms the importance of transparency about how personal data is used.
Photo from Unsplash.com
