How web scraping, AI, and the EU have come together to sweep away fake discounts in Europe

Every year during the holiday season, unsuspecting European shoppers flock to stores (or their electronic devices) in search of good deals. What many of them don't realize is that the sales on offer are not what they seem.

Web scraping and AI are helping to identify fake discounts across the European Union

That's why, since 2017, Apify has been putting its web scraping powers to good use by using that technology and its platform to distinguish real discounts from fake ones. If you're unfamiliar with the term 'web scraping,' it's an automated method of extracting data from the web.

For about the same amount of time (from 2018), Apify has been teaming up with software house, TopMonks, and the data platform service, Keboola, to monitor the misleading practices in Czech online retail markets in late November every year. All three companies are working on the non-profit project Shop Watcher (Hlídač shopů), whose goal is to protect consumers from the misleading marketing tricks of large e-shops in the Czech Republic. Shop Watcher gives consumers clear charts of price changes in major e-shops. Thanks to a widely-used extension, the charts show up directly in their browser when viewing products.

Even before then (since 2014), the European Commission had been conducting sweeps in various sectors across Europe. If you're unfamiliar with that term, "a sweep," according to the EU Commission, "is a set of checks carried out on websites simultaneously to identify breaches of EU consumer law in a particular sector." These checks are performed by national consumer protection authorities under the coordination of the EU Commission and last year they focused on Black Friday sales.

That sweep was the result of Shop Watcher, Apify, and TopMonks coming together to develop a bespoke tool to help the European Commission and national authorities of the EU Member States monitor compliance with its consumer protection directive during the Black Friday season of 2022.

The challenge: monitoring prices of 16,000 products across 176 websites

The European Commission needed a tool to help national officials monitor the compliance of shops with the new European directive. In total, it needed to monitor 16,000 products in 176 e-shops throughout 11 EU member states and 2 EEA countries (Iceland and Norway).

I'm very pleased that we were able to capitalize on our team's experience on a project for the European Commission. Moreover, this was the first project where we could use not only our platform but also our new AI tool for generic price extraction. The results across Europe also showed that our work to cultivate the Czech e-commerce environment was not in vain. I'm even happier now that a large part of this year's Black Friday in Czechia already de facto followed the new directive.
Jakub Balada, Co-founder of Apify

The results: 43% of monitored stores offered discounts that breached the directive

The tool that Apify and TopMonks developed used artificial intelligence to collect prices for specific products, take screenshots of the relevant web pages, and automatically detect declared discounts that didn't comply with the new rules. The AI price extractor monitored prices during Black Friday and 30 days before the sales. In this way, it served a double purpose. On the one hand, it signaled to officials what they should look for; on the other hand, it provided an automated collection of evidence demonstrating behavior inconsistent with EU rules.

Chart demonstrating how prices fluctuate over time on online retailer websites

The chart above is an example of how breaches of the directive are identified. The blue line in the above chart indicates the product referral price. The red lines are the offered discounted prices that were scraped. We can see where the infringement of the EU regulation occurred. In early December, the referral price dropped to PLN 1,000, but after a couple of days it went up again for no apparent reason. That means the items on sale were not being sold at a discounted price, as claimed.

I'm proud of our small team that has managed to deliver a tool in a short time that builds on our years of experience in developing and running Hlídač shopů (Shop Watcher). At first, we reused just the chart, but in the background, we had our tools for data quality monitoring, our experience on how to do effective scraping at scale, and our platform for creating and running serverless solutions.
Ales Roubíček, Web Developer at TopMonks

The European Commission has now released the results that the tool delivered. Half of the products monitored were offered at a discount on a Black Friday sale. One in four shops failed to comply with the new rules, with 43% of the monitored stores offering discounts that breached the directive.

The results of this sweep clearly show, on the one hand, that many traders in Europe still display misleading discounts and need to align with the new requirements of EU consumer protection law as soon as possible. On the other hand, this sweep proves that advanced IT tools leveraging AI can be the key in detecting infringements of consumer protection legislation on a large scale. It is therefore essential to invest in advanced digital tools and appropriate enforcement techniques to protect consumers in digital markets.
Marie-Paule Benassi, Head of Consumer Enforcement and Redress Unit, DG Justice and Consumers, European Commission

What's next?

What will become of those shops that didn't meet the requirements? That depends largely on the consumer protection authorities in each state. Some traders may receive warnings, while others may face fines based on the evidence from the data collected.

This isn't the end of the EU Commission's Black Friday sweeps, much less the conclusion of the Shop Watcher project, but it could well be the beginning of the end for fake Black Friday sales and a new era of high levels of consumer protection across the European Union.