What Data Scraping Actually Means

Data scraping is the automated collection of information from websites or online services — anything from product prices to news headlines to job listings. It's a legitimate and widely used technique in research, competitive analysis, AI training, and business intelligence. The ethical and legal questions arise not from scraping itself, but from what you collect, how you collect it, and what you do with it afterwards.

What Is Generally Permitted

Scraping information that is genuinely public and non-personal sits in relatively safe territory, provided you're not causing harm in the process. Courts in the US (hiQ v. LinkedIn, 2022) and guidance across the EU have generally supported the view that publicly accessible, non-personal data can be collected. That said, 'publicly accessible' does not mean 'freely usable for any purpose'.

  • Publicly listed prices, product names, or business contact details (non-personal)
  • News headlines or article metadata where no login is required
  • Government or open-data sources that explicitly permit reuse
  • Your own data, exported from platforms you use
  • Data collected via official APIs with appropriate permissions

What Is Clearly Off-Limits

Several categories of scraping are either unlawful, in violation of platform terms, or both. Mixing any of the following into your workflow creates real legal and reputational risk.

  • Personal data (names, emails, photos, addresses) without a lawful basis — this triggers GDPR in Europe
  • Content behind a login or paywall you haven't paid for
  • Data explicitly prohibited by a site's robots.txt or Terms of Service
  • Scraping at a volume or speed that degrades a site's performance (this can constitute a denial-of-service attack)
  • Re-selling or republishing scraped content that is copyright-protected
  • Using scraped data to train AI models where the source prohibits it

The robots.txt File and Terms of Service

A site's robots.txt file signals which parts of the site the owner does not want crawled. Ignoring it is not automatically illegal, but it is a strong signal of bad faith — and courts have taken note of it. More binding are a site's Terms of Service: if you've accepted them (by creating an account, for example), scraping in violation of those terms can expose you to breach-of-contract claims.

Always read both before you build a scraper. If a site offers an official API, use it. APIs exist precisely to allow controlled, legitimate data access — and they're far more stable than scraping anyway.

GDPR and Personal Data: The Hard Line

If you're operating in or targeting people in the European Economic Area, GDPR is non-negotiable. Scraping personal data — even from public profiles — without a clearly identified lawful basis is a regulatory violation. 'It was publicly posted' is not a lawful basis on its own. Several companies have received substantial fines for exactly this. If your scraping project touches names, emails, phone numbers, or any other data that can identify a living person, you need legal advice before you start.

Practical Steps for Ethical Scraping

  • Check the site's robots.txt and Terms of Service before writing a single line of code
  • Use official APIs wherever they exist — they're more reliable and clearly sanctioned
  • Rate-limit your requests so you don't overload the target server
  • Strip or anonymise any personal data you collect incidentally
  • Document your legal basis for collecting the data, especially if GDPR applies
  • Never scrape and republish copyrighted content without a licence
  • If you're unsure, contact the site owner and ask — many will say yes

The Honest Bottom Line

Scraping is not inherently unethical. Large parts of the modern web — search engines, price comparison tools, AI datasets — depend on it. What makes it ethical or otherwise is whether you respect the rights of site owners, the privacy of individuals, and the terms you've agreed to. The grey areas are real, and the law is still catching up with the technology. When you're uncertain, the safest and most professional move is always to ask permission or find an alternative data source.