Scrape

Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling.

Join 0+ developers using this skill

skill

Browser Automation

intermediate

0 installs

View Source

Last updated: April 23, 2026

AI Skill Market Insights

Real data. Real impact.

Popularity

Rising

Emerging

Active Users

Developers

Quick Start

Manual Installation

No automatic installation available. Please visit the source repository for installation instructions.

View Installation Instructions

TEAR & SHARE

3-5hrs/WK

RISING

0.0K+ USING

Pre-Scrape Compliance Checklist

Before writing any scraping code:

robots.txt — Fetch
```
{domain}/robots.txt
```
, check if target path is disallowed. If yes, stop.
Terms of Service — Check
```
/terms
```
,
```
/tos
```
,
```
/legal
```
. Explicit scraping prohibition = need permission.
Data type — Public factual data (prices, listings) is safer. Personal data triggers GDPR/CCPA.
Authentication — Data behind login is off-limits without authorization. Never scrape protected content.
API available? — If site offers an API, use it. Always. Scraping when API exists often violates ToS.

Legal Boundaries

Public data, no login — Generally legal (hiQ v. LinkedIn 2022)
Bypassing barriers — CFAA violation risk (Van Buren v. US 2021)
Ignoring robots.txt — Gray area, often breaches ToS (Meta v. Bright Data 2024)
Personal data without consent — GDPR/CCPA violation
Republishing copyrighted content — Copyright infringement

Request Discipline

Rate limit: Minimum 2-3 seconds between requests. Faster = server strain = legal exposure.
User-Agent: Real browser string + contact email:
```
Mozilla/5.0 ... (contact: you@email.com)
```
Respect 429: Exponential backoff. Ignoring 429s shows intent to harm.
Session reuse: Keep connections open to reduce server load.

Data Handling

Strip PII immediately — Don't collect names, emails, phones unless legally justified.
No fingerprinting — Don't combine data to identify individuals indirectly.
Minimize storage — Cache only what you need, delete what you don't.
Audit trail — Log what, when, where. Evidence of good faith if challenged.

For code patterns and robots.txt parser, see

code.md

Scrape

AI Skill Market Insights

Quick Start

Manual Installation

TEAR & SHARE

Tags

Exa MCP Server

Be Part of the 0+ Developer Community

Pre-Scrape Compliance Checklist

Legal Boundaries

Request Discipline

Data Handling

Firecrawl MCP Server

Browserbase MCP Server

Apify MCP Server

Linkedin MCP Server

Channels

Learn

Compare

Company

Agents