Web crawling and scraping tool with LLM-optimized output. 网页爬虫爬取工具 | Web crawler, web scraper, spider. DuckDuckGo search, site crawling, dynamic page scrapin...
Real data. Real impact.
Emerging
Developers
Per week
Open source
Skills give you superpowers. Install in 30 seconds.
Web Crawling 网页爬虫 | Web Scraping 网页爬取 | LLM 优化输出
智能网页爬虫和爬取工具，支持搜索、全站爬取、动态页面抓取。Free web crawler and scraper with LLM-optimized Markdown output.
pip install crawl4ai-skill
# Search the web with DuckDuckGo crawl4ai-skill search "python web scraping"
# Scrape a single web page crawl4ai-skill crawl https://example.com
# Crawl entire website / spider crawl4ai-skill crawl-site https://docs.python.org --max-pages 50
# Crawl documentation site with spider crawl4ai-skill crawl-site https://docs.fastapi.com --max-pages 100
爬虫效果 | Crawler Output:
# Search and scrape top results crawl4ai-skill search-and-crawl "Vue 3 best practices" --crawl-top 3
JavaScript 渲染的页面爬取（雪球、知乎等）：
# Scrape JavaScript-heavy pages crawl4ai-skill crawl https://xueqiu.com/S/BIDU --wait-until networkidle --delay 2
| 命令 Command | 说明 Description |
|---|---|
| Web search 网页搜索 |
| Web scraping 单页爬取 |
| Web crawling 全站爬虫 |
| Search + scrape 搜索并爬取 |
# Web Search 搜索 --num-results 10 # Number of resultsWeb Scraping 爬取
--format fit_markdown # Output format --output result.md # Output file --wait-until networkidle # Wait strategy for dynamic pages --delay 2 # Additional wait time (seconds) --wait-for ".selector" # Wait for specific element
Web Crawling 爬虫
--max-pages 100 # Max pages to crawl --max-depth 3 # Max crawl depth
智能提取，节省 80% Token。Smart extraction, save 80% tokens.
crawl4ai-skill crawl https://example.com --format fit_markdown
保留完整结构。Preserve full structure.
crawl4ai-skill crawl https://example.com --format raw_markdown
✅ 免费爬虫 Free Crawler - 无需 API key，开箱即用
✅ 智能爬取 Smart Scraper - 自动去噪，提取核心内容
✅ 全站爬虫 Site Crawler - 支持 sitemap，递归爬取
✅ 动态爬取 Dynamic Scraping - JavaScript 渲染页面支持
✅ 搜索集成 Search Integration - DuckDuckGo 搜索内置
No automatic installation available. Please visit the source repository for installation instructions.
View Installation Instructions
1,500+ AI skills, agents & workflows. Install in 30 seconds. Part of the Torly.ai family.
© 2026 Torly.ai. All rights reserved.