crawler — independent software & tools
-
AstroCrawl
Async web crawler powered by Playwright headless Chromium — JavaScript rendering, AI-powered extraction (OpenAI / Anthropic / Google), CSS rule engine with visual preview, and a full PySide6 desktop GUI. SQLite WAL crash recovery, proxy circuit breaker, RFC 9309 robots.txt, ReDoS-immune regex, 3,822 tests. pip install astrocrawl[full,gui,fast].
-
apify
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
-
apify-client
Apify API client for JavaScript
-
@crawlee/core
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
-
firecrawl-cli
Command-line interface for Firecrawl. Scrape, crawl, and extract data from any website directly from your terminal.
-
@mendable/firecrawl-js
JavaScript SDK for Firecrawl API
-
FireRed-OCR
Extract text from images using a robust OCR model designed for accuracy and efficiency in varied visual contexts.
-
google-account-automanager
🌐 Automate Google account tasks with FastAPI and Vue 3, including management, 2FA setup, and eligibility checks for efficient batch operations.
-
Scrapling
Simplify web scraping by extracting data from modern websites with an easy-to-use Python library designed for efficiency and clarity.
-
cna-oab-attorney-data-scraper
📊 Scrape attorney data from the OAB CNA Directory with this Python tool, collecting names, OAB numbers, and UFs efficiently and reliably.
-
Silent-Snake
🕵️♂️ Scrape WebApp content efficiently with Silent-Snake, inspired by Screaming Frog for deeper insights and better SEO analysis.
-
notion-md-crawler
A library to recursively retrieve and serialize Notion pages with customization for machine learning applications.