Crawlee

Visit Tool

Crawlee is an open-source web scraping and browser automation library for Node.js, designed to build reliable crawlers. It extracts data for AI, LLMs, and RAG, supporting various file types and working with Puppeteer, Playwright, and Cheerio.

Claim this tool

No Views Yet

At a glance

Pricing

Open Source

Free tier

Yes

API

Yes

Skill level

Technical

About

What is crawlee?

Crawlee is a powerful, open-source web scraping and browser automation library built for Node.js, enabling developers to create reliable crawlers in JavaScript and TypeScript. It's specifically designed to extract data for AI, LLMs, RAG, or GPTs, supporting the download of HTML, PDF, JPG, PNG, and other file types. The library works seamlessly with popular tools like Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP, offering both headful and headless modes. Key features include integrated proxy rotation, session management, and a persistent queue for URLs, making it suitable for complex and large-scale scraping tasks while appearing human-like to bypass bot protections.

Best used for

Ideal for developers and data scientists who need to programmatically scrape data from websites, automate browser interactions, and collect information for AI, LLMs, or RAG systems. Especially valuable for building robust, scalable crawlers that can handle modern bot protections and integrate with various browser automation tools.

Common actions

scrape web data

automate browser tasks

extract data for AI

build web crawlers

automated workflowlow-code/no-codeopen-sourceworkflowsface swappingdeepfakecollaboration"AI Agents"github copilot

Capabilities

Key features

Web scraping library
Browser automation
Proxy rotation
Session management
Persistent URL queue
Headful/headless modes
TypeScript support

Target Audience

developerdata scientist

Integrations

puppeteerplaywrightcheeriojsdom

Pricing & Plans

Open Source

Free

FAQs

What programming languages does Crawlee support?

Crawlee is primarily a Node.js library, designed for use with JavaScript and TypeScript. It leverages the Node.js ecosystem to provide robust web scraping and browser automation capabilities for developers.

Can Crawlee bypass modern bot protections?

Yes, Crawlee is built to appear human-like and can fly under the radar of modern bot protections, even with its default configuration. It includes features like proxy rotation and session management to enhance its stealth.

What kind of data can Crawlee extract?

Crawlee can extract various types of data, including HTML content, PDF files, images (JPG, PNG), and other files from websites. It's particularly useful for gathering data to train AI models, LLMs, RAG systems, or GPTs.

Trending

Subcategories trending in Coding & Development

Open Source & Models Code Assistants DevOps & Infrastructure No-Code / Low-Code Testing & QA Backend & APIs

Trending

Also listed in

This tool also appears in

Data & Analytics › Web Scraping & Extraction

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce