Shypd.aiThe largest open index of AI toolsgpt-crawler is an open-source AI tool that crawls websites to generate knowledge files for creating custom GPTs. It allows users to extract content from one or multiple URLs and can be run locally or as an API.
Trending
About
gpt-crawler is an open-source tool designed to simplify the creation of custom GPTs by generating knowledge files from website content. Users can crawl single or multiple URLs, extracting relevant information to train their AI models. The tool offers flexible configuration options, allowing users to define the starting URL, match patterns for links, specify content selectors, and set limits on pages to crawl. It supports local execution, containerized deployment with Docker, and can also be run as an API server. The generated output, typically a JSON file, can then be uploaded to OpenAI to create custom assistants or GPTs, making it an efficient solution for developers and content creators looking to leverage their existing web content for AI applications.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
