Is crawler.sh free to use?

Yes, crawler.sh is a free tool available for download without requiring an account, offering both CLI and desktop app versions.

What platforms does it support?

It supports macOS (Universal binary for Apple Silicon and Intel), Linux (x86_64 and ARM64 via .deb package), and Windows (desktop app).

How many pages can it crawl by default?

By default, it crawls up to 100 pages, but this limit is configurable based on user settings.

What SEO checks does it perform?

It performs 16 automated SEO checks, including detection of missing titles, duplicate descriptions, noindex directives, thin content, and long URLs.

Crawler.sh

Free Local AEO & SEO Spider and a Markdown content extractor

Visit

What is Crawler.sh

Crawler.sh is a fast, local-first web crawler and SEO analysis tool that enables users to crawl entire websites quickly from the terminal or a desktop app. It performs automated SEO checks, extracts content as clean Markdown, and exports results in multiple formats like JSON, CSV, and Sitemap XML, making it ideal for technical SEO and content management workflows.

Key Features

Concurrent crawling with configurable depth, page limits, and delays

Automated SEO analysis with 16 checks for issues like missing titles and duplicate descriptions

Content extraction to clean Markdown, including word count and author byline

Multiple output formats: NDJSON, JSON, Sitemap XML, and SEO CSV/TXT

Real-time progress tracking with live streaming and dashboard visualization

Use Cases

SEO professionals conducting technical audits and identifying on-page issues
Content marketers extracting articles for repurposing into blogs or AI training data
Developers automating web scraping tasks for monitoring or data collection
Startups analyzing competitor websites for SEO insights and content gaps

Why do startups need this tool?

Startups need crawler.sh because it provides a cost-effective, local solution for conducting SEO audits and content extraction without high subscription fees. Its automation capabilities allow small teams to quickly analyze websites, identify technical issues, and gather content insights, which is crucial for improving online visibility and competing effectively in the digital space.