crawler.sh

Crawl sites fast, extract Markdown and SEO insights.

What is crawler. sh? crawler. sh focuses on one job: crawling websites quickly from a local machine, then turning that crawl into structured data for SEO audits, content archiving, and sitemaps. It comes as both a desktop app and a command line tool, so it works for technical SEOs, developers, and marketers who prefer a visual dashboard. Key Features: High-speed site crawling: Crawls entire domains in seconds with configurable concurrency, depth limits, and polite delays so users can tune performance without hammering servers. Content extraction to Markdown: Automatically isolates main article content on each page and converts it to clean Markdown, with word count, author byline, and excerpt for consistent downstream use. Automated SEO analysis: Runs 16 checks per URL to flag missing titles, duplicate meta descriptions, noindex tags, thin content, long URLs, and similar issues, then exports them as CSV or human-readable TXT. Multiple export formats: Streams crawl results as NDJSON for pipelines, or exports as JSON arrays, Sitemap XML that follows W3C guidelines, and Markdown content archives. Desktop dashboard and CLI: The CLI provides crawl, info, export, and SEO subcommands, while the desktop app adds a visual dashboard, real-time crawl feed, SEO issues panel, and per-URL content previews.