How CrawlHawk Works
Last updated: 10 July 2026
Every crawl follows the same workflow: pick a tool (sitemap, AI extraction, or custom crawl), set a scope (how much of the site to crawl), choose which link types and output format you need, then run. Credits are deducted per URL processed, results delivered in your chosen format ready for download or integration. Free tier covers 500 URLs without a credit card.
Step 1 — Choose Your Tool
CrawlHawk offers three main tools, each optimized for a different workflow. The XML Sitemap Generator crawls your domain (or selected scope), identifies every indexable URL, and produces a search-engine-ready XML sitemap following the sitemaps.org protocol. Output is accepted by Google Search Console, Bing Webmaster Tools, and major search engines without modification. Best for SEO teams generating sitemaps for new launches, post-migration rebuilds, or ongoing maintenance.
The AI Product Scraper extracts structured product data from e-commerce pages using an AI model that reads pages like a human would — no XPath selectors, no per-site recipes, no template setup. Extracts titles, descriptions, prices, variants, image galleries, SKUs, and other attributes across Shopify, WooCommerce, Magento, BigCommerce, and custom sites. Best for e-commerce migrations, supplier data imports, and competitive product research.
The Custom Link Crawler delivers raw link data for analysis — for each link discovered, you receive source URL, target URL, anchor text, HTTP status, link type, and rel attributes. Best for SEO audits, broken link maintenance, internal linking reviews, and any task that needs granular link data combining multiple link categories in one consolidated dataset.
Step 2 — Set the Crawl Scope
CrawlHawk supports five scope modes that control how broadly the crawler spreads from your starting URL: Full Domain (the entire site including subdomains), Main Domain Only (excludes subdomains), Subdomain Only (one subdomain in isolation), Single Path (one section, such as /products/), and Single Page (one URL only). For a full breakdown of when to use each — with a task-to-scope quick-reference table — see Crawl Scope Explained.
Scope determines credit consumption — a Single Path crawl on 1,000 URLs costs roughly 1,000 credits regardless of how large the rest of the site is. Picking the right scope for your task is the single biggest lever for credit efficiency.
Step 3 — Select Link Types
Choose which link categories the crawl reports. Combine any combination in one crawl — every link type is included in every plan. Internal Links are the foundation of internal linking audits and content silo analysis. External Links are outbound links to other domains, used for brand-safety audits and outbound traffic analysis. Broken Links are links returning 4xx or 5xx HTTP status codes, critical for technical SEO maintenance.
Orphan Pages are pages that exist but have no inbound internal links — invisible to most crawlers and hard for users to find. File Links include PDFs, DOCXs, ZIPs, MP4s, and other downloadable resources. Image Links cover every image URL across the crawled scope with alt text and dimensions where available. The resulting dataset shows each link with its full attributes, filterable by type, status, source page, or target.
Step 4 — Choose the Output Format
CrawlHawk delivers crawl results in seven formats — pick whichever fits your downstream workflow. CSV for direct spreadsheet review and analysis. JSON for programmatic processing and API integration. Excel XLSX for structured reports and pivot-table analysis. HTML for visual review and shareable rendering. PDF for client deliverables and archives. XML sitemap for direct search engine submission when sitemap generation is the goal. And clean, main-content Markdown (.md) for feeding crawled pages into AI and LLM workflows.
All formats include the same underlying dataset structure — the format choice does not change what data is captured, only how it is presented. You can export the same crawl in multiple formats without re-running it, which is useful when different stakeholders need the same data in different presentations (developer wants JSON, SEO consultant wants Excel, client wants PDF).
Step 5 — Run the Crawl
Click Run, and CrawlHawk begins processing. The crawler discovers URLs via breadth-first traversal — pages at depth 1 (linked directly from your starting URL) are crawled first, then depth 2, and so on. This means high-importance pages are processed early, and the crawl produces useful results even if interrupted partway through. The maximum crawl depth is configurable. Crawls can also be scheduled to run automatically at a chosen interval — daily, weekly or monthly — which turns any one-off audit into ongoing monitoring without manual re-initiation; scheduled runs consume credits the same way as manual ones.
For JavaScript-rendered sites (React, Vue, Angular SPAs, AJAX-loaded content), enable JavaScript rendering. This uses additional credits per URL because the page is fully rendered before content extraction — but it captures content that would be invisible to a raw HTML fetch. Many modern sites require this mode for accurate results.
Crawls run in the cloud and continue when you close the browser. Smaller crawls (a few hundred URLs) typically complete in 1-5 minutes. Mid-sized crawls (5,000 URLs) take roughly 15-45 minutes. Larger crawls (50,000+ URLs) may take several hours depending on target server response times and configured concurrency. Progress is visible in your dashboard, and webhook notifications fire when a crawl completes for integration into automated workflows.
How Credits Work
CrawlHawk uses a pay-once credit system — buy a credit pack, credits never expire, and you spend them as you crawl. Standard credit consumption is one credit per URL crawled. Additional credits are consumed per URL for JavaScript rendering (required for content loaded client-side via JavaScript) and for AI extraction via the AI Product Scraper (the AI processing is the most computationally expensive part of the pipeline).
Exact credit cost is shown before each crawl starts so there are no surprises. The free tier covers 500 URLs without a credit card or signup commitment — enough to run several proof-of-concept crawls. Credit packs are available from your account dashboard at multiple tiers. There is no per-seat licensing. See the Pricing page for current credit-pack pricing.
How AI Extraction Works
The AI Product Scraper uses a language model that reads pages like a human would, identifying product attributes regardless of the underlying HTML structure. This eliminates the need for per-site XPath selectors or scraping recipes — the AI works on a site you have never seen before, the first time, and continues working when sites redesign their layouts.
When you submit a product URL, the AI extracts product title and full description, price (including currency and any sale price), SKU and product identifiers, brand, variant data (sizes, colors, configurations), image gallery URLs, dimensions, weight, materials, specifications, and stock status — whatever is present on the page. Output is delivered as a row per product with each attribute as a separate column, ready for direct import into Shopify, WooCommerce, ERP systems, or PIM tools.
The AI works across Shopify, WooCommerce, Magento, BigCommerce, custom-built shops, and most marketplaces. The same AI handles different e-commerce platforms without per-platform setup or template selection — the structural variety in product pages is what the AI is specifically trained to handle.
Integration Options
CrawlHawk integrates with your existing workflows through several mechanisms. API access is available on every account for programmatic crawl initiation, status checks, and result retrieval — standard REST endpoints with API key authentication, callable from any HTTP client or workflow automation platform.
Webhook notifications fire when a crawl completes, with the result payload delivered to your configured webhook URL. This is useful for automating downstream processing — Slack alerts to your team channel, dashboard updates, automated import flows into your CRM or PIM, conditional re-pricing logic based on competitor data. Webhooks deliver structured JSON matching the format of the standard API response.
Scheduled crawls let you set a crawl to run on a recurring schedule (hourly, daily, weekly, or custom) without manual intervention. Useful for ongoing data imports, recurring SEO audits, or any task that benefits from regular automated execution. Scheduled crawls consume credits on each scheduled run, same as manual crawls.
Behind the Scenes — How a Web Crawler Works
A web crawler works in a simple loop: it fetches a page, parses the HTML to find every link on it, adds the newly discovered URLs to a queue, and repeats — expanding outward from the starting URL until the chosen scope is covered. Along the way it records what each page and link returned (status codes, link attributes, content), which is the raw data behind sitemaps, link audits and extractions. CrawlHawk runs this loop breadth-first, so the most prominent pages are processed first.
CrawlHawk runs on EU-hosted infrastructure based in Hungary, with GDPR-compliant processing and data retention. Our engine uses enterprise-grade fetching infrastructure for JavaScript rendering, IP rotation across global proxy pools, and handling sites with sophisticated bot protection. AI extraction is performed via OpenAI's API; OpenAI does not use API content for model training, and CrawlHawk does not opt in to training-data sharing — see the Privacy Policy for the full data flow.
Rendered HTML is available for download for 90 days after each crawl completes, supporting debugging workflows where you need to verify exactly what the crawler saw on JavaScript-rendered pages. Crawl results and reports are likewise retained for 90 days by default, with deletion available on request earlier. Credits, account data, and billing information follow standard GDPR-compliant retention policies.
Common Workflows
SEO site audit — Submit your domain as a Full Domain crawl with all six link types selected. Review the results in Excel: broken links to fix, Orphan Pages to surface, Internal Links patterns to optimize, External Links for brand-safety review. Re-run quarterly to measure improvements over time.
E-commerce migration — Submit your old product URLs to the AI Product Scraper. Export the structured product data as XLSX. Import into your new platform (Shopify, WooCommerce, Magento) with light column mapping. The same workflow handles catalogs from a few hundred to tens of thousands of products.
Competitor product research — Submit a competitor's product category URL with Single Path scope and AI Product Scraper enabled. The AI extracts all products in the category with full attributes. Schedule the same crawl to run daily or weekly for ongoing competitor monitoring without manual re-initiation.
Sitemap regeneration — Submit your domain to the XML Sitemap Generator in Full Domain scope. Download the XML output. Upload to Google Search Console and Bing Webmaster Tools. Schedule a monthly regeneration for sites with frequently changing content.
Pre-publication QA — Submit a draft page URL via Single Page with all link types selected. Verify every internal link points to a live page, every external link still works, every image loads correctly. Catches problems before publication rather than after.
→ Start your first crawl · See specific tools: XML Sitemap Generator · AI Product Scraper · Custom Link Crawler · Pricing: Pricing
Frequently Asked Questions
How long does a typical crawl take?
Depends on size and target server response times. A 500-URL free-tier crawl typically completes in 1-5 minutes. A 5,000-URL site takes roughly 15-45 minutes. A 50,000-URL site may take several hours. Larger crawls scale proportionally with configured concurrency settings. JavaScript rendering adds time per URL because each page is fully rendered before extraction.
What happens if I run out of credits mid-crawl?
The crawl stops at the credit limit and reports what was completed up to that point — no surprise overage charges. You can purchase additional credits and continue with a follow-up crawl on the remaining URLs. Partial results from the stopped crawl are downloadable in your chosen format.
Can I cancel a crawl in progress?
Yes — cancel from your account dashboard. Credits consumed up to the point of cancellation are not refunded, but no further credits are spent. The partial result up to cancellation is available for download.
Where is my data stored?
EU-hosted infrastructure in Hungary. Crawl results, including rendered HTML, are retained for 90 days post-crawl. Both deletable on request earlier. Full data flow details are in the Privacy Policy.
Do I need technical skills to use CrawlHawk?
No — the AI Product Scraper and standard crawl modes require no XPath, no selectors, no scripting. The interface is point-and-configure: submit URL, pick scope, pick link types, pick output format, run. For developers, the API enables custom integrations and workflow automation.
What happens if the target site blocks the crawl?
Some sites use aggressive anti-bot protection. Our engine handles most modern protections (rotating IPs across global proxy pools, browser fingerprinting, JavaScript rendering). Crawls on heavily protected sites may produce partial results — testing on a small sample before large crawls is the recommended approach. Crawl behavior is governed by the Acceptable Use Policy.
Is pricing per seat?
No — CrawlHawk has no per-seat licensing and no per-user fee. Credits belong to the account, and each account is designed for a single user.
Is web crawling legal?
Crawling publicly accessible websites is a long-established, widely used practice — search engines are built on it — but legality depends on what you crawl and how you use the results: the target site's terms, robots.txt, copyright, database rights and data-protection law all apply, and responsibility for each crawl rests with the user. CrawlHawk respects robots.txt by default; the rules are set out in the Acceptable Use Policy. This is general information, not legal advice.
Does CrawlHawk work on JavaScript-heavy single-page applications?
Yes — with JavaScript rendering enabled (additional credits per URL), the crawler captures content rendered client-side via JavaScript including React, Vue, Angular SPAs, and AJAX-loaded content. For static HTML pages, JavaScript rendering is unnecessary and saves credits.