Crawl Scope Explained: Full Domain, Subdomain, Path or Single Page?

Every CrawlHawk crawl starts with one decision: how much of the site to crawl. This guide explains the five scope options, when to use each, and how each affects credit consumption — so you crawl exactly what you need, and nothing you don't. 500 URLs free, no signup.

Why Scope Is the First Decision

Scope defines the boundary of the crawl: which URLs the crawler is allowed to visit while following links from your starting URL. It matters twice. Completeness: too narrow a scope misses pages you meant to audit. Cost: too wide a scope crawls pages you never needed — and since every URL consumes credits, scope is also your budget control. The right scope depends on the question you're answering: "audit everything" is a different crawl from "list the images on this one page".

Full Domain — Crawl the Entire Website

The widest scope: every internal page reachable from the starting URL, across the main domain and its subdomains (www, blog, shop, docs). Use it for complete site audits, full XML sitemap generation, migration inventories, and any task where "did we miss anything?" is the failure mode. This is the default for broken-link checks and orphan-page detection, where completeness is the point. Credit note: on large sites this is naturally the most credit-intensive scope — check the page count estimate before starting, or narrow to a path first.

Main Domain Only — Exclude Subdomains

Crawls every page on the main domain but stops at subdomain boundaries: example.com is crawled, blog.example.com and shop.example.com are not. Use it when subdomains are separate concerns — a different team's documentation portal, a third-party-hosted shop, a staging subdomain — and you want the audit to reflect only the core site. It's also the practical choice when a subdomain is huge (a forum, a wiki) and would dominate the credit consumption without being relevant to the task.

Subdomain Only — One Subdomain, Nothing Else

The inverse: crawl a single subdomain (blog.example.com) and stay inside it. Use it to audit one property in isolation — the blog's internal linking, the docs portal's broken links, the shop's product URLs — without touching the rest of the organisation's web estate. Typical for teams that own one subdomain of a larger company site and only have a mandate (and a credit budget) for their own patch.

Single Path — One Section of the Site

Restricts the crawl to URLs under a path prefix, such as /blog/ or /products/. This is the credit-efficiency workhorse on large sites: a 100,000-page marketplace doesn't need a full crawl to audit its 2,000-page blog. Use it for section-level audits, category-level product extraction, and iterative work — fix a section, re-crawl the same path cheaply, verify, move on. It's also the recommended way to sample a very large site before committing credits to a full-domain crawl.

Single Page — One URL Only

The narrowest scope: exactly one URL, no link following. Use it when the question is about a page, not a site — every image on this page, every outbound link in this article, the product data on this listing. A standard page costs one credit, returns in seconds, and is the fastest way to test what CrawlHawk's output looks like before running anything bigger.

Which Scope Should I Choose? — Quick Reference

Your task	Recommended scope
Complete site audit, sitemap, orphan detection	Full Domain
Site audit, but subdomains are someone else's problem	Main Domain Only
Audit one property (blog, docs, shop) in isolation	Subdomain Only
One section of a large site; credit-efficient iteration	Single Path
One page's links, images or product data; quick test	Single Page

Scope and Credits

One standard URL costs one credit, in every scope — scope simply controls how many URLs the crawl can reach. Advanced modes cost more per URL (JavaScript-rendered or protected pages, and AI product extraction consume additional credits); the exact cost is always shown before the crawl starts. The free tier covers 500 URLs per crawl regardless of scope, which comfortably handles single pages, paths and small sites. For larger crawls, credit packs are pay-once and credits never expire, so an aggressive full-domain audit this month doesn't force a subscription for the quiet months after. See Pricing for current rates.

Frequently Asked Questions

How do I crawl an entire website?

Choose Full Domain scope and enter the site's homepage URL. CrawlHawk follows every internal link, including across subdomains, until all reachable pages are covered — up to and beyond 50,000 URLs.

Can I crawl just one subdomain?

Yes — Subdomain Only scope crawls a single subdomain (for example blog.example.com) and never leaves it.

What's the difference between Full Domain and Main Domain Only?

Full Domain includes subdomains (blog., shop., docs.); Main Domain Only stops at the subdomain boundary and crawls only the core domain. If your subdomains are separate systems, Main Domain Only keeps them out of your report and your credit bill.

Does a narrower scope cost fewer credits?

Indirectly, yes — credits are consumed per URL crawled in every scope, so a narrower scope reaches fewer URLs and costs less. A standard single page costs one credit; a path costs as many credits as pages under it, with advanced modes (rendering, AI extraction) consuming more per URL.

Can I change scope after starting a crawl?

Each crawl runs with the scope it was started with. To widen or narrow, start a new crawl — narrow-first is the credit-safe order on unfamiliar sites: sample a path, then commit to the full domain.

Which scope does the XML Sitemap Generator use?

Your choice — a sitemap can cover the full domain, the main domain only, one subdomain or one path, depending on which property the sitemap is for. Full Domain is the default for a complete sitemap.

Start crawling — 500 URLs free, no credit card required →