Crawl execution, history, and comparison
Read the coverage and publication state of each crawl before treating their difference as an improvement or regression.
Bounded crawl
Traceable evidence
Comparable verification
Read the whole run lifecycle
Crawls records requested, reserved, dispatch-pending, dispatching, queued, discovering, crawling, rendering, analyzing, aggregating, completed, partial, failed, cancelled, balance-skipped, and expired outcomes. The UI groups these into accepted, worker pending, active, stopping, finished, or failed states without discarding the stored status.
Keep discovery and processing counters distinct
Requested cap, planned URLs, discovered URLs, admitted URLs, discovered-but-not-admitted URLs, queued, leased, completed, failed, skipped, and cancelled tasks answer different questions. Resource and external-link work have their own queues and can continue after primary page fetching.
Separate crawler problems from SEO findings
Dispatch delays, missing worker heartbeat, lease recovery, target throttling, timeouts, blocked requests, rendering saturation, materialization backlog, publication failure, and billing state describe collection reliability. They can reduce evidence and must not be rewritten as proof that a section has no SEO problems.
A terminal crawl can still be publishing
Completed and partial primary runs remain Processing until secondary publication finishes. The publication phase can include link graph, finding projections, issue summary, link metrics, aggregates, health score, AI-readiness evidence, and alerts. Comparisons and final issue totals should wait for this boundary.
Compare two published snapshots
The Compare view chooses the newest two published completed or partial runs by default. Source is the earlier run and target the later run. Failed, cancelled, balance-skipped, expired, or still-publishing runs are not valid comparison inputs.
The summary keeps crawled pages, Site Health when publishable, issue occurrences, affected URL instances, errors, warnings, and notices separate.
If settings changed between runs, review crawl boundaries and rules before comparing the affected counts.
Choose the right comparison contract
| Kind | Boundary |
|---|---|
| Crawl to crawl | Two different completed or partial runs from the same saved Site Audit site. |
| Migration | Two runs with explicit URL-normalization and mapping rules for the migration question. |
| Staging to production | Explicit staging authorization plus a host or URL mapping tied to the selected runs. |
Make URL mapping reviewable
The stored mapping policy can strip or retain sorted query parameters, normalize trailing slashes, replace hosts, map path prefixes and locales, and define exact URL pairs. Defaults strip queries and normalize trailing slashes.
A convenient mapping can also merge distinct pages. Review the policy before interpreting added, removed, or changed evidence.
Know what the comparison can retain
Materialized deltas cover added and removed URLs; status, title, description, canonical, noindex, robots, and structured-data changes; new, fixed, regressed, and persistent findings; added and removed links; added, removed, and changed resources; and segment or tree pressure changes.
The union scan and persisted delta table are bounded. A truncated comparison reports that state; its retained rows are not proof that no other difference exists.
Health history has a stricter comparison rule
Two runs can be valid general comparison inputs while their Site Health points are not comparable. Health also requires matching score identities and scope, measured confidence, exact materialization, no truncation, and similar primary coverage.
History detail changes with age
Recent and pinned eligible runs can retain URL-level detail. Compacted runs keep summaries and trend evidence but no longer support every page, link, resource, or URL dossier lookup. The UI should show summary-only or expiry state rather than returning an invented empty result.
Verify a remediation with both run IDs
Example: Checking a relaunch
Before a relaunch, a team crawls the staging site and compares it with production using a staging to production contract and a host mapping rule.
The comparison lists removed URLs without redirects and pages whose canonical changed. After launch, a crawl to crawl comparison confirms which issues were fixed and which regressed.
Frequently asked questions
Which runs can be compared?
Published completed or partial runs. By default the newest two are selected, the earlier as source and the later as target. Failed, cancelled or still publishing runs are not valid inputs.
Can I compare a migration with changed URLs?
Yes. The migration and staging contracts use mapping rules for query parameters, trailing slashes, hosts, path prefixes, locales and exact URL pairs.
Why is an old run shown as summary only?
Older runs are compacted: summaries and trends remain, but URL level detail for pages, links and dossiers is no longer available.