Crawl execution, history, and comparison
Bounded crawl
Traceable evidence
Comparable verification
Read the whole run lifecycle
Crawls records requested, reserved, dispatch-pending, dispatching, queued, discovering, crawling, rendering, analyzing, aggregating, completed, partial, failed, cancelled, balance-skipped, and expired outcomes. The UI groups these into accepted, worker pending, active, stopping, finished, or failed states without discarding the stored status.
Keep discovery and processing counters distinct
Requested cap, planned URLs, discovered URLs, admitted URLs, discovered-but-not-admitted URLs, queued, leased, completed, failed, skipped, and cancelled tasks answer different questions. Resource and external-link work have their own queues and can continue after primary page fetching.
Separate crawler problems from SEO findings
Dispatch delays, missing worker heartbeat, lease recovery, target throttling, timeouts, blocked requests, rendering saturation, materialization backlog, publication failure, and billing state describe collection reliability. They can reduce evidence and must not be rewritten as proof that a section has no SEO problems.
A terminal crawl can still be publishing
Completed and partial primary runs remain Processing until secondary publication finishes. The publication phase can include link graph, finding projections, issue summary, link metrics, aggregates, health score, AI-readiness evidence, and alerts. Comparisons and final issue totals should wait for this boundary.
Compare two published snapshots
The Compare view chooses the newest two published completed or partial runs by default. Source is the earlier run and target the later run. Failed, cancelled, balance-skipped, expired, or still-publishing runs are not valid comparison inputs.
The summary keeps crawled pages, Site Health when publishable, issue occurrences, affected URL instances, errors, warnings, and notices separate.
Choose the right comparison contract
| Kind | Boundary |
|---|---|
| Crawl to crawl | Two different completed or partial runs from the same saved Site Audit site. |
| Migration | Two runs with explicit URL-normalization and mapping rules for the migration question. |
| Staging to production | Explicit staging authorization plus a host or URL mapping tied to the selected runs. |
Make URL mapping reviewable
The stored mapping policy can strip or retain sorted query parameters, normalize trailing slashes, replace hosts, map path prefixes and locales, and define exact URL pairs. Defaults strip queries and normalize trailing slashes.
A convenient mapping can also merge distinct pages. Review the policy before interpreting added, removed, or changed evidence.
Know what the comparison can retain
Materialized deltas cover added and removed URLs; status, title, description, canonical, noindex, robots, and structured-data changes; new, fixed, regressed, and persistent findings; added and removed links; added, removed, and changed resources; and segment or tree pressure changes.
The union scan and persisted delta table are bounded. A truncated comparison reports that state; its retained rows are not proof that no other difference exists.
Health history has a stricter comparison rule
Two runs can be valid general comparison inputs while their Site Health points are not comparable. Health also requires matching score identities and scope, measured confidence, exact materialization, no truncation, and similar primary coverage.
History detail changes with age
Recent and pinned eligible runs can retain URL-level detail. Compacted runs keep summaries and trend evidence but no longer support every page, link, resource, or URL dossier lookup. The UI should show summary-only or expiry state rather than returning an invented empty result.