Site Audit
Begin with site and crawl setup so the audit covers the website boundary you actually want to investigate.
Bounded crawl
Traceable evidence
Comparable verification
Use Site Audit as an evidence loop
Choose the view that owns the question
| View | Question it can answer |
|---|---|
| Sites | Which saved property, authorization state, and latest run are in scope? |
| Overview | Where does the selected run deserve investigation? |
| Issues | Which recorded rules affect which URLs or resources? |
| Pages | Which crawled URLs match a reproducible technical filter? |
| Statistics | Which measured distributions point to a template or section pattern? |
| Structured Data | What markup, validation evidence, and feature profiles were recorded? |
| Performance | Which sampled or covered URLs have field, lab, or diagnostic evidence? |
| Internal Links | Which source-target relationships, occurrences, and graph signals were retained? |
| Compare | What changed between two published completed or partial runs? |
| Crawls | Did collection and report publication finish, and where did work fail? |
| Automation | Which recurring run and alert policies are active, skipped, or paused? |
| Settings | What data contract will future crawls use? |
Keep site, run, and evidence date together
Most views are run-scoped. The site selects the saved property; the run selects an immutable crawl snapshot. A new crawl does not refresh an old run, and opening an older run must not be reported as current site state.
Compare, Automation, Settings, and the Sites portfolio have their own scopes. Follow the site and run switchers instead of assuming every visible number shares one timestamp.
Authorization changes what the crawler may do
Verified and admin-approved sites can use the saved full-crawl policy. Unverified sites are limited to a public probe of at most 100 URLs and depth 2, with robots.txt respected, rendering disabled, conservative pacing, and a separate monthly quota. A blocked site cannot start a run.
Verification proves control of the saved host through a current challenge. It does not prove that every subdomain, login area, third-party resource, or environment belongs in the crawl.
Read collection quality before Site Health
A small issue total can mean a healthy site, a narrow crawl, blocked discovery, failed fetching, unfinished processing, or compacted historical detail. Check requested, discovered, admitted, completed, failed, skipped, and publication state before interpreting the result.
Site Health is published only when required categories, materialization, terminal state, and confidence meet the score policy. A missing score is an evidence state, not a score of zero.
Collection and report publication are separate stages
A run moves through request, dispatch, discovery, fetching, analysis, aggregation, and terminal states. After primary crawl work ends, link projections, findings, issue summaries, link metrics, aggregates, Site Health, AI-readiness evidence, and alerts can still be publishing.
A completed or partial label without secondary completion is shown as Processing. Wait for the published report before treating issue counts, health, comparisons, or alerts as final for that run.
Treat Site Health as a versioned summary
The current score combines seven categories: crawlability and availability, indexability and canonicalization, content and metadata, links and architecture, structured and international signals, performance and resources, and security and protocol. Category prevalence and severity shape the burden; verified blockers can cap the published result.
The score stores score, ruleset, eligibility, scope, coverage, confidence, exactness, and truncation context. Trend comparisons require matching versions and scope, measured confidence in both runs, exact materialization, no truncation, and no material coverage gap.
Move from summary to the owning evidence
- Issues groups recorded rules and movement, then opens affected evidence.
- Page Explorer answers bounded URL-set questions with server filters and pagination.
- Internal Links separates relationships from repeated occurrences and marks count exactness.
- The URL dossier keeps one URL and run across issues, links, assets, markup, changes, and technical facts.
- Structured Data and Performance expose their own coverage and missing-evidence states.
Use issues, pages, and links to move from the audit summary to the URLs and relationships behind a finding.
Visible rows can be a bounded projection
Tables, graphs, issue samples, URL details, histories, and exports have explicit read or retention limits. Counts can be exact even when samples are paginated; other views label lower bounds, truncation, incomplete catalog state, or missing detail.
Do not turn a loaded page, top list, or graph viewport into a sitewide claim. Use the displayed total, exactness flag, next page, saved filter, or export that belongs to the question.
Use history and automation for repeatable observation
Compare works from two published completed or partial runs and can retain URL, metadata, finding, link, resource, segment, and tree changes. Mapping rules are explicit for crawl-to-crawl, migration, or authorized staging-to-production work.
Automation can schedule daily, weekly, or monthly runs and evaluate active alert rules against a previous comparable run. A due schedule, accepted run, generated alert event, queued email, and delivered email are different facts.
For regular review, configure monitoring and alerts around the evidence someone is responsible for checking.
Know when detailed evidence expires
The newest ten terminal runs and every run no older than 30 days keep full detail. Older runs can be announced for compaction; from roughly six to twelve months only one completed or partial monthly anchor survives in compact form. Most runs older than twelve months are deleted, while the newest terminal result for a dormant site is protected.
Up to five finished runs per site can be pinned with a reason against intermediate compaction, but a pin does not override the twelve-month rule. Active comparisons, active exports, and unsettled billing also defer retention work. Billing evidence remains separate from crawl-detail retention.
Close the work with comparable evidence
A code change, ticket status, or higher score is not enough. Reuse an equivalent property, authorization, scope, rendering policy, and completion standard; check the exact issue or URL evidence; and look for regressions in related indexability, linking, rendering, performance, and structured-data signals.
Example: Checking a shop after a template release
A shop team ships a new product page template. After the release, they start a crawl limited to the /products/ path with the same URL limit and rendering settings as the previous run, so both runs describe the same scope.
The Issues view shows one rule on nearly every product URL: the canonical tag points to a filtered variant instead of the product itself. Because the finding repeats across one template, the team fixes the template once instead of editing pages one by one.
A comparison with the previous run then lists the issue as resolved while scope and coverage stay the same. That is what makes the result trustworthy: fewer issues from an unchanged crawl mean a real fix, not a smaller crawl.
Frequently asked questions
What is the difference between a full crawl and a Quick Check?
A full crawl requires a server-side verified property. A public Quick Check is deliberately restricted: at most 100 URLs, depth two, robots.txt respected, no rendering and low concurrency. Its results are useful for a first look but never represent full site coverage.
Why did my health score improve although nothing was fixed?
The score describes only the selected run. A narrower scope, blocked areas, failed fetches or changed limits can raise the score while shrinking the evidence. Check coverage first: below 70 percent evidence coverage the basis is not strong enough for a judgement, and between 70 and 85 percent it stays provisional.
How do I prove that a technical fix worked?
Run a new crawl with an equivalent scope and compare it with the run before the fix. The comparison shows which findings are new, resolved or changed. Only when scope and coverage match is a resolved finding real evidence.
Does a new crawl overwrite the previous one?
No. A new crawl creates a separate result. Full details remain for the ten newest runs with a final status and all runs up to 30 days old. Older runs can be compacted or deleted under the retention rules. Pin important runs to delay compaction, or export their data before the details expire.
What does a crawl cost?
Before a crawl starts, Crawl Foundry shows a cost estimate and your available balance. Usage charges and crawl limits both apply: the standard monthly allowance is 15 full crawls and 10 Quick Checks per organization, with URL and site limits depending on your plan. Check the effective limits in crawl setup before confirming.