DaylytixThe crawler is the first step of every audit: it finds your pages and collects the data the checks run on. This page explains how it finds pages, what limits apply, how it treats robots.txt and parameters, and which settings you can change.
How pages are found
In crawl mode, Daylytix starts from the URL you enter, reads your XML sitemap (from robots.txt or the usual locations) and follows internal links, working outwards level by level until it reaches your page limit. In list mode it audits only the URLs you paste, with no link discovery.
- Internal vs external:
www.example.comandexample.comcount as the same site. Other subdomains, such asblog.example.com, are treated as external. - Query strings: internal links are normalised without their query string, so
/shop?sort=priceand/shopare crawled once. Parameter URLs are reported separately as possible crawl traps. - Page limit: the crawl stops at the Max pages to crawl value on the New Audit form (default 50), capped by your plan: 100 pages on the trial, 500 on Starter, 2,000 on Pro and up to 60,000 on Agency. There is no separate depth limit.
robots.txt
The crawler reads robots.txt to find your sitemaps and to report on it: whether it may block all crawlers, whether it lists a sitemap, which AI crawlers it allows, and which crawled URLs it blocks. It does not skip blocked URLs - it still fetches them so they can be audited, and flags them as blocked. If you don’t want certain sections crawled, use the exclude patterns below.
Settings you can change
On the New Audit form, under Advanced crawl options:
| Setting | What it does |
|---|---|
| User-Agent | Default is a desktop Chrome browser. Presets: Googlebot Desktop, Googlebot Mobile, Bingbot, Chrome Desktop, iPhone Safari, or a custom string. Available on every plan. |
| Delay between batches | A pause in milliseconds between batches of requests (0 = no pause). Use it for slow or rate-limited servers. |
| Include URL patterns | Regular expressions, one per line; only matching URLs are crawled. |
| Exclude URL patterns | Regular expressions, one per line; matching URLs are skipped. |
| Custom extractions | CSS selectors or XPath; the extracted value is stored for every page. |
Outside that section: Max pages to crawl, Capture page screenshots (adds about 1-2 minutes) and Client info.
Speed and server load
Pages are fetched by five workers in parallel, in batches, with no pause by default. On a fragile or rate-limited server, set a delay between batches. If a server answers 403 or 429, the crawler retries with other browser user agents before giving up on the page.
Data collected per page
For each page the crawler stores the status code, title, meta description, headings, canonical, robots directives, word count, a text excerpt, internal and external links, images, structured data and response details. Checks and reports are built from this data, so nothing is fetched twice.
Soft 404 detection
A soft 404 is a page that answers 200 OK but is really an error page. The crawler flags a page when it shows typical “not found” wording on a short page (under 200 words), or when it has almost no text and an empty or tiny title.
Live progress
While an audit runs, the dashboard shows live progress: queued, crawling, each check as it runs, and done. If you close the browser, the audit keeps running; open it again to catch up. You can also cancel a running audit.
How long it takes
It depends on the number of pages, how fast the site responds, the delay you set, screenshots, and which data sources are connected. Small sites usually finish in a few minutes; start with a small page limit on a new site to get a feel for it.