Technical checks

Technical checks your own site from the outside. It reads the HTML each page serves, your robots.txt and your sitemap, and it simulates whether seven AI crawlers may read your site by your robots rules. Nothing on your site is changed.

Open Content in the left rail, then Technical. Come here after a site read or after you change your site. Technical is open on every plan, and a check is free.

What is checked

• AI crawler access. Whether seven crawlers may read your homepage, and each checked page, by your robots rules: Google, Bing, OpenAI training (GPTBot), ChatGPT search (OAI-SearchBot), ChatGPT browsing (ChatGPT-User), Claude (ClaudeBot) and Perplexity (PerplexityBot). Each reads "Allowed", "Blocked" or "Unresolved". It simulates your rules. No request pretends to be one of these crawlers.
• robots.txt. "robots.txt reachable", "robots.txt syntax" and "Sitemap declared in robots.txt".
• Your sitemap. For each sitemap file: "Sitemap structure", "Sitemap addresses", "Sitemap duplicates", "Sitemap host", "Sitemap dates" and "Sitemap size".
• Every page. The checks below, read from the HTML the page serves.

Keep three things in mind:

• Only the HTML source is read. JavaScript is not run, so text or a heading that a script draws is not seen.
• Where a check quotes a number, such as a title of 30 to 60 characters, a meta description of 70 to 170 characters or at least 100 words in the HTML, it is a writing guideline. Search engines can still rewrite a title or choose a different snippet.
• "Experience and reputation" and the four wording checks are always for information. The wording checks run only on pages in English.

How to run a check

Autopilot runs the checks on its own after each site read. To run them yourself:

1. Make sure the project has a website domain. Without one, Technical reads "No website to check." Add your website domain in project settings.
2. Open Content, then Technical.
3. Press "Check your site" the first time, or "Check again" after that.
4. The results land on the page when Autopilot finishes. On a first check, the page reads "Checking your site." until then. After that, the last results stay on screen under "Autopilot is checking your site again. These results are replaced when it finishes."

You can also ask Spot to check your site again. Spot prepares a change card, and the check starts when you press "Apply", unless your setting under Settings, Agents applies that kind of change at once.

Reading a result

The lead card comes first:

• "Readable by 5 of 7 AI crawlers by their robots rules" (or all 7, or none of the 7). "Which crawlers are blocked" opens their names.
• The counts, such as "3 to fix and 12 worth a look across 40 pages." or "Every check passes across 40 pages." When there are fewer things to fix than last time, the line opens with that: "2 fewer things to fix than at the last check."
• When and why it ran: "Checked 12 Oct 09:14 UTC after a site read" (or "after the site index", "on your request"), with the pages read and how many did not answer.

Under the lead, the row "AI crawler access to your homepage" reads "All 7 crawlers" or how many are blocked, such as "2 of 7 crawlers blocked". Opening it lists each crawler with its access. Then come the rows that need you. Each row shows:

• a status, the check's name and a hint beside the name that says what the check looks at and what it cannot tell you
• a scope line, such as "3 of 40 pages", "2 of 3 sitemap files" or "robots.txt"
• "What to do" (or "Details"), which opens the recommendation and up to six sample pages, most with one line of evidence such as "38 characters in the title" or "No canonical link"

The checks that pass and those for information wait in two closed groups, such as "12 checks pass" and "5 for information". "Show all pages" opens a table of every checked page, with "Page", "Answered", "To fix" and "Worth a look".

A row takes the status of its worst page, so one page is enough to put a check under "Needs a fix".

The lead can also read "Your robots rules could not be read" (crawler access is unresolved, not blocked), "Could not read your site" (no page answered, with the reason) or "No pages to check yet" (read your site first).

What to fix first

The page already sorts the work: "Needs a fix" before "Worth a look", and within each, the checks that touch the most pages first. A practical order on top of that (the first four show under "Worth a look", not "Needs a fix"):

1. Robots rules that could not be read. Until your robots.txt answers, access is unresolved for all seven crawlers. Fix the robots.txt endpoint and check again. A robots.txt that answers with a 4xx error, such as not found, counts as no file, which lets every crawler in ("No robots.txt (every crawler allowed)").
2. Pages that did not answer. Such a page is checked only for crawler access, so everything else about it stays unknown. "Show all pages" gives the reason, such as "Did not answer, answered HTTP 404".
3. Crawlers you did not mean to block. Blocking is your call. The recommendation is to allow a crawler only if your site should be readable by AI engines.
4. A noindex you did not mean. Under "Indexing directives", remove noindex only if the page should be discoverable.
5. "Needs a fix", from the top. Then the rest of "Worth a look" as you have time. Rows for information ask nothing of you.

The llms.txt draft

When a check finishes, Autopilot composes a draft llms.txt from your site index. It is a Markdown file with your site's name and a one-line description, then up to 20 useful pages. Docs, guides, help, about, product, service, pricing and reference pages come first. Sign-in, account, checkout and cart pages, and any address with a query string, are left out.

Press "Download llms.txt draft" to save it, edit it, and publish it at your root (/llms.txt) if you adopt the convention. llms.txt is a proposed convention: publishing it does not guarantee crawling, training use, retrieval or citations.

Limits

• One check runs at a time per project, and you can press "Check again" at most 3 times in an hour.
• A check covers the pages a site read has already read, up to 200. The check Autopilot runs on its own after its light read covers 40 of them. A check you start covers every page Autopilot has read so far, up to 200, including the pages later rankings read for topics with no page yet. To check more of your site, open "Set up" on Gaps and choose "Read your site" (Growth, Agency and Enterprise).
• Each page gets 8 seconds to answer. A slower page reads "took too long to answer".
• Up to 5 sitemap files are checked, and up to 2 MB of each is read.

Good to know

• The check time on the page is in UTC.
• If the page reads "Could not start." or "The checks did not finish.", press "Check again" to start it once more. The checks also run again after the next site read.