Permission and page access
Free AI crawlability checker
Check robots rules, page directives and source content for a public URL. See which rules allow or block each named crawler token.
Use the AI crawlability checker
Your input
No account required. Fair-use limits apply. This check analyses the input you provide.
Check the details. Then check the answers.
See when AI names your brand and which pages it cites in a free visibility report.
Get your free reportRead the crawler rules for this URL
Include the path you want tested
Enter the exact public URL, including its path and query string. The tool fetches its origin’s /robots.txt and the page.
Compare policy and page content
See rule outcomes for seven crawler tokens alongside page indexing directives, source-text counts and main-content landmarks.
Review the matching restrictions
Use the rule evidence to investigate a block. Check any redirected URL separately, and preserve exclusions that are intentional.
The access table simulates rule matching. It does not prove a crawler visited the page or an AI answer cited it.
Understand the access decision
Allowed by robots is only the first question.
Different crawler product tokens can match different groups in the same robots.txt file. A page may allow one token and block another, while a page-level directive adds a separate indexing instruction. This checker makes those distinctions visible for the URL you submit.
The access table evaluates Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. These are seven product tokens, not seven AI engines. The tool also tries to inspect the page’s HTML and relevant directives, without impersonating those crawlers.
Permission, a response and readable content are separate checks
Follow the container-gardening URL from its tested path to the matching robots rule and the returned page.
https://example.com/guides/container-gardens
Use the exact path and query string. If the page redirects, check the final address separately for its robots policy.
A practical workflow
How to use the AI crawlability checker
Enter the complete public URL
Include the exact path and relevant query parameters. Use the final destination directly when you already know the address redirects.
Review each crawler row
Read Allowed, Blocked or Unknown with the selected groups and winning rule. An unresolved retrieval should not be treated as a confident permission decision.
Verify the remaining access layers
Check redirects, server logs, firewall behavior and browser-only content separately. Retest any changed robots policy against the affected paths.
One URL, separate questions
Read policy, path and response in context.
Follow the selected crawler group, matching path rule and page response as separate pieces of evidence.
Test the page that actually matters.
Robots matching uses the submitted path, including its query string. A result for the homepage cannot describe every directory and parameter on the same site.
- Originhttps://example.com
- Path/guides/container-gardens
- Robots filehttps://example.com/robots.txt
Only this submitted path is evaluated.
Interpret the table
Unknown is useful information.
A robots retrieval error, a server error or HTTP 429 leaves access unresolved in this tool. A 4xx response other than 429 is treated as an unavailable robots file permitting crawling under the evaluator’s policy. Read the retrieval finding before interpreting individual rows.
The table preserves the rule and group evidence so you can distinguish an intentional exclusion from a surprising match. Path matching is case-sensitive. Wildcards and an end-of-path dollar sign can materially change which URLs a rule covers.
Follow the final destination
Redirects can move the policy question.
The access table evaluates the originally submitted URL. If the page redirects, its final response may belong to a different origin with a different robots.txt file. The report calls out this distinction; run the final URL separately to review that destination’s policy.
A permitted rule does not bypass authentication, rate limiting or a firewall challenge. Nor does this tool reproduce every bot-specific response. Use server evidence and the relevant provider documentation when investigating an actual crawler incident.
Robots.txt checker
Test fetched or pasted robots rules against a specific URL and inspect the matching lines.
LLM HTML visibility checker
Inspect a page’s extracted source text, structure and script counts.
GEO visibility checker
Compare how three AI engines answer the same buyer question and describe your brand.
Further reading
Questions, answered
AI crawlability checker: your questions answered
A closer look at the inputs, method and next steps.
Does the tool send requests as each crawler?
No. It fetches robots.txt and simulates rule matching for the listed product tokens. It does not impersonate those crawlers or prove what their own infrastructure would receive.
Does Allowed mean my page will be cited?
No. Allowed describes the evaluated crawl policy for one path. Actual retrieval, indexing, model training and citation decisions are separate processes that this check does not measure.
Why can the same page allow one token and block another?
A more specific user-agent group can define a different policy from the wildcard group. Inspect the selected groups and matching rule in each row to understand the difference.
What does Unknown mean?
The robots policy could not be resolved confidently, for example after a network failure, server error or rate-limited response. Retry or investigate the endpoint before drawing an access conclusion.
Does this inspect every page on the domain?
No. It tests the submitted path and query string. Other directories, filenames and parameters can match different rules and should be checked separately when relevant.
Can robots.txt protect private information?
No. Robots instructions are voluntary crawl controls. Use authentication and authorization for private content; do not rely on a disallow rule to keep a public URL confidential.
Why should I test the final redirect URL?
The original access table does not switch policy origins after a page redirect. Testing the final URL separately exposes the policy that applies to the destination.
Navigate the AI landscape. Place your brand on the map.
See when AI names your brand, which competitors appear, and the pages those answers cite. Turn what you find into a practical next step.
Five AI engines. No card required.