Search
Discovery for search experiences
Read the search-token results separately from training tokens. An allowed rule does not guarantee retrieval or inclusion in an answer.
Separate the purposes
Make your crawler rules a deliberate choice.
AI search, model training and user-requested retrieval are different activities. Compare the robots.txt rules declared for your page path and understand the limits of those controls.
Compare crawler rules
AI Crawler Access Checker
Test the path you actually care about.
Enter a public page URL. The tool fetches that origin’s robots.txt and evaluates nine named tokens against the path and query. It does not fetch the page itself or impersonate those bots.
02 / Separate the purposes
Your crawler report
Enter a public URL to compare crawler rules.
Plain-text report
Check page-level indexing signals separately with the Indexability Checker.
Put it to work
Start with the purpose, then read the rule.
A single “AI allowed” label would hide important differences. The report separates four categories so you can review each deliberately.
Training
Potential use in model development
Review the relevant training-token declarations. This report cannot show whether content was used historically or undo past use.
Content-use control
A control is not always a separate crawler
Google-Extended is a content-use control, not a separate HTTP crawler. Do not read its result as a Google Search indexing decision.
User-requested retrieval
A person asks a service to fetch something
Provider behaviour varies. Some user-requested fetchers may not apply robots.txt rules, so a matching disallow is not proof of enforced access.
Illustrative example
Search and training can have different rules.
Illustrative OpenAI token example, not a universal policy recommendation. Review your full robots.txt file and current provider documentation before changing it.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /OAI-SearchBot
Search rule: allow
The search token has its own group. This permission does not promise citations or answer placement.
GPTBot
Training rule: disallow
The training token has a separate declaration. This is not evidence about historical use of your content.
The report evaluates declared rules. To understand actual requests, check server logs, access controls and provider guidance separately.
Before you move on
From a rule to a policy you understand.
Review the exact origin and path
Subdomains, schemes and paths matter. Test the specific public URL rather than assuming one homepage result describes every resource.
Read the selected group and matching rule
A named group is evaluated separately from the wildcard fallback. The report shows which rule was selected for the entered path.
Verify actual access separately
A firewall can block a permitted crawler; some retrieval services may not apply robots.txt. Review logs and access controls alongside the declared preference.
Your questions
Rules are one part of the picture.
Will allowing a search bot make my business appear in AI answers?
No. This report does not measure AI visibility, indexing, citations or rankings. It checks the declared robots.txt rules for the entered path.
Why is a result labelled Unknown?
A failed, missing or ambiguous robots.txt response does not produce a reliable permission conclusion here. Read the fetch finding and confirm provider handling separately.
Does this send my page to an AI model?
No AI API key is needed. The checker retrieves robots.txt using its own user agent and evaluates the rules. It does not send the page to an AI model.
Can robots.txt protect private content?
It is not authentication or an access-control barrier. Keep private resources behind appropriate access controls; a disallow rule is not a substitute.
Need a clearer policy across your website?
We can help review crawler declarations alongside your content goals, technical setup and actual request evidence—without treating access permission as a promise of visibility.
Send an enquiry