Allowed does not mean indexed.
The selected rules do not block this path. The page may still have a noindex directive, return an error or fail other indexing checks. The result does not confirm that a search engine has visited or indexed it.
Robots TXT Testing
Find the rule behind a blocked path.
Check a website’s robots.txt against a specific path and crawler. Understand what the result means before deciding what needs to change.
Test a crawl rule
Start with the affected URL
One website. One path. One crawler.
Enter the public website URL, then the page path beginning with / and a user-agent. The tool fetches that website’s robots.txt and reports the matching rule. Leave the path empty to review the fetched file and validation summary without a path verdict.
Crawl permission
Your rule check
Enter a website and a path to see which rule applies.
User-agent: Googlebot
Disallow: /private/ Crawl permission does not guarantee indexing. Robots.txt is not access control.
Read the rule in context
A folder can be blocked. A path inside it can be allowed.
A result makes more sense when you can trace it back to the file. In this simple example, the narrower Allow rule creates an exception to the broader restriction.
Fictional file · manually explained examples, not fetched results
For Googlebot in this example
/drafts/new-service/- Disallowed
Matches the broader /drafts/ restriction.
/drafts/example/- Allowed
The more specific Allow rule takes priority.
/services/- Allowed
Neither path rule restricts this URL.
Real files can have different groups for different crawlers. Repeat the check with the relevant user-agent and the exact path you are investigating.
What the verdict tells you
Crawl permission is one part of the story.
Disallowed does not mean hidden.
A crawler may be restricted from fetching the page, but the URL can still appear in search. Use proper authentication for private information; robots.txt is a public file, not a security boundary.
A failed fetch is not a verdict.
If the tool cannot retrieve the file, check the website URL and availability. Do not interpret a request error as proof that the page is allowed or blocked.
Turn the finding into a next step
Check the impact before changing the file.
A small edit can affect many URLs. Keep the investigation focused on what should be accessible and why.
Confirm the intended behaviour.
Identify which crawler needs access and whether this page should be crawled. Compare the verdict with that intention, rather than removing a restriction simply because it exists.
Review neighbouring paths.
Check another URL inside the folder, an intended exception and a page outside it. Plan the rule change with the person responsible for your website and test the proposal before publishing.
Recheck after an approved change.
Once the updated file is live, fetch it again and repeat the affected path checks. For an indexing concern, also inspect the page response and indexing directives; this tool does not perform a complete site audit.
Your questions
Before you change anything.
Google’s robots.txt rule specificationDoes this tool update my robots.txt?
No. It retrieves and checks the public file. Any approved edits must be made separately through the system that manages your website’s robots.txt.
Can I use Disallow to remove a page from search?
Disallow controls crawling, not reliable removal from search. A crawler needs access to see a page’s noindex directive. Choose the appropriate removal method for the content and its privacy requirements.
Should I test more than one user-agent?
Yes, when your file contains crawler-specific rules or the issue concerns a particular crawler. A result for Googlebot does not describe every crawler’s rule group or behaviour.
Does an allowed result mean the page will be indexed?
No. The result concerns the robots.txt rules checked for that path and crawler. Indexing also depends on access, page directives, canonical signals and the search engine's assessment of the page.
Can robots.txt keep confidential information private?
No. It is a public set of crawler instructions, not an access-control system. Content that must be private needs appropriate authentication and server-side access controls.
Should I recheck the file after a website migration?
Yes. Check the relevant live paths and crawler groups after the move, especially if staging used restrictive rules. Also verify page-level directives and access; robots.txt is only one part of the launch checks.
The rule looks right. The page still is not being found?
Tell us the affected URL, what you expected and what the check showed. We can help work through the technical SEO questions around it.
Talk through a crawl issue