SEO & web / Quick tool
Test one URL path against Allow and Disallow rules for a selected user agent.
Test one URL path against Allow and Disallow rules for a selected user agent.
Paste content, enter parameters or choose the files you need in “Robots text and test settings”.
Choose “Test path” for the job. Processing runs in this browser.
Review the result under “Match decision”, then copy or download it after checking the output.
Result guide
The tester evaluates one path against the matching user-agent group, then reports the longest matching Allow or Disallow rule. Equal-length matches prefer Allow. It supports standard prefix matching plus * and $ patterns.
Use a real user-agent token such as googlebot or bingbot and a path beginning with /. The result shows the selected group, applicable rule count, winning rule and specificity instead of returning only a generic allowed or blocked label.
A broad Disallow can be overridden by a longer Allow for the same path. The matcher supports prefix rules, * wildcards and the $ end anchor, and reports how many unsupported lines were ignored.
Robots.txt controls crawler access. It does not guarantee that a known URL disappears from search, and a blocked crawler cannot read a noindex directive on that page. Treat this as a deployment test, not an index-removal tool.
The most specific matching path wins. This local tester compares rule-path length and prefers Allow when equally specific rules match the same path.
Not reliably. Robots.txt controls crawling, not indexing. A blocked URL can still be known from links. Use an index-control method that crawlers can actually read when removal is required.
No. It covers standard prefix, wildcard and end-anchor behavior, but crawler-specific extensions or fetch conditions can differ. Confirm critical production cases with the crawler owner's tools.
Anonymous feedback
Found a problem or an idea to make this tool better? Tell us. We review every report and start working on actionable improvements within 2 days, continuously improving reliability and usability.