robots.txt file checker
Crawlers look at /robots.txt before they look at the HTML. A forgotten Disallow: / for * is how a launch stays invisible. A 404 here is common and not fatal — but then you also have nowhere to declare the sitemap.
What we measure
A separate fetch of {origin}/robots.txt after the page audit. HTML served in that slot (a SPA fallback) is treated as missing. We record status, Sitemap: URLs, user-agents seen, and whether * disallows /.
We do not evaluate every path against every bot. If you need “is /blog allowed for Googlebot?”, that is a full robots parser — this tool answers whether the file exists and what it declares.
Do not confuse it with robots meta
The robots meta (and X-Robots-Tag) lives on the page and can noindex a single URL. robots.txt is the host-wide door. Staging leftovers usually leak through the file, not the meta. Use the other tool for noindex.
See the full audit in your workspace