Robots.txt checker: what can crawlers see on your site?
Enter a website address: we read its robots.txt and its sitemap. We tell you whether the files exist, which crawlers, AI ones included, are blocked, which sitemaps are declared and whether the page you entered is listed. The check runs from the queue of our full audit.
What it checks
For robots.txt: whether it exists, whether it blocks every crawler or the AI ones, the access given to each AI crawler we know, the warnings and the declared sitemaps. For the sitemap: whether it exists, whether it is an index or a single sitemap, how many addresses it lists and whether the page is among them.
We do not test an arbitrary address against the file's rules: we read the file and report what it says.
How to read the result
A robots.txt that does not block every crawler, with a declared sitemap that lists your page, is the healthy picture. "Blocks every crawler" is almost always a mistake: it takes the site out of search engines.
"Blocks AI crawlers" is not a mistake: it is a choice, but it keeps you out of those assistants' answers, so check that it is intended. A page missing from the sitemap is a warning, not a fault.
One check only? Here is the full picture.
This tool reads one part of the audit. The full SEO audit measures the homepage with Lighthouse, checks whether AI assistants can read it and, for the cities we have scanned, compares it with their sites.
The short answers.
The report says something's off.
We can fix it.
SEO errors, slow pages, sites invisible to AI: these are exactly the problems we solve for businesses and professionals every day.