NFW Web Studio
News

Robots.txt checker: what can crawlers see on your site?

Enter a website address: we read its robots.txt and its sitemap. We tell you whether the files exist, which crawlers, AI ones included, are blocked, which sitemaps are declared and whether the page you entered is listed. The check runs from the queue of our full audit.

Free, no sign up. We check the page you give, not only the homepage.

What it checks

For robots.txt: whether it exists, whether it blocks every crawler or the AI ones, the access given to each AI crawler we know, the warnings and the declared sitemaps. For the sitemap: whether it exists, whether it is an index or a single sitemap, how many addresses it lists and whether the page is among them.

We do not test an arbitrary address against the file's rules: we read the file and report what it says.

How to read the result

A robots.txt that does not block every crawler, with a declared sitemap that lists your page, is the healthy picture. "Blocks every crawler" is almost always a mistake: it takes the site out of search engines.

"Blocks AI crawlers" is not a mistake: it is a choice, but it keeps you out of those assistants' answers, so check that it is intended. A page missing from the sitemap is a warning, not a fault.

One check only? Here is the full picture.

This tool reads one part of the audit. The full SEO audit measures the homepage with Lighthouse, checks whether AI assistants can read it and, for the cities we have scanned, compares it with their sites.

free SEO audit

// frequently asked questions

The short answers.

It is the file where a site tells crawlers what they may read. It can also point to where the sitemap is.
It is the list of pages you want search engines to find. It helps most on new sites and on sites with many pages.
Because many sites block them without noticing, through a CMS setting or a security service. The robots.txt rules are one of the points of our AI Crawlability Score.
No: this check reads the file and summarises it, it does not simulate a crawler on an address of your choice.
It is one part of the full audit, which checks robots.txt and the sitemap next to HTTPS, llms.txt and Lighthouse. For the whole picture, run the free SEO audit.
Yes: NFW Audit, a free browser extension for Chrome (and Brave and Opera), Firefox and Edge. It runs the audit and the single checks on the page you have open, with no account.

The report says something's off.
We can fix it.

SEO errors, slow pages, sites invisible to AI: these are exactly the problems we solve for businesses and professionals every day.