NFW-Audit, NFW Web Studio's scanner
If you found us in your site's logs, this page tells you who we are, what we read and how to stop us. We read public pages, one at a time and without hurry, and we stop when you ask.
What NFW-Audit is
NFW-Audit is NFW Web Studio's scanner. It does two things. First, on-demand SEO audits, which someone asks for a site and which read that site's main page. Second, periodic studies of Italian business websites, which the State of the web reports are built from.
We do not copy a site's content, rework it or use it to train models. We measure how a page is built (title, structured data, speed, security, files for search engines and for AI) and store the measurements.
How to recognise us
Our plain requests and the browsers we read pages with end their User-Agent with this mark:
NFW-Audit/1.0 (+https://www.nfw.run/state-of-the-web/about)
Which IP addresses we come from
The addresses below are where the scanner's requests leave from. If you want to allow or block us at network level, use these.
Lighthouse audits, the Google tool we measure speed and performance with, keep Lighthouse's own User-Agent (it contains "Chrome-Lighthouse") and leave from the same addresses. To block them as well, use the addresses, not the User-Agent.
| Machine | Address |
|---|---|
| Main machine | 169.58.174.135 |
| Helper machine 1 (Lighthouse only) | 138.199.155.90 |
| Helper machine 2 (Lighthouse only) | 2.28.129.215 |
What we read
For each site we scan we fetch public resources only:
- the main page, and for the periodic studies the first pages listed by the sitemap;
- robots.txt, the sitemap, llms.txt and the discovery files for AI agents (the ARD manifest);
- the page's assets (images, scripts, stylesheets), loaded by a real browser to measure how it looks and how heavy it is;
- Lighthouse audits, which reload the page several times to measure it.
How we behave
For each domain we make at most two requests at a time, at least a quarter of a second apart. If a site answers 429 (too many requests) or 503 with a wait time (Retry-After), we wait as long as it asks, up to two minutes, and retry at most twice. If the site keeps asking us to slow down, we give up and remember it: we do not insist.
We do not solve CAPTCHAs and we do not get around bot protection. A site that says no to us is counted as blocked.
What we respect
We follow rules that name NFW-Audit and the TDM-Reservation signal. We don't treat generic "User-agent: *" rules as addressed to us.
In practice, we stop if the site:
- has a robots.txt group naming NFW-Audit that disallows the page (we read robots.txt before the page, as RFC 9309 describes);
- declares the TDM-Reservation: 1 signal, through the HTTP header, the meta tag or a TDMRep file in /.well-known/;
- is on our exclusion list, fed by the requests we receive by email (see below).
How to ask us not to scan your site
The quickest way is to add this group to the site's robots.txt:
User-agent: NFW-Audit Disallow: /
Opt out and removal of data
You can also write to audits@nfw.run with the domain. We check the request comes from the site (an email address on the domain, or another way you suggest for us to check) and then: we stop scanning the domain, delete the data we stored and exclude it from any commercial contact from us. If the domain already has public audits, we take them down.
If you think a site was refused or scanned by mistake, write to the same address.
Sites we do not audit
We do not perform public audits of adult or gambling sites and the like (drugs, violence, hate, dating, malware and phishing, piracy). Every public audit has a page with the site's name on it, indexable and signed NFW, and we do not want to put those there.
The periodic city studies collect aggregate figures only and publish no single site's result.