How we measure business websites

Where the businesses come from

The businesses come from Overture Maps, an open dataset of places and businesses that several maps also use. We count the ones whose point falls within the city's administrative limits, the municipality rather than the province, after leaving out the categories that are not relevant to an online presence.

Each business's website is the one listed on its entry. This is not the official business register: these are the businesses a customer finds on a map, with the details the map carries.

Where the data comes from

The businesses and the website they list come from Overture Maps, an open dataset of places. We use no other source to find websites: every site analyzed is the one listed on the business's Overture entry.

Places have no single licence for the whole theme: each provider has its own, all permissive. CDLA Permissive 2.0 for Meta, Microsoft, PinMeTo, Krick, RenderSEO, DAC and BrightQuery, Apache 2.0 for Foursquare (whose data was transformed to Overture's schema) and CC0 1.0 for AllThePlaces.

Administrative boundaries (the divisions theme) are licensed ODbL and come from OpenStreetMap, geoBoundaries, Esri Community Maps and LINZ. That is why our reports carry the line Overture gives for data that includes OpenStreetMap.

Overture's page states no sharing or derived data obligations beyond that line: we add no conditions that are not there, and refer to each source's own licence for the details.

Attribution: © OpenStreetMap contributors, Overture Maps Foundation

How a business gets its category

Every Overture business carries a base category (basic_category). We assign it to one of the report's sectors, following the table below. A business's sector depends only on that category: we do not pick it by hand, business by business.

Before assigning sectors we drop the categories that are not relevant to an online presence: places of worship, parks, hospitals, schools, banks, fuel stations, government offices, museums and stations. A category that appears in no group goes to "other", which the reports do not show. The table is the one of release 2026-08-19.0 of the data.

What counts as a working website

A website works when at least one of its pages answers without an error. If our tool stops halfway, because of a timeout or a Lighthouse error, the site still counts as working: that is our limit, not a problem with the site.

A social profile, an app, or a page on a booking, delivery or marketplace platform does not count as a website, even when the business lists it as one. Every figure about the web covers working websites only.

Businesses and sites: two different units

Businesses are counted one by one. Everything about the web is counted by site: a shopping center whose 40 stores list the same website counts as one site, not 40. Basic structured data is the exception and is counted by business, because it describes each business.

Every figure says how many sites or businesses it is based on. A measurement that failed on a site does not count as a failure: that site is left out of that figure. That is why figures in the same report can cover slightly different numbers of sites.

Quick scans and full scans

In a quick scan the crawler reads each site's home page and the first 4 pages of its sitemap. Lighthouse and technology detection (Wappalyzer, after the JavaScript has run) run on the main page only.

In a full scan the crawler reads more pages per site; Lighthouse stays on the main page, so scores from different cities remain comparable.

Lighthouse and Core Web Vitals

Lighthouse is the Google tool behind PageSpeed Insights. It simulates a mid-range phone on a slow mobile connection. These are lab measurements, the same for every site, not the times of real visitors: they are for comparing sites with each other, and they are longer than on a computer on a fiber connection.

Since 29 September 2026 every page is measured in rounds of 3 runs and we publish the median, with a CPU slowdown calibrated on PageSpeed Insights (explained in the two sections below), so the scores compare with its own. Before, it was a single run with Lighthouse's default slowdown (4x), which gives longer times and lower scores, the blocking time above all: scans made before that date, such as Verona's of September 2026, compare with each other, not with today's audits. From that date we also record Speed Index and Total Blocking Time.

Scores run from 0 to 100, in Lighthouse's bands: 0 to 49 poor, 50 to 89 needs improvement, 90 to 100 good. For times we use the thresholds Google publishes for the Core Web Vitals and, for Speed Index and blocking time, which are not Core Web Vitals, the bands Lighthouse scores them on. Interactivity (INP) cannot be measured in the lab, so it does not appear.

The machines that measure, and calibration

Every scan puts the pages to measure on a shared queue. Scans requested by a person waiting for the result, such as a public audit, go ahead of all the others.

The queue is served by a main machine, the one the scan runs on, and by any number of helper machines that only run Lighthouse. Each machine has several slots, and each slot uses 2 processor cores and a Chrome browser of its own, so two measurements never compete for the same processor. On the main machine 2 cores stay free for the rest of the scan. If a helper machine stops, another one picks up its work.

A slower machine would score the same site lower, so before it measures anything every machine, the main one included, is calibrated on PageSpeed Insights, the score a site's owner sees. It picks the CPU slowdown that makes its scores match PageSpeed's: the reference is the median of 5 calls to PageSpeed Insights, 90 seconds apart, on 12 steady home pages. A slower machine is never a reason to turn it away; the only reason is that no slowdown brings it close to PageSpeed, and then it measures nothing and tries again after an hour. Calibration is repeated when the hardware or the Lighthouse version changes, after a week, and when the machine slows down steadily.

Real measurements never call PageSpeed Insights: it is slow and a third party service. We use it only to calibrate.

Runs and the median

A measurement is made in rounds of 3 runs. A run is set aside when the processor was struggling, never because of its score: that happens when Lighthouse's benchmark index is under 85% of the machine's normal level, or when more than 10% of the cores' time was taken by other users of the same virtual machine.

At the end of a round, if there are at least 3 good runs the measurement is done; otherwise another round starts, possibly on another machine, up to 3 rounds, which is 9 runs at most.

The published result is the median of the good runs: the run in the middle, taken whole, so its figures add up to its score (with an even number of runs, the one just above the middle). Not the best one: the maximum grows with the number of runs, and the luckiest run, for example one where a third party script did not load, is often the best. When fewer than 3 runs are good, we publish the median of the ones we have and the result is marked as degraded.

Averages, medians and percentiles

The median is the value that splits the sites in half: half above, half below. The 25th and 75th percentiles hold the middle half of the sites, the 10th and 90th leave out the extremes. Percentiles are interpolated between the two closest values, as SQL's percentile_cont does.

Averages and shares are rounded to one decimal. When a report shows both the average and the median, the gap between them tells how much the extremes weigh.

Basic structured data

A business has complete basic structured data when the pages of its site carry two JSON-LD blocks: a LocalBusiness, or one of its subtypes (Restaurant, Dentist, Store...), with name, an address structured as a PostalAddress, telephone, URL and opening hours; and a WebSite with name and URL.

Each block can appear on any of the analyzed pages. A block missing even one of these counts as incomplete. An address written as free text does not count: Google would have to interpret it.

Sitemap, canonical and Search Console

A sitemap is valid when we find it, at the Sitemap line of robots.txt or at a common location, and it is a list of URLs, a urlset or a sitemap index. The canonical is the address the home page declares as its own main version.

Search Console counts as verified when we can see it from outside: a meta tag on the home page or a DNS record. Owners who verified it another way (a file uploaded to the server, Google Analytics or Tag Manager) are invisible to us, so the real share can be higher than the one we report.

The AI Crawlability Score

The AI Crawlability Score says how well an assistant such as ChatGPT, Claude or Gemini can read a site and tell what the business is. It runs from 0 to 100 and is the weighted average of eleven checks, explained one by one on its own page, together with the versions of the rules. A check that could not be verified does not count as failed: the site is judged on the others.

The ARD (Agentic Resource Discovery) manifest, one of the eleven checks, is looked for in this order: the Agentmap line of robots.txt, a rel="ard" or rel="ai-catalog" link in the home page's HTML as the server sends it, the same links in the home page's HTTP headers, and only with none of these the files /.well-known/ard.json and /.well-known/ai-catalog.json. The first one named explicitly decides which manifest we judge, even when it then fails to open. We validate it with the same conformance test Lighthouse uses in its Agentic Browsing category, the one PageSpeed Insights shows too, so our verdict matches its own. One difference: to Lighthouse a site without a manifest is not applicable, to us it fails the check.

Spam blocklists

We check the IP address of each working site's server against three blocklists: Spamhaus ZEN, Barracuda and SpamCop. A site is blocklisted when at least one of the three lists its server. Spamhaus's policy lists, which only flag addresses that should not send email directly (PBL), do not count. When a list does not answer, the site stays without an answer.

Being blocklisted is about email, not the website: email sent from that server is likely to land in customers' spam folders and never be read, while the site usually keeps working and showing up on Google as before. The server is often shared with hundreds of other sites, and the flag comes from one of them.

The national average

The national average is not an official statistic: it is the same report computed over every city in a country we have scanned up to that month, using each one's latest finished scan. A business found by several scans counts once.

We only show it once it covers at least two cities, all scanned on the same scanner version as the report: with one, it would compare the city with itself, and another version measures differently, so its figures do not compare. A patch release (1.2.1 after 1.2.0) changes no measurement and compares with its own minor. A value is in line with the average when the difference is within 5% of the average; beyond that, we say higher or lower.

How often we scan a city

A city can be scanned at most once a month. Each scan keeps its own report, which does not change when the city is scanned again: two months are two reports, and they can be compared.

While a scan is running, its report refreshes at most once a minute, and the pages of this site once an hour.

How the scanner identifies itself and what it respects

The scanner is called NFW-Audit and says so: the plain requests and browsers it uses end their User-Agent with "NFW-Audit/1.0" and the address of a page that describes it. Lighthouse audits keep Lighthouse's own User-Agent, so the measurements stay what they were, and leave from the same IP addresses.

It respects rules that name NFW-Audit: a robots.txt group with that name (read before the page), the TDM-Reservation signal (in the header, a meta tag or a TDMRep file in /.well-known/) and an exclusion list fed by owners' requests. It does not treat generic "User-agent: *" rules as addressed to it. A site that excludes it is not measured and leaves no data.

It is also polite: at most two requests at a time per domain, a quarter of a second apart, and when a site answers 429 or 503 with a wait time it waits and, if the site insists, gives up. A site that asked it to slow down is counted as such in the report.

What these figures do and do not measure

A report says precise things, and only those. These are the limits to keep in mind when quoting one of its figures.

• Homepage only. Lighthouse and the AI Crawlability Score measure each site's main page, not the whole site: the figure is that of a shop window, not of every page.
• Working sites only. The web figures cover the sites that answer. Sites that do not answer, or that block our tool, are left out: a site behind a bot check can be unmeasurable while working for a visitor.
• Only businesses that list a website on Overture. A business with no website on its listing is not measured on the web, and a listing with an old website leads to a page that does not open. Every report says how many businesses list a website and how many have a working one.
• A sample, not a register. Overture is not the business register: it covers some businesses and some areas better than others, so the picture can lean towards businesses with a well kept listing.
• A site counts once. When several businesses list the same site, it counts once in the web figures. Business counts count each one.
• One language. Our tool presents itself as an Italian browser: a multilingual site is measured in its Italian version.
• Search Console is recognised only from a meta tag or the DNS, so the real share can be higher than the one we show.
• The AI Crawlability Score is our own measure, not a market standard: it cannot be compared with outside tools. It changes with rules versions, and two reports compare only on the same scanner and the same rules.
• Cities only, and only averages and medians. Reports do not say who is best in a sector or which studios do better, and they publish no single business's result.

What we do not publish

Aggregate figures only. No single business's or single site's result appears in the reports or in the public data. Anyone who wants to see their own can ask for it on a free call.

The aggregate figures of every scan are also available as JSON from the dashboard's public API (GET /state-of-the-web/v1/scans/{scanId}), for anyone who wants to cite or reuse them.

Where the data is archived

The full report of every scan, in JSON and CSV, is archived on Zenodo with a DOI of its own and the CC BY 4.0 license. The same files are on Hugging Face, in one folder per scan, and on Kaggle, as one dataset with every city.

A frozen report no longer changes: its figures, its breakdowns and the national average it is compared with stay as they were on the day of the deposit, even if the way of calculating them is updated later. A report that is not frozen yet can be recalculated when a new version of the statistics ships. To cite a report, use the DOI of its edition: the concept DOI groups every edition of a city.

• Zenodo: the State of the Web community
• Hugging Face: the dataset
• Kaggle: the dataset