We published our datasets: an open dataset of websites
On this page
We published our datasets. The data behind our State of the Web reports, city by city and month by month, is now free to download on Hugging Face, Zenodo and Kaggle. It is an open dataset of business websites, all in aggregate form.
What is in the data
Every city report we publish is in it: speed and Core Web Vitals (Lighthouse lab data, mobile), AI readability through the AI Crawlability Score, structured data, contacts, technologies and sectors. It also has security headers and HTTPS, spam blacklists and media formats.
There is one row per city scan, plus one row for the Italy average. Most tables also come split by town size: small, medium and large.
The businesses come from Overture Maps places data. We crawl the homepage of each business that lists a website. The full approach is on the method page.
Global datasets such as the HTTP Archive describe the web as a whole. Ours is city level and about local businesses, so the two complement each other.
Why we published it
We could not find open data on the websites of Italian local businesses. So we are publishing ours. We believe in open source, and we want to give companies and professionals something useful to improve with.
Anyone can do what they want with this website dataset: research, an article, an AI project, or just browsing the numbers. Among the free datasets we looked at, we found no website statistics like these for Italian local businesses.
Where to find it
Hugging Face
The dataset has 40 subsets: scans (one row per city scan, with month, DOI, report URL and headline columns such as lcp_p50_ms, cwv_all_good_pct, ai_score_median and https_pct), then metrics, technologies, sectors, schema_types, contacts, media, cdn, blacklists and ai_checks, each also split into -small, -medium and -large towns.
Start in the browser: the dataset viewer and the Data Studio give you a SQL console, no setup. Then load it from code:
from datasets import load_dataset
scans = load_dataset("nfw-web-studio/state-of-the-web", "scans", split="train")
It also works with pandas and Polars. The CSV files are converted to Parquet automatically. Open the dataset on Hugging Face.
Zenodo
Each frozen city report is its own record with its own DOI, so you can cite an exact edition and it never changes. A record holds a README with the file dictionary and the method, report.json, national-average.json, breakdowns.json and CSV files for sectors, technologies, metrics, contacts, media, blacklists, AI checks, schema types, CDN and issues. Browse the Zenodo community, or open the Verona record of October 2026 as an example.
Kaggle
One dataset with every city, updated monthly. Open it in a Kaggle notebook with the Code button, or download it with kaggle datasets download nfwwebstudio/state-of-the-web. Open the dataset on Kaggle.
The aggregate figures of every scan are also available as JSON from the dashboard's public API, at GET /state-of-the-web/v1/scans/{scanId}.
License and how to cite
The data is under CC BY 4.0: use it as you like, just credit NFW Web Studio and link the report or the DOI of the edition. Every Zenodo record has a ready "How to cite" line. For example:
NFW Web Studio (2026). State of the Web: Verona (VR), Italy, October 2026 [dataset]. Zenodo. 10.5281/zenodo.24054894
To cite every edition of a city at once, use its concept DOI (for Verona, 10.5281/zenodo.24054243).
We publish counts and percentiles only. No single business, website, phone number or email is in the data, and we never publish the result of a single site.
What will you build with it?
Tell us what you are using it for: contact us and show us. If you want to see where your own site stands, run a free SEO audit. And if you want help fixing what it finds, our SEO services can take it from there.
