# We published our datasets: an open dataset of websites Source: [Home](https://www.nfw.run/en)[Blog](https://www.nfw.run/blog) // article Published: November 11, 2026Written by [Enrico Pasetto](https://www.linkedin.com/in/enrico-pasetto) () - [State of the web](https://www.nfw.run/blog/tag/state-of-the-web) ![We published our datasets: Hugging Face, Zenodo and Kaggle, free, aggregated data only, CC BY 4.0](https://www.nfw.run/blog/state-of-the-web-open-dataset/cover-en.webp) We published our datasets. The data behind our [State of the Web](https://www.nfw.run/state-of-the-web) reports, city by city and month by month, is now free to download on Hugging Face, Zenodo and Kaggle. It is an open dataset of business websites, all in aggregate form. ![The State of the Web page with the new Open data links to Hugging Face, Zenodo and Kaggle](https://www.nfw.run/blog/state-of-the-web-open-dataset/state-of-the-web-open-data-en.webp) The State of the Web page now links to the three open data repositories. ## What is in the data Every city report we publish is in it: speed and Core Web Vitals (Lighthouse lab data, mobile), AI readability through the [AI Crawlability Score](https://www.nfw.run/state-of-the-web/ai-crawlability-score), structured data, contacts, technologies and sectors. It also has security headers and HTTPS, spam blacklists and media formats. There is one row per city scan, plus one row for the Italy average. Most tables also come split by town size: small, medium and large. The businesses come from [Overture Maps](https://www.nfw.run/state-of-the-web/overture-maps) places data. We crawl the homepage of each business that lists a website. The full approach is on the [method page](https://www.nfw.run/state-of-the-web/method). Global datasets such as the HTTP Archive describe the web as a whole. Ours is city level and about local businesses, so the two complement each other. ## Why we published it We could not find open data on the websites of Italian local businesses. So we are publishing ours. We believe in open source, and we want to give companies and professionals something useful to improve with. Anyone can do what they want with this website dataset: research, an article, an AI project, or just browsing the numbers. Among the free datasets we looked at, we found no website statistics like these for Italian local businesses. ## Where to find it ### Hugging Face The dataset has 40 subsets: `scans` (one row per city scan, with month, DOI, report URL and headline columns such as `lcp_p50_ms`, `cwv_all_good_pct`, `ai_score_median` and `https_pct`), then `metrics`, `technologies`, `sectors`, `schema_types`, `contacts`, `media`, `cdn`, `blacklists` and `ai_checks`, each also split into `-small`, `-medium` and `-large` towns. Start in the browser: the dataset viewer and the Data Studio give you a SQL console, no setup. Then load it from code: ``` from datasets import load_dataset scans = load_dataset("nfw-web-studio/state-of-the-web", "scans", split="train") ``` It also works with pandas and Polars. The CSV files are converted to Parquet automatically. [Open the dataset on Hugging Face](https://huggingface.co/datasets/nfw-web-studio/state-of-the-web). ![The Hugging Face dataset viewer showing the scans subset](https://www.nfw.run/blog/state-of-the-web-open-dataset/hugging-face-dataset-viewer.webp) The dataset viewer on Hugging Face, with the scans subset. ### Zenodo Each frozen city report is its own record with its own DOI, so you can cite an exact edition and it never changes. A record holds a README with the file dictionary and the method, `report.json`, `national-average.json`, `breakdowns.json` and CSV files for sectors, technologies, metrics, contacts, media, blacklists, AI checks, schema types, CDN and issues. [Browse the Zenodo community](https://zenodo.org/communities/state-of-the-web), or open the [Verona record of October 2026](https://zenodo.org/records/24054894) as an example. ![The Zenodo record of the Verona report of October 2026](https://www.nfw.run/blog/state-of-the-web-open-dataset/zenodo-verona-record.webp) The Verona record on Zenodo, with its DOI and files. ### Kaggle One dataset with every city, updated monthly. Open it in a Kaggle notebook with the Code button, or download it with `kaggle datasets download nfwwebstudio/state-of-the-web`. [Open the dataset on Kaggle](https://www.kaggle.com/datasets/nfwwebstudio/state-of-the-web). ![The Kaggle page of the State of the Web dataset](https://www.nfw.run/blog/state-of-the-web-open-dataset/kaggle-dataset-page.webp) The dataset on Kaggle, with every city in one place. The aggregate figures of every scan are also available as JSON from the dashboard's public API, at `GET /state-of-the-web/v1/scans/{scanId}`. ## License and how to cite The data is under CC BY 4.0: use it as you like, just credit NFW Web Studio and link the report or the DOI of the edition. Every Zenodo record has a ready "How to cite" line. For example: > NFW Web Studio (2026). State of the Web: Verona (VR), Italy, October 2026 \[dataset]. Zenodo. 10.5281/zenodo.24054894 To cite every edition of a city at once, use its concept DOI (for Verona, 10.5281/zenodo.24054243). We publish counts and percentiles only. No single business, website, phone number or email is in the data, and we never publish the result of a single site. ## What will you build with it? Tell us what you are using it for: [contact us](https://www.nfw.run/contact-us) and show us. If you want to see where your own site stands, run a [free SEO audit](https://www.nfw.run/seo-audit). And if you want help fixing what it finds, our [SEO services](https://www.nfw.run/seo-services) can take it from there. ## Related services If you want to go from reading to doing: - [SEO services](https://www.nfw.run/seo-services) - [Web design services](https://www.nfw.run/website-design) - [AI search optimization](https://www.nfw.run/ai-search-optimization) ## More to read [All articles](https://www.nfw.run/blog) ![Google Search Console: An error occurred, try again in a few hours.](https://www.nfw.run/blog/google-search-console-heavy-page/cover-en.webp) October 27, 2026The studioState of the web ### [Search Console "An error occurred" on Test Live URL](https://www.nfw.run/blog/google-search-console-heavy-page) "URL is not on Google", then Test Live URL fails with "An error occurred". Our fix was page weight: JavaScript cut from 2.83 MB to 1.04 MB. Read the article ![Title card of the video: PHP2Shell, two PHP flaws, NFW Web Studio](https://www.nfw.run/blog/php2shell-php-vulnerability/video-poster-en.webp)Video October 22, 2026SecurityState of the web ### [PHP2Shell: two PHP flaws on half the sites we scan](https://www.nfw.run/blog/php2shell-php-vulnerability) Two serious PHP flaws can let a stranger take over a site without a password. What to ask your host, which versions are affected, what our scans show. Watch the video ![Milan businesses: speed, Google and AI readiness, measured in October 2026, with the outline of the city](https://www.nfw.run/blog/milan-websites-state-of-the-web/cover-en.webp) October 20, 2026State of the webWeb presence ### [Milan business websites: what we found in 27,460 sites](https://www.nfw.run/blog/milan-websites-state-of-the-web) We scanned 27,460 websites of Milan businesses: the same slow mobile pages as Verona, a gap in AI readability, thin structured data and old image formats. Read the article ## Got a project in mind? Let's talk. [All articles](https://www.nfw.run/blog) [Book a call](https://www.nfw.run/contact-us)