How we calculate the AI Crawlability Score

The AI Crawlability Score measures how ready a site is to be read and cited by AI assistants and AI search. It runs from 0 to 100 and is the weighted mean of eleven checks, from the content in the HTML to the ARD (Agentic Resource Discovery) manifest, computed for every working site we audited.

Check by check, and why that weight

The weights follow three ideas. The three heaviest checks, 50 points out of 100, decide whether an AI can read and understand the site at all: content in the HTML, robots.txt and structured data. If they fail, the rest matters little. Next, with 10 each, come the two files that help find the pages: the sitemap and llms.txt. At the bottom, with 5 points each, the basics almost everyone has, which separate neglected sites rather than reward the best ones, and the ARD manifest, which almost nobody has yet.

1. 1. Content in the HTML

   Weight 20

   This is the heaviest check because it decides whether the others matter. Many AI crawlers do not run JavaScript: if the content only appears after the browser has worked, they see an almost empty page, and no title, structured data or sitemap can save it. It gives partial credit because a site can have some pages ready and others not.

2. 2. llms.txt

   Weight 10

   It is the only file written for AI assistants: a short, clean map of the site in plain text, which an assistant reads without having to interpret the design. It weighs 10 because it is cheap to have and a deliberate signal. It does not weigh more because nobody guarantees that every assistant reads it, while the checks about access apply to any crawler. Until rules 2.0.0 it weighed 15.

3. 3. robots.txt allows AI

   Weight 15

   A site closed to AI crawlers cannot be read or cited, however well it is built. It often happens by accident: a default rule from a plugin or a host that blocks everything. That is why it weighs as much as the content in the HTML: if the answer is no, the rest never arrives.

4. 4. Structured data

   Weight 15

   JSON-LD states in a machine readable form who the business is, where it is and when it is open, without having to infer it from the text. The point of the score is whether an AI can tell what the business is, so this check weighs as much as the two about access.

5. 5. Sitemap

   Weight 10

   It lists the pages of the site, so a crawler reaches the ones that cannot be found from the home page. It helps with finding, not reading, and a crawler can still follow links: that is why it weighs 10 and not 15.

6. 6. Title

   Weight 5

   It is the first label every assistant reads on a page. Almost every site has one and on its own it says little, so the weight is low: it separates neglected sites, it does not reward good ones.

7. 7. Meta description

   Weight 5

   It is the summary that is often reused in results and answers. A crawler can do without it by reading the page, which is why the weight is low.

8. 8. Canonical

   Weight 5

   It says which address is the main version of a page, so an AI that finds several copies cites the right one. It is about tidiness more than access: low weight.

9. 9. HTTPS

   Weight 5

   Many tools treat a site without HTTPS as untrustworthy, and some crawlers avoid it. Almost everyone has it today, so it separates few sites and the weight is low.

10. 10. Lighthouse SEO

    Weight 5

    It collects many basics the other checks do not look at, such as links crawlers can follow and pages that can be indexed. It weighs little because it partly overlaps with title, meta description and canonical, which we already count. It gives partial credit because Lighthouse scores from 0 to 100.

11. 11. ARD manifest

    Weight 5

    The ARD manifest lists the resources a site offers to AI agents, such as an MCP server or a skill, in a file that agent catalogs index: after being read, it is the step to being used. It weighs 5 because very few sites publish one today, and it only matters to those with something to offer agents. We validate it with Lighthouse's test, so our verdict matches PageSpeed Insights'. It counts from rules 2.0.0.

The AI crawlers we check for

For the robots.txt check we look at whether the site is closed to every bot, or to any of these:

• GPTBot
• OAI-SearchBot
• ChatGPT-User
• ClaudeBot
• Claude-Web
• anthropic-ai
• PerplexityBot
• Google-Extended
• CCBot
• Applebot-Extended

When a figure is missing

A check the audit cannot answer is left out for that site, and the remaining weights are rescaled. A site whose page rendering is unknown, for instance, is scored out of 80 rather than given zero: missing data never counts as a failure.

Two scores: the one of its day and today's

The score's rules, its checks and their weights, carry a version number and never change once published: any change is a new version. Every audit and every report keeps the score computed with the rules of the day it was made, so published figures do not move. When the rules have changed since, it shows the score with today's rules beside it, and that is the one we use to compare two sites or two audits. A scan made before a check existed, such as the ARD manifest before 2 October 2026, does not count it: that check is left out, not failed.