Study · 2,216 domains · August to September 2026
Almost none of them stop AI assistants from citing them: at most 3.4% block a citation crawler, and among product sites just 1.2%. What most of them lack is simpler — 7 in 10 product sites don’t say in their markup what they sell. And the file that looks most like a GEO signal, /llms.txt, is on 41.5% of sites, often because the store platform published it for them.
# robots.txt, 2,216 domains blocks any AI crawler 151 6.8% blocks a citation crawler 51 2.3% blocks training only 75 3.4% # schema.org, 927 product sites has useful schema 73.1% names what it sells 30.4% # /llms.txt present 41.5% # first line of 5 in a sample of 10: # Agent Instructions — [store name]
Every number is counted per unique domain, only where that check ran, and checked against a hand-verified sample. How it was measured.
151 of 2,216 sites (6.8%) block at least one AI crawler in robots.txt. Only 51 (2.3%) block one that AI assistants cite from, such as OAI-SearchBot, Claude-SearchBot or PerplexityBot. Another 75 block training crawlers only, which keeps them out of future models but not out of today’s answers — blocking GPTBot does not stop ChatGPT citing a site. 25 could not be classified, so the real share of citation blocks sits between 2.3% and 3.4%.
| Site type | Domains | Block citation | Block training only | Unresolved |
|---|---|---|---|---|
| Product sites | 927 | 11 (1.2%) | 34 (3.7%) | 0 |
| Editorial sites | 666 | 26 (3.9%) | 18 (2.7%) | 25 |
| Unclassified sites | 623 | 14 (2.2%) | 23 (3.7%) | 0 |
| All sites | 2,216 | 51 (2.3%) | 75 (3.4%) | 25 |
The blocks are concentrated in publishers. Editorial sites block citation more than three times as often as product sites, and of the 51 sites that block it, one is a SaaS product site. For a software company, robots.txt is rarely the reason an assistant ignores it. The AI crawler directory lists which user-agent does what.
Mostly not. 73.1% of product sites have some useful schema.org markup on the home page, but for most of them it describes the company — its name and logo — rather than what it sells. Only 30.4% use a type that names what they sell: Product, SoftwareApplication, Service or Offer. Without it, an assistant summarising a category has nothing structured to attribute a recommendation to. That is check 02 of the audit.
| Site type | Domains | Any useful schema | Schema naming what it sells |
|---|---|---|---|
| Product sites | 927 | 73.1% | 30.4% |
| Editorial sites | 666 | 74.6% | 8.6% |
| Unclassified sites | 623 | 55.9% | 10.1% |
| All sites | 2,216 | 68.7% | 18.1% |
9.2% of all sites use FAQPage markup, the structured form of the question-and-answer format assistants quote most. SaaS sites lead at 18.7%; e-commerce sites barely use it, at 3.4%. This is the only FAQ number reported here: the pipeline’s looser test, which reads headings, counted section titles such as “What to Expect” as questions in 3 of 10 sampled sites, so it was left out.
| Segment | Domains | FAQPage markup | /llms.txt present |
|---|---|---|---|
| SaaS | 780 | 18.7% | 33.6% |
| Direct-to-consumer e-commerce | 1,291 | 3.4% | 48.3% |
| Education | 144 | 9.7% | 22.9% |
| All sites | 2,216 | 9.2% | 41.5% |
More common than expected: 41.5% of sites serve a real /llms.txt, one that stays at that address, is not an HTML page and starts like a Markdown document. E-commerce leads at 48.3%. The reason shows up in the files themselves: in a sample of 10, 6 were generated by the store platform — 5 by Shopify, all opening with “# Agent Instructions”, and 1 by Wix. Whether any major assistant reads the file is still unconfirmed, as the llms.txt check explains, so presence alone says little about intent.
That the loud problem is rarely the real one. A robots.txt block is easy to find and makes a dramatic finding, but it applies to about one product site in 85. The quiet gaps are the common ones: schema.org markup that never says what the company sells, and no page written as a direct question and answer. Those are checks 02 and 03 of the five Rankavex checks, and the ones most sites fail.
The free checker applies the same definitions behind these numbers to your own site, in a few seconds.
Run the free checkPrefer a human look? Send the domain.
The dataset
Audit results for 2,216 unique domains found through DuckDuckGo searches for problem-aware keywords in SaaS, direct-to-consumer e-commerce and education, collected from 2026-08-15 to 2026-09-24 and re-audited on 2026-09-24. Measured per domain: whether robots.txt blocks an AI citation crawler or only training crawlers, whether the home page’s schema.org markup names what the site sells, whether it uses FAQPage markup, and whether /llms.txt exists.
Cite this
Across 2,216 audited sites, between 2.3% and 3.4% blocked a crawler AI assistants cite from — 1.2% of product sites. The common gap is elsewhere: only 30.4% of product sites had schema.org markup naming what they sell.
Igor Almeida, Rankavex. Last updated 2026-09-24https://rankavex.com/data/ai-visibility-study/
Released under CC BY 4.0: reuse it anywhere, commercial use included, as long as you credit Rankavex with a link to this page.