How does the Rankavex pipeline audit a site?

Which sites are in the sample?

Sites that rank for long-tail, problem-aware searches in three segments — the kind of query a buyer types before they know which product they want. That makes it a sample of small and mid-sized sites that already earn some search traffic, not a random slice of the web. Directories, review aggregators and the largest platforms are filtered out before any request is made; a few large publishers still get through, because they rank for the same queries.

The current run covers 2,216 unique domains audited between 2026-08-15 and 2026-09-24: 780 SaaS, 1,291 direct-to-consumer e-commerce, 144 education and 1 other. Each site is also classified by what its home page sells: 927 product sites, 666 editorial sites, 623 unclassified sites.

What does each check read?

AI crawler access. The robots.txt is parsed by user-agent group, as RFC 9309 defines it, including “User-agent: *”. Twenty AI user-agents are checked and split into training crawlers and the crawlers assistants cite from; the AI crawler directory documents each one. A site counts as blocking citation only if at least one citation crawler is disallowed from the whole site.

Structured data. Every JSON-LD block on the home page is parsed. “Any useful schema” counts types that describe the business or its content, such as Organization, Product or Article; WebSite and BreadcrumbList alone do not count. “Schema naming what it sells” is narrower: Product, SoftwareApplication, Service, Offer, ProfessionalService or LocalBusiness.

Question-and-answer content. Reported only as FAQPage markup, which is unambiguous. A looser test based on headings exists in the pipeline but is not published, for the reason below.

/llms.txt. A site counts only if the file answers 200 at /llms.txt without redirecting elsewhere, is not an HTML page, and starts with “#” or “>”, the way the format’s Markdown structure does.

How were the numbers verified?

By drawing random domains from each group, fetching them again and reading the evidence by hand. For robots.txt, 30 domains were drawn: the classification held for all 19 that could be reached in the citation and training-only groups, and 8 of the unresolved ones were settled, 5 of them no longer blocking anything. For structured data, 13 of 15 matched. For /llms.txt, 15 of 15 matched. The heading-based FAQ test failed: in 3 of 10 sampled sites it had read a section title as a question, so only FAQPage markup is reported.

What are the known limits?

Only the home page is read for structured data and question-and-answer markup, so a site with good markup deeper in has it undercounted. Home pages built entirely by JavaScript (71 in this run) can hide markup the pipeline never sees, so they count as having none — one more reason the structured-data figures are a floor, not a ceiling. Every file is a snapshot of the day it was fetched, and sites change: most robots.txt files with long lists of blocked crawlers no longer blocked anything when fetched again. 25 domains still have an unresolved robots.txt classification, which is why the citation figure is reported as a range. And the search position that brought each site into the sample comes from DuckDuckGo, not Google.