How to Optimise Your Website for AI Search: What We Got Right, What's Changed, and What the Evidence Now Says
Last September we published a post arguing that fast, accessible, well-structured websites were exactly what AI search engines wanted to read and cite. Ten months later, half the landscape it described has changed, and some of the claims we made, in honesty, were better instincts than evidence.
So rather than quietly tweak a few paragraphs, we've rewritten the whole thing. The central argument survives, and if anything, the evidence for it is now stronger. But the field has moved quickly, some of our specifics have aged, and you deserve to know which is which. That's how we try to operate at SiteBeacon: confident where the data backs us, upfront where it doesn't.
What's Changed Since September 2025
Quite a lot, as it turns out.
Google's AI Mode, the conversational search experience that answers questions directly rather than showing ten blue links, passed one billion monthly users within a year of launch, with Google saying queries have more than doubled every quarter. AI Overviews, the AI-generated summaries at the top of ordinary Google results, now sit above a huge share of searches. ChatGPT search, Perplexity and Microsoft's Copilot have all matured from novelties into places where real buyers ask real questions.
The traffic data tells a sobering story. According to SparkToro's analysis of Similarweb panel data, roughly 68% of Google searches in early 2026 ended without a click to the open web, up from about 60% in 2024. And Similarweb's own figures show that while visits to AI platforms grew about 29% between January 2025 and January 2026, referrals from those platforms out to websites stayed roughly flat. AI tools are built to answer questions themselves, not to send you visitors.
Here's the part that matters, though. When AI platforms do send visitors, they're unusually good ones. Similarweb found AI platforms are growing quickly, but referral traffic remains small beside traditional search. Similarweb's data suggests that ChatGPT referrals can be high-intent: in one April–May 2026 clickstream panel, they converted at 7.1%, second only to paid search. Treat that as a useful directional signal, not a universal benchmark. Fewer clicks, better clicks. Which means the question is no longer "how do I rank?" but "how do I get read, trusted and cited by the systems doing the answering?"
That's the question our original three-pillar argument was built for. Let's rebuild it properly, with sources this time.
Pillar One: Performance and Efficiency (With an Honest Correction)
First, the correction. Our original post claimed that AI companies, watching their compute bills, would preferentially crawl lightweight sites. We called efficiency "a survival mechanism" for them. It was a tidy argument. It was also unsourced, and having gone looking, we can't find solid evidence that any AI crawler deliberately favours lightweight sites in its crawl scheduling. So we'll say it plainly: that remains a reasonable bet, not a proven fact. Consider it corrected.
Here's what the evidence does support, and it's arguably more important.
A Quick Definition: What Is a Crawler?
A crawler (or bot) is an automated program that visits web pages and reads their contents, the same way a search engine has always done, so the information can be indexed or fed to an AI model.
Research by Vercel and Merj analysed billions of crawler requests and found that the major AI crawlers, including OpenAI's GPTBot, Anthropic's ClaudeBot and PerplexityBot, do not execute JavaScript. Google's Gemini can, because it rides on Google's existing rendering infrastructure, but the rest fetch your raw HTML and read what's there.
Why This Matters
Many modern websites are built as JavaScript applications: the initial page the server sends is nearly empty, and the actual content is assembled in the visitor's browser by code.
A human sees a finished page. GPTBot sees the empty shell.
If your pricing, your case studies and your expertise only exist after JavaScript runs, then to most AI systems they barely exist at all.
The fix is server-side rendering: making sure your server sends complete, readable HTML in the first response. If you're not sure whether yours does, try viewing your page's source (not the developer inspector, the raw source) and checking whether your key content is actually in it.
Why Clean Technical Foundations Still Matter
The same research found AI crawlers are surprisingly wasteful, with ChatGPT and Claude each burning roughly a third of their requests on dead links and error pages. These crawlers do not behave like a human visitor, and the research shows they can spend a large share of requests on errors and redirects. That makes clean URL management and critical content in plain, initial HTML sensible technical hygiene.
And speed matters at another moment too. Search-oriented AI systems can retrieve and surface web content, while others may rely partly on indexed, cached or training data.
Fast, reliable pages remain a low-regret investment but no platform has published a rule saying response time directly improves AI citations. We won't dress that up as a published ranking factor, because nobody has published one.
But "be fast and be readable without JavaScript" is about as low-regret as advice gets, because it also makes your site better for every human visitor and, since fewer transferred bytes mean less energy used, lighter on carbon as well.
Pillar Two: Accessibility Is Machine Readability
This was the claim in our original post we're most pleased with, because time has been kind to it.
An accessible website is one built so that people with disabilities can use it: someone navigating by keyboard, someone using a screen reader (software that reads a page aloud for blind and partially sighted users), someone who needs clear structure to follow along.
Screen readers generally use the accessibility information a browser exposes from your HTML, ARIA and rendered interface, not the visual design alone:
the heading levels that say "this is a main topic, this is a subtopic"
the alt text that describes images in words
the labels that explain what a button does
The Same Structure Helps AI
Now notice what an AI crawler does. It doesn't look at your design either. It parses your underlying HTML.
A page built with semantic HTML, meaning HTML elements used for their actual meaning (a heading marked as a heading, a list marked as a list, a navigation area marked as navigation) rather than anonymous styled boxes, hands both audiences the same gift: structure they can interpret with confidence.
A logical heading hierarchy is an outline of your argument.
Alt text is your images, translated into the only language a text-based model reads.
A wall of <div> tags with the meaning painted on visually is, to a machine, closer to mush.
What the Evidence Actually Says
We'd love to cite a controlled study proving accessible sites get cited more, and we won't pretend one exists.
But this pillar doesn't rest on a correlation study.
It rests on how these systems mechanically work: they consume text and structure, and accessibility best practice is precisely the discipline of making text and structure carry your meaning.
Two audiences, one fix.
The principle has become more commercially relevant as AI-assisted search reaches more people.
Pillar Three: Structured Data, and the Truth About llms.txt
Schema markup (also called structured data) is a standardised vocabulary you add to a page's code to label what things are:
this is a product and here's its price
this is an article and here's its author
this is an FAQ
It converts "probably a price, judging by the pound sign" into certainty.
What We Know About Schema
What's the actual evidence it helps with AI? Mixed, and worth stating carefully.
On the confirmed side:
Google uses structured data to understand page content and support eligible search features.
Microsoft has said schema helps Bing's language models understand content for Copilot.
Google also says no special schema or file is required for its generative search features.
Since Google and Bing's indexes feed a large share of AI answers, that's meaningful.
On the unproven side:
OpenAI, Anthropic and Perplexity haven't said they use schema at all.
At least one study found no correlation between schema coverage and citation rates.
Our view: implement schema for the platforms that demonstrably use it, treat any effect on the others as a bonus, and expect no miracles from markup alone.
It's plumbing, not magic.
The Reality of llms.txt
Then there's llms.txt, which barely existed when we wrote the original.
It's a proposed standard: a plain text file at your site's root offering AI systems a curated, simplified guide to your content, in the spirit of the long-established robots.txt file that tells crawlers where they may go.
Proposed is the operative word.
Google has said its systems don't use it, and an Ahrefs study of over 137,000 domains in mid-2026 found that although 28% of sites now publish one, 97% of those files received zero requests in the month studied, and AI bots weren't going looking for missing ones either.
So:
cheap to publish
currently close to ignored for AI-search visibility
still needs ownership and review because a stale or compromised file can mislead any agent that does read it
Add one if you like, but never instead of the fundamentals above.
Anyone selling llms.txt as an AI visibility breakthrough is selling you a file nobody's reading.
What to Actually Do This Quarter
If you run a site, or run sites for clients, here's the work in priority order.
Check that your key pages deliver their real content in server-rendered HTML, because if the major AI crawlers can't read you, nothing else on this list matters.
Fix your error pages and broken links, since AI crawlers waste a third of their visits on them and every wasted visit is a page of yours they didn't read.
Get your page speed and page weight down, which helps live retrieval, humans and your carbon footprint in one move.
Audit your accessibility properly: heading hierarchy, alt text, labels, semantic elements, not just a colour-contrast once-over.
Layer schema markup onto your most important page types, products, articles, FAQs and organisation details first.
Keep your claims about your own business clear, factual and consistent across the site, because you're now writing for systems that quote.
Notice what's not on that list: tricks.
There's no secret tag that gets you cited.
There's a well-built website, and there's everything else.
Our Take
The web is being read by a new audience that doesn't see your design, doesn't run your JavaScript and doesn't wait around.
The genuinely good news is that everything this audience rewards, we already knew how to build: fast pages, accessible structure, honest semantics.
Performance, accessibility and carbon efficiency were always the same discipline wearing three hats.
AI search just added a fourth.
Some of this will age too, and when it does we'll rewrite it again and tell you what changed.
In the meantime, if you'd like to know how your site holds up across all three pillars, run a free SiteBeacon analysis and see for yourself, or get in touch if you'd rather talk it through.