“Is our site AI-ready?” is a question with a long answer, because an assistant reading a page goes through several separate steps, and a destination website can fail at any of them. This is a walk through those steps for a single page, say a town page or an Operator’s Product, with what goes wrong at each one and what a marketing team can check without a developer.

The assistants differ in detail and change often. The shape below holds for the main ones: ChatGPT, Claude, Perplexity and Google’s AI features.

Step 1: permission

Before an assistant’s software fetches anything, it reads your robots.txt file and checks whether its user agent is allowed. The large AI companies now run separate agents for separate jobs. OpenAI’s GPTBot collects training data, OAI-SearchBot surfaces sites in ChatGPT search and ChatGPT-User fetches pages when a person asks (OpenAI⁠ (opens in a new tab)). Anthropic splits ClaudeBot, Claude-SearchBot and Claude-User the same way (Anthropic⁠ (opens in a new tab)), and Perplexity separates PerplexityBot from Perplexity-User (Perplexity⁠ (opens in a new tab)).

OpenAI is direct about the consequence: for your content to appear in ChatGPT’s summaries and snippets, don’t block OAI-SearchBot (OpenAI Help⁠ (opens in a new tab)). Whether to allow the training agents is a separate policy choice. Some sites allow everything; many block training and allow search. A file that looks like this blocks one and allows the other:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
Example robots.txt

What goes wrong: a blanket block added by a security plugin or a hosting default, which nobody on the marketing team knows exists. Firewall and bot-protection services can also turn crawlers away before robots.txt is ever read.

Step 2: finding the page

An assistant rarely browses your site from the home page. It reaches a page through a search index (its own, or a search engine’s), a link from another page, or a URL the person pasted in. Your sitemap helps search indexes find pages. An llms.txt file offers a curated map, though our llms.txt explainer is honest about how few assistants read one today.

What goes wrong: important pages that nothing links to, PDFs holding information that has no web page, and old URLs that redirect in chains or end in a 404.

Step 3: the fetch

The software requests the page over HTTP and waits. If the server is slow, returns an error, or presents a challenge page meant for suspicious traffic, the fetch fails quietly and the assistant moves on to another source.

What goes wrong: pages that are slow on the first byte, cookie walls that hide the content behind a consent overlay in the HTML, and geo-blocking or rate limits that treat a crawler as an attack.

Step 4: render, or not

This is the step that catches destination sites most often. A browser runs a page’s JavaScript and builds what you see. Most AI fetchers don’t. A study by Vercel in December 2024 found that none of the major AI crawlers it measured rendered JavaScript, naming OpenAI’s, Anthropic’s and Perplexity’s, while Google’s Gemini uses Googlebot’s infrastructure and does render (Vercel, December 2024⁠ (opens in a new tab)). That study is two years old and the fetchers will change, so test your own pages instead of relying on it. But the safe assumption is still that text which only appears after scripts run may never be seen.

On a destination site, that often means the most useful content: an events calendar loaded by a widget, Operator details pulled in client-side, opening hours inside a map pop-up, a carousel of this evening’s gigs.

What goes wrong: the page an assistant receives is a header, a footer and an empty box where the content should be.

Step 5: extraction

From the HTML it receives, the software strips navigation, footers, cookie banners and scripts, and keeps what looks like the main content. Some sites now hand agents a clean version directly. Cloudflare’s Markdown for Agents, for example, converts a page to markdown on the fly when an agent asks for it with an Accept: text/markdown header (Cloudflare, February 2026⁠ (opens in a new tab)).

What goes wrong: content that exists only in images (an itinerary as a JPEG, opening hours on a poster), headings that are styled text instead of real headings, and tables built from layout boxes.

Step 6: understanding

Now a language model reads the extracted text. It does best with what a careful human reader also likes: one topic per page, real headings, facts stated in sentences, dates written out, distances with units, and structured data that marks up what a thing is, such as an event with a start date and a venue.

What goes wrong: pages that describe everything and state nothing. “Discover the magic of the valley” gives a model nothing to repeat. “The valley is 90 minutes from Adelaide and most cellar doors open Thursday to Monday” does.

Step 7: the answer and the citation

Finally the assistant writes its answer, and may or may not link to you. Travel is the category where links are most likely: Similarweb found 22.6% of ChatGPT travel answers included web citations in May 2026, against a 6.8% average (Similarweb⁠ (opens in a new tab)). Pages that are fetched, read and understood are the only ones in the running.

A ten-minute test

  1. Open yoursite.com.au/robots.txt and look for rules naming GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot or PerplexityBot. If you see Disallow: / under the search agents, ask why.
  2. Open an important town page, turn JavaScript off in your browser’s settings, and reload. Whatever disappears may be invisible to most AI fetchers.
  3. Right-click and view the page source. Search it for an Operator’s opening hours or an event date. If it isn’t in the source, it is being loaded by a script.
  4. Check that your key facts are text, not images.
  5. Ask two assistants a question your page answers, and note which source they cite. Then measure what they send you.

On Roam

On every Roam site, AI Discovery is live at no extra cost. Step 5 is handled at the source: when an AI agent visits, it receives clean, structured markdown instead of raw HTML. Each site also generates and maintains an llms.txt file, for the assistants that read one. See how AI Discovery works.