Skip to content
2pizza.teamBlog

2,591 Bytes: The Shop That Did Not Exist for AI Crawlers

Ivan Bolonikhin
Founder, 2pizza.team

TL;DR: I ran our audit tooling against a pet-supply delivery shop in Bali. The site returned 2,591 bytes to AI crawlers, an empty JavaScript shell with no product content in it. A crawl of all 110 URLs in the sitemap found 110 pages sharing one title, one description, no h1, and a canonical pointing elsewhere. Robots.txt allowed everything and every page returned HTTP 200, so ordinary SEO checks called the site healthy. We shipped a fix in two layers. Re-measurement is scheduled and the numbers go here when we have them.

This is the first published case from our AI-visibility work, and the client asked to stay unnamed, so there is no brand and no domain in this article. What matters here is the failure mode, not who had it. It is a Vite and React single-page app on the front end with a Django API behind it, which is one of the most common shapes a small e-commerce site takes in 2026.

What the site looked like from a crawler's seat

The shop returned 2,591 bytes of HTML to AI crawlers, and none of it was content. That is the size of an empty single-page-app shell: a title tag, a root div, a script tag, nothing a reader or a model could use. Every product, every category, every word of the About page lived behind a JavaScript bundle that had to execute before any of it appeared.

GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot do not execute JavaScript. They fetch the raw HTML document and read what is in it. For those four agents, this shop had no products, no delivery information, no address and no opening hours. It was not blocked and it was not broken. It was blank.

The frustrating part is that every conventional signal was green. Robots.txt allowed all agents. Every URL answered HTTP 200 with a normal response time. There were no redirect chains, no noindex tags, no server errors. If you had run this site through a standard checklist you would have signed it off, and you would have been wrong about the only thing that mattered.

Why standard SEO tools miss this entirely

Standard SEO crawlers miss this because they execute JavaScript and AI crawlers do not. Screaming Frog, Sitebulb, most site-audit SaaS and Google's own rendering pipeline all run a headless browser. They wait for the app to mount, then read the finished DOM. What they report is what a browser sees, which in this case was a complete, well-structured shop.

The AI retrieval agents work the other way. They are optimised for cost and throughput across a very large crawl surface, so they take the document as served and move on. The gap between what a rendering crawler reports and what a non-rendering crawler receives is the whole bug, and it is invisible unless you deliberately look at the raw bytes.

What this means in practice

  • A perfect SEO audit score tells you nothing about AI visibility on a client-rendered site.
  • The check that matters is the raw response body, before any JavaScript runs.
  • A site can rank acceptably in Google, which renders, and be completely absent from AI answers, which mostly do not.
  • The failure is silent. There is no warning, no error page and no console message anywhere in the stack.

The finding that made it worse: 110 out of 110 pages were the same page

A full crawl of all 110 URLs in the sitemap returned 110 pages with an identical title, an identical description, no h1 element and a canonical tag pointing at a different URL. Not most of them. All of them. Because a single-page app serves one shell for every route, the catalogue page, every category and every product card were byte-for-byte the same document as the homepage from a crawler's point of view.

This is worse than an empty homepage. An empty homepage is a gap. A hundred and ten identical documents with a canonical pointing somewhere else is an active signal that the site is one page of duplicated boilerplate. There is nothing for a retrieval system to index per product, nothing to distinguish a dog food page from a cat litter page, and a canonical instruction telling any consumer of that markup to disregard the URL it just fetched.

This is the finding a rendering tool structurally cannot produce. In a headless browser each of those 110 URLs mounts the router, fetches its own data and paints a distinct page with its own title. The tool reports 110 unique pages and a clean bill of health. Only the raw fetch shows one document repeated 110 times.

What the buying-intent prompts showed

We ran a panel of real purchase questions across three engines and the brand was not named in a single recall answer. The prompts were the ones a customer would actually type, along the lines of best online pet supply stores in Bali with delivery. The answers were not empty. They were assembled from local competitors, from marketplace listings on Shopee and delivery apps like Grab and Gojek, and from general Bali expat guides.

That is the commercially useful version of the technical finding. The demand exists, the engines answer the question confidently, and the answer is built entirely out of sources that are not the shop. Every one of those citations is a page that served readable HTML to a crawler that could not run JavaScript.

What ChatGPT said when asked about the brand by name

Asked about the brand directly, ChatGPT described the site as client-side and said the About and Contact details had not loaded on its side. It said so plainly, without being prompted to diagnose anything. It also reported that it could find no independent trace of the business anywhere else, which lines up with the recall panel.

The last item in that answer was the one I would act on first. It could not find a Google Maps listing for the business. For a local delivery business that is worth more than half the technical fixes in this article combined, because the map listing is the entity record that map results, local packs, aggregator sites and, downstream, AI answers all lean on. Fixing the HTML gives a model something to read. The map listing gives it a reason to believe the business is real and operates where it says it does.

What we shipped, in two layers

We fixed the rendering problem in two layers: a static content layer inside index.html, and a build-time prerender step for the catalogue and product pages. Neither of them required moving the site to a new framework, which was the constraint the client cared about most.

Layer one: real content in the shell

The first layer puts genuine content and JSON-LD inside index.html so it is present in the first byte of the response. That means a written description of the business, the delivery zones, contact details, and Organization plus PetStore structured data with the areas served. React replaces that markup when it mounts, so a human visitor sees the app exactly as before, and a crawler that never runs the bundle still gets a readable document.

Why this is not cloaking

This is not cloaking because there is one file and everyone receives it. Cloaking is serving different content depending on who is asking, usually by sniffing the user agent or the IP. Here the server has a single index.html, the bot and the human both download the same bytes, and the only difference is that one of them runs the JavaScript afterwards. The static content also has to be truthful and match what the app renders, which is the part people get wrong when they treat this as an SEO trick rather than as progressive enhancement.

Layer two: build-time prerender

The second layer is a script that runs after the build, calls the API, and writes real static HTML for the catalogue and every product card. Each generated page carries its own title, its own description, its own h1 and Product structured data with the fields a shopping answer needs. The same script writes a real sitemap into the dist directory, which fixed a quiet second bug: the single-page-app fallback route had been answering requests for sitemap.xml with the HTML shell, so the sitemap the crawlers were fetching was not a sitemap at all.

What changed in the served bytes

  • The homepage went from 2,591 bytes of empty shell to 6.9KB of meaningful HTML.
  • Product pages now serve their own title, description and Product markup instead of the homepage boilerplate.
  • The sitemap route returns a sitemap instead of an HTML page.
  • Organization and PetStore data with delivery zones is present before any script runs.

How to check your own site in ten seconds

Fetch your own homepage the way a non-rendering crawler does and count the bytes. From a terminal: curl -A GPTBot https://yoursite.com | wc -c. If the number is in the low thousands, open the response and look for a sentence a customer would recognise. If there is no such sentence, you have the same problem this shop had, and no amount of content strategy on top of it will reach an AI answer.

If you would rather not use a terminal, our readability scan does the same fetch and reports what the four main AI crawlers receive from your URL, at /geo/scan. It is the free diagnostic half of the audit and it does not require talking to anyone. The full audit at /geo adds the sitemap-wide crawl that produced the 110 of 110 finding, plus the buying-prompt panel.

The honest part: we do not have results yet

Everything above is the engineering half of the job and the effect on AI answers has not been measured. AI systems update their view of a site on their own schedule, and the realistic window for a re-crawl to work through to changed answers is roughly four to eight weeks. Anyone who tells you they moved an AI answer in a week is either describing a live-search result rather than a model's own recall, or telling you a story.

We will re-run the identical prompt panel on the same three engines and publish what it returns, including the case where nothing has changed. That is the only version of this article that is worth anything to the next person reading it. The technical findings above are measurements we made and can show; the outcome is not one yet, so we are not going to write it as if it were.

If you run a client-rendered site, check the raw bytes before you spend anything on content. The free scan is at /geo/scan and takes about ten seconds. The full audit, including the sitemap-wide crawl and the buying-prompt panel, is at /geo, and the findings are yours whether or not you have us do the fixes. Ivan / 2pizza.team

Want us to look at your setup?

Free 30-min audit. We tell you what to automate first and what it would cost.

Book a free audit
Automation Tools
Make vs Zapier vs n8n in 2026: Honest Comparison from 80+ Shipped Projects
16 min read
GEO & AI Search
Generative Engine Optimization: What the Data Actually Supports in 2026
17 min read
GEO & AI Search
Does llms.txt Do Anything? The Data Says No
9 min read