The Privacy Act and AI Automation: What an Australian Business Actually Has To Do
Does llms.txt Do Anything? The Data Says No
Ivan Bolonikhin
Founder, 2pizza.team
Founder, 2pizza.team
TL;DR: llms.txt is currently placebo. Ahrefs looked at 137,000 domains and 97% of the files received zero requests. Retrieval bots were about one percent of the small remainder. Google has publicly said it will not use it. Add one in ten minutes if you want the box ticked. Do not pay a retainer for it.
llms.txt is a proposed convention: a markdown file at the root of your site that tells language models which pages matter and summarises what your site is about. The idea is reasonable and the analogy to robots.txt is obvious, which is exactly why it spread so fast through agency pitch decks.
What the measurement showed
Ahrefs studied roughly 137,000 domains that had implemented the file and looked at whether anything actually requested it. The overwhelming majority - about 97 percent - received zero requests. Among the small remainder that saw any traffic at all, the retrieval bots that would need to read the file to make it useful accounted for about one percent.
Separately, Google has said publicly that it does not use llms.txt. That matters because a convention only works if the consumers adopt it. robots.txt works because every crawler honours it. A file nobody reads is not a standard, it is a suggestion.
Why it still shows up on invoices
It is cheap to produce, it looks technical, and it is impossible for a client to disprove in the short term. A deliverable that takes ten minutes, sounds sophisticated and cannot be falsified within the engagement window is close to an ideal line item for an agency that is not being careful. We do not think most people selling it are lying - the convention is genuinely plausible and the measurement is recent.
The one honest reason to add it
Cost. It takes ten minutes and it does no harm, and there is a non-zero chance adoption improves later. Treat it the way you treat a well-formed humans.txt: hygiene, not strategy. What is not acceptable is a slide, a line item, or a monthly fee attached to it.
A related trap: make sure yours does not 404 or 500
We have found live sites where the llms.txt route threw a server error because of a framework routing default. That is worse than not having the file: an error page at a well-known path is a small negative signal about site health. If you add one, check that it actually returns the file, then forget about it.
What to do with the budget instead
The published research points at third-party mentions as the strongest correlate of AI visibility - Ahrefs found r = 0.664 for brand mentions against 0.218 for backlinks across roughly 75,000 brands. Before that, check the boring thing: whether AI crawlers can reach your site at all. Cloudflare has blocked them by default on new zones since July 2025, and that single setting outweighs every content decision underneath it.
- Check whether OAI-SearchBot and the other retrieval agents are blocked at your firewall. This is the highest return per hour in the discipline.
- Check what a crawler actually receives from your site. Client-rendered apps often serve an empty shell.
- Fix entity contradictions - two brands under one legal entity, prices that disagree between pages.
- Then spend on third-party placements, mapped from the sources an engine already cites in your category.
If an agency has put llms.txt on a proposal in front of you, ask them for the request logs. It is a fair question and the answer tells you a lot about how the rest of the engagement will go. Our own take on what does work is at /geo. Ivan / 2pizza.team
Want us to look at your setup?
Free 30-min audit. We tell you what to automate first and what it would cost.