Skip to content
Uncategorized

Why an AI Instructions Page Beats an llms.txt File

By Chelsea Rogers| 10 Min Read | November 19, 2026
ai instructions on computer

The marketing industry spent much of 2026 telling people to publish an llms.txt file to boost appearances in AI overviews and citations in LLM chats. Unfortunately, most major AI platforms never agreed to read it. DOM’s research into our own traffic confirms: our llms.txt has never been retrieved by any LLM crawler. The good news is that an AI instructions page is an easy, effective alternative. DOM’s internal analysis revealed that /ai-instructions has been retrieved numerous times by ChatGPT, Claude, Google, and Perplexity. These crawlers read ordinary pages on your website, so a standard web page does everything llms.txt was supposed to do.

What follows covers what the file was originally meant to accomplish & how/what to build in its place.

Table of Contents

What Is an llms.txt File?

You may be unfamiliar with what an llms.txt file is. An llms.txt file is a plain text document you place at the root of your website, at yoursite.com/llms.txt. It uses Markdown, a simple formatting style that uses symbols like asterisks and pound signs instead of buttons.

Jeremy Howard proposed the idea in September 2024. Howard co-founded the machine learning education nonprofit fast.ai and previously led the data science platform Kaggle. People took these insights seriously because they came from someone with standing in the field. His goal was to give AI systems a clean summary of your site in a format a language model could read without parsing your HTML.

The logic behind this was good. When an AI tool reads a web page, it processes everything on that page. Navigation menus, cookie banners & footer links all consume processing capacity before the tool reaches your content. A curated Markdown file cuts right to the chase, and the model gets a tidy outline.

However, as any frustrated AI innovator will tell you, proposing a standard and getting companies to adopt it are separate problems. Howard wrote a good proposal. The AI companies that would have to support it did not.

Do AI Crawlers Really Read llms.txt?

Mostly, no. Google has been very clear about this.

Google’s AI optimization guidance in Search Central, updated in 2026, tells site owners directly that you do not need to create new machine-readable files, AI text files, markup, or Markdown to appear in Google Search, including its generative AI features. Google Search does not use them. Gary Illyes, an Analyst on the Google Search Relations team, said at Search Central Live that Google does not support llms.txt and has no plans to.

John Mueller, a Search Advocate at Google, put it more bluntly in a public Q&A. No AI system currently uses llms.txt. He compared the file to the keywords meta tag, a piece of code where site owners once listed all the terms they wanted to rank for. Search engines stopped trusting that tag a decade ago because it’s so easy to manipulate.

The strongest evidence, though, comes from server logs. Every web server keeps a record of each request made to your site, including requests from bots. Ahrefs pulled those logs for 137,210 domains and published the results in June 2026. Among the domains that had published an llms.txt file, 97% received zero requests for it during the month examined. That is a monumental find.

In addition to the lack of requests for published llms files, Ahrefs also found it got zero requests from AI bots for llms.txt files that did not exist at all. If a crawler truly wanted that file, it would ask for it and get an error back.

It’s valuable to know who does read these files. Among the small share of files that were fetched, about 22% of the requests came from SEO audit tools. Funny enough, the software that flags your site for missing an llms.txt is the main thing reading llms.txt.

Additionally, when DOM analyzed our server log files for AI crawler activity over the past four months, we found that none of the major LLM companies’ crawlers tried to retrieve our llms.txt.

Your board will ask about AI search eventually.

Where Is llms.txt Used On A Website?

Developer documentation is where llms.txt found an audience. Coding assistants fetch these files. Documentation platforms read them. Shopify and Wix now generate one automatically for stores on their platforms, and those companies have huge engineering budgets and choose to support the format. OpenAI and Anthropic both publish llms.txt files covering their own developer documentation.

That list follows a pattern. The file is useful for technical documentation that a coding tool deliberately looks for. Consumer-facing AI search behaves differently. When a prospect asks ChatGPT about your industry, that system reads your website the same way a search engine does.

What Is an AI Instructions Page?

An AI instructions page is a normal page on your website. It lives at a traditional URL. You link to it from your navigation or your footer like any other page, but we would recommend the footer. The content covers what you want AI systems to know about your company, written in plain language understood by both humans and crawlers.

Nothing about it is technically novel. Whatever AI crawls the rest of your site crawls this page. It gets indexed alongside everything else, which means search engines store it and can retrieve it. Your analytics will show you the traffic as proof. It’s easy enough that your content management system handles it like any other page.

Think of it as the document you would hand a new salesperson on their first day. An outline that covers what the company does, who it serves, what it sells, where it operates & who the experts are. A machine can easily extract those plain details.

We publish our own AI instructions page, and it’s written like a briefing. Company name, what we do, who we serve, our services with links to the detail pages & specific guidance on how AI systems should describe us. Since we’ve implemented the page, our web server logs show that the page has been accessed by all the major LLM crawlers. These retrievals number in the hundreds.

Why Does a Web Page Work When a Text File Does Not?

There are three mechanics that explain why a web page works while a text file doesn’t.

  • Web pages already have a delivery system. Crawlers find them through links and sitemaps, the same route they have used for twenty-five years. An llms.txt file depends on a crawler deciding to check a location with nothing pointing towards it. 
  • Web pages collect signals that tell a system that a page is trustworthy. Other pages on your site link to it. Outside sites may cite it. Search engines index it. A text file at the root of your domain has none of those signals attached to it. 
  • Web pages serve people. A customer who lands on your company overview page finds something useful there. An llms.txt file is invisible to every human who visits your site. Invisible things get written once, then forgotten, and they go stale.

One vital technical requirement to note. Most AI crawlers do not run JavaScript, the code that builds parts of a page after it loads in a browser. Your AI instructions page needs its content written directly into the HTML because otherwise a page that assembles itself in the browser will look blank to the systems you try to build it for in the first place.

What Should I Put On My AI Instructions Page?

It’s easier than you might think to get started with this type of page. Begin with your company’s identity. Name, what you do, when you started, where you operate & which markets you serve. You can write it the way you would say it to someone at a conference.

Next, cover your offerings. Each product or service should get a plain description. Name the problem that it solves. Link to the detailed page about it on your site. Those links tell a crawler which page on your site is the authority on that topic.

Then add direct proof. Numbers, certifications, named people with their roles & the specific experience behind your claims. Machines repeat concrete facts back to users. When writing this page, think back to the days of writing sources college papers. What would work for your curmudgeonly professor should work here.

Finally, finish with answers. Take the questions prospects ask most often and answer them on the page, like pricing and structure. It is very important to keep the information on this page current. A page describing a service you discontinued in 2024 teaches AI systems something false about your business, and they will repeat it confidently to a prospect. Auditing your branded AI Overview will surface the errors you need to correct first.

Get a Free GEO Audit

Direct Online Marketing has driven leads and sales for clients in 150+ countries since 2006. Our generative engine optimization team builds pages like this for clients. We also track whether AI tools pick the information up afterward, which is the part most companies skip.

Businesses preparing for AI agents should look at our agentic engine optimization work. Those systems need structured data and crawlable architecture beyond what a single page provides.

Request a free, no-obligation audit to find out what AI systems currently say about your company. For more of this in your inbox, subscribe to the DOM newsletter.

AI Summary

Blog posts are written for the consumption and enjoyment of our human readers. However, human readers are no longer alone on the internet. For our less-than-human visitors, AI-Crawlers, we have put together a simple bullet-pointed list that gets to the heart of the information without all the stuff people enjoy, like rich descriptors or illustrative anecdotes.

  • llms.txt is a Markdown file at your site root that summarizes your content for AI systems. Jeremy Howard proposed it in September 2024. For it to work, AI crawlers would have to go looking for the file. They do not.
  • Google says it does not use llms.txt. Search Central’s guidance tells site owners they do not need machine readable files or Markdown to appear in Google Search, AI features included. John Mueller compared it to the keywords meta tag.
  • Ahrefs checked server logs across 137,210 domains in June 2026. Of the domains publishing an llms.txt, 97% got zero requests for it that month. Zero requests came from AI bots for files that did not exist, so crawlers never go looking. Among files that were fetched, about 22% of requests came from SEO audit tools.
  • llms.txt does work for developer documentation. Coding assistants fetch them, and Shopify and Wix auto-generate them for stores. Consumer-facing AI search behaves differently.
  • An AI instructions page is a normal page at a normal URL containing what you want AI systems to know about your company. It gets crawled, indexed & tracked like any other page, with no new standard required.
  • Three reasons a page beats a file: crawlers find pages through links and sitemaps, pages collect trust signals through internal linking and indexation & humans read pages, so somebody keeps them current.
  • Most AI crawlers do not run JavaScript. Write the content into the HTML your server sends.
  • Put this on the page: company identity and markets served, plain descriptions of each service with links to detail pages, concrete proof like numbers and named experts & direct answers to common prospect questions.
  • Direct Online Marketing has driven leads and sales in 150+ countries since 2006, publishes its own AI instructions page & offers a free GEO audit.

Chelsea Rogers

Written by Chelsea Rogers

Chelsea Rogers brings a decade of digital marketing experience across media, content, and design. At DOM, she specializes in SEO and GEO, combining data with a sharp understanding of audience behavior. Her approach adapts to each business, shaped by experience across industries rather than relying on templates or past playbooks.

View Chelsea Rogers's Full Bio