What is llms.txt?
Llms.txt is a text file on your website that gives AI assistants an overview of your most important pages, but it does not control access and does not replace a good site structure.
What exactly does llms.txt do
Llms.txt is a simple text file that you place on your website, usually on the home page under a fixed name. It contains a list of your most important pages, with a brief description per link. The idea is that an AI assistant reads this file and quickly understands where your core pages are located, without having to search through the entire site.
The file was conceived as a kind of route map for AI systems, similar to how a sitemap works for search engines. The difference is that a sitemap is a technical list of all URLs, while llms.txt is intended as a readable summary with explanations per page. So it is not a replacement, but a supplement with a different purpose.
There is no obligation to have this file. It is a proposal that has been picked up by part of the sector, but not every AI system uses it in the same way or consistently. Some assistants look at it, others ignore it completely and work purely on the basis of what they already know or find through live searches.
Why this file matters for discoverability
Discoverability with AI assistants works differently than with a search engine. A search engine crawls your site, indexes pages and displays them in a results list. An AI assistant often does not search live, but provides answers based on what it already knows, supplemented by targeted lookups. Llms.txt helps with the latter: it gives the system a brief, structured overview instead of having to do the research itself.
For an entrepreneur, this means that the chances increase that an AI assistant will cite the correct page from your site, rather than an outdated or less relevant page. This is similar to what happens with grounding, where an AI answer is supported with current sources instead of only training knowledge, as we explain in what is grounding.
Llms.txt does not work on its own. It is one part of a larger whole in which the technology of your site, the quality of your texts and the accessibility for different bots also play a role. Without clear pages and without allowing AI bots to visit your site, an llms.txt file does little good.
How you check whether it is set up for you
The quickest way to check whether you have an llms.txt file is to visit the link itself: your domain name followed by /llms.txt. If you see a readable list of pages and descriptions, it is present. If you get an error message or a blank page, it is missing.
The existence of the file says nothing about whether AI assistants actually use it. So it is not a guarantee, more of an offered tool. More important is often whether the bots of these assistants are allowed to visit your site at all. You regulate this separately, per bot, as can be read in what is GPTBot and what is ClaudeBot.
If you do not have a technical background, it is advisable to have this checked by someone who manages your site. The file itself is simple in structure, but the question of which pages you include and how you describe them requires consideration. An incorrect or outdated list can actually send an AI assistant off in the wrong direction.
The most common misconception
The biggest misconception is that llms.txt controls access: that an AI system can only reach you if this file is present. That is not correct. Access is regulated separately via another file, which indicates per bot what may and may not be visited. Llms.txt is purely an overview, not a lock or key.
A second misconception is that the file replaces what a good site structure should already do. If your site is unclear in structure, with outdated pages and illogical navigation, an llms.txt file will not solve that. It is a summary, not a repair of underlying problems.
Some entrepreneurs also think that this file affects their ranking at Google. That is not the purpose. Llms.txt is aimed at AI assistants, not traditional search engines. For search engines, other matters remain leading, such as how Bingbot and Google-Extended crawl and interpret your site, as explained in what is Bingbot and what is Google-Extended.
How this fits with the rest of your discoverability
Llms.txt is one puzzle piece in a larger system. AI assistants such as those from Perplexity work, for example, with a combination of their own knowledge and current lookup actions, which has to do with how retrieval augmented generation works, a technique in which a language model supplements an answer with retrieved information from current sources, as explained in what is retrieval augmented generation.
Apart from llms.txt, it is useful to know which bots visit your site and what they are for. PerplexityBot, for example, retrieves current information for answers, while Applebot-Extended gathers data for Apple's AI features. You can read more about this in what is PerplexityBot and what is Applebot-Extended.
For an entrepreneur, the most important thing is not to get stuck on one technical detail. Llms.txt is useful, but the foundation remains: a clear site, current information and accessibility for the bots you want to allow. Those who have that in order stand stronger, with or without this specific file.
Frequently asked questions
Should every website have an llms.txt file?
No, it is not an obligation and not a standard that all AI systems use. It can help to present your most important pages clearly to AI assistants that do read the file. For smaller sites with a clear structure, the added value is often limited. For sites with many pages, it can offer a bit more clarity.
Does llms.txt regulate whether AI bots can visit my site?
No, that is a common misconception. Access for bots is regulated separately, per bot, via a different mechanism. Llms.txt is only a list of pages with explanations, intended to help AI assistants find their way around faster. Without permission for the bot itself, an llms.txt file for that bot makes no difference.
Does llms.txt also help with Google or Bing?
Not directly. The file was set up with AI assistants in mind, not traditional search engines. For your position at Google and Bing, other factors remain decisive, such as the quality of your content and the way their bots can crawl and understand your site. Llms.txt is an addition to that, not a replacement.
How is your own website doing?
Two figures, within a minute, free of charge — and you don't need to leave anything behind.
Prefer the full report right away? Comprehensive scan — € 49 →
Want to know what's causing it?
This scan gives you the status. The report gives you the causes and the order in which you address them.
- For each finding, what's wrong and what needs to happenThe free scan gives you the status. The report gives you the list — in plain language, ranked by how much it matters, without you needing to understand anything technical.
- All four AI assistants instead of oneThe free result is a sample from one assistant. The report asks eight questions of all four, so you know whether the issue is with one assistant or all of them.
- Which companies do get mentionedWho gets the customer you're losing? Those names are in the report, with how often they appear where you're missing.
- A deeper measurement of your websiteThe free scan looks at ten pages. The report goes through up to fifty, including the pages where your customers ultimately end up.