ClaudeBot: what it is and whether you should allow it

What exactly is ClaudeBot

ClaudeBot is an automated program from the company Anthropic. It visits websites, reads the text, and uses that text for two purposes: training the language model Claude, and answering questions that users are currently asking Claude. So it is not a human viewing your site, but software that retrieves pages on a large scale.

Claude is Anthropic's AI assistant, comparable to ChatGPT but with a different company behind it and a different method of operation. When someone asks Claude something about your sector, your region or your type of business, Claude can decide to search the internet for current information. If that happens and your site appears in those search results, it is ClaudeBot that has previously read your pages or is doing so at that moment.

The distinction from other AI robots lies in the behavior. Claude uses websearch, which means he can actively consult the web during a conversation instead of relying only on what he once learned during training. At the same time, Claude is reluctant to cite: he often does not mention the source from which he obtained information, even when he has indeed used it.

How does ClaudeBot differ from comparable robots

Anthropic is not the only company with such a robot. OpenAI has a comparable one, as we discuss at GPTBot: what it is and whether you should allow it. The technique is similar: both read web pages and use that text for an AI assistant. The difference lies mainly in what happens after the reading, and there ClaudeBot is a separate category.

Whereas some assistants often show a source citation or link when answering, Claude does this less consistently. So you cannot simply assume that he will mention a source as soon as he has used your information. This makes it more difficult to verify whether your texts have actually influenced an answer that a user saw.

This links to a question that many entrepreneurs also ask about ChatGPT, namely whether their website is actually read, as elaborated at does ChatGPT read your website, and how do you find out. With Claude, that question is even harder to answer, precisely because citing is not a fixed habit. You see less often in black and white that your site was the source.

Should you give ClaudeBot access to your site

There is no general good answer, and be wary of anyone who claims there is. Giving access means that your texts can be counted when Claude answers a question about your industry or your type of service. This can be favorable if potential customers use Claude to compare suppliers, something that is becoming increasingly common alongside the use of Google.

Refusing access can also be a valid choice. Some businesses do not want their texts, prices or product descriptions to be used to train an AI model without anything in return. That is a matter of principle, not technology. There is nothing wrong with deciding that you do not want that.

A practical middle way exists: you can give the robot access to answer current questions, but not to train the model in the long term. Anthropic provides separate rules for this within the file that tells robots what is and is not allowed, the so-called robots.txt file on your site. For most SME entrepreneurs, this is a setting that a technical administrator sets up once.

If you do not want to figure out yourself which choice is best, first look at what is already happening. Without knowing that, any decision is guesswork.

What does websearch mean for your discoverability

The fact that Claude uses web search means he doesn't only rely on outdated knowledge from his training, but can actively search during a conversation. This is relevant for you, because it makes the question of whether your site is readable and up-to-date important again. A page that hasn't been updated for six months makes a different impression than a page that shows recent information.

This works differently than with an assistant that only draws from training data. With web search, it matters less whether Claude ever learned about you during training, but whether he finds something useful about you at the moment the question is asked. That's a comparable principle to the refreshing of knowledge we discuss regarding how often ChatGPT refreshes what he knows about your company, although each assistant works in its own way.

For you, this mainly means that basics must be in order: clear text, no hidden information behind complicated technical structures, and a site that doesn't block what should simply be readable. Large technical interventions are rarely needed; a readable, up-to-date site is the most important step.

Why reluctant citation can be a problem

If Claude uses your information but doesn't mention where it comes from, the user won't see your company name. That's a real disadvantage compared to assistants that show sources more often. You may end up feeling that your efforts remain invisible, even if they genuinely have an effect on what is being said.

This makes measuring more difficult. With an assistant that cites often, you can, with a bit of searching, check whether your company is mentioned, as discussed under how to measure whether ChatGPT mentions your company. With Claude, you'll have to test more often yourself and pay close attention to whether your name, your services, or your specific approach come through in the answer, even without explicit source attribution.

Yet invisibility in source attribution is not the same as invisibility in the answer. It's possible that Claude is indeed using your information, only he's presenting it as general knowledge rather than as a quote. That's less satisfying for you to see, but the effect on the user reading the answer can still be present.

What you can concretely do

Start by checking whether ClaudeBot currently has access to your site. That should be stated in your robots.txt file. If there's nothing about Claude mentioned, then the robot is presumably welcome, because the default setting is usually accessible. If you want to adjust that, it's a small technical change.

Then ask Claude a number of test questions yourself about your industry, your service, or your region. Don't just look at whether your company is mentioned, but also whether the information is correct. Incorrect or outdated information about yourself is often a bigger problem than simply not being mentioned, and that applies equally to other assistants, as you can read under why ChatGPT mentions your competitor and not you.

Don't expect miracles from small adjustments. There are no tricks that are guaranteed to work with Claude, and be skeptical of providers who promise otherwise. What does help: clear texts, current information, and a site that presents no technical obstacles. That's the same foundation that also helps with other AI assistants and with regular search engines.