Skip to main content
A property manager in Tustin reviews the robots.txt file for the company website with a web developer.
All postsGetting started

How Claude finds and cites you

Claude searches the web when a question depends on current or changing facts, such as details about a specific business, and its search answers cite the pages

Quick answer

Claude searches the web when a question depends on current or changing facts, such as details about a specific business, and its search answers cite the pages used. Three Anthropic bots matter: ClaudeBot gathers training data, Claude-SearchBot indexes pages for search, and Claude-User fetches pages when someone asks. Blocking the last two may reduce your visibility in Claude. Clear, current, quotable pages give Claude something to cite.

Source: Claude Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?, 2026.

Source: Claude Platform Docs, Web search tool, 2026.

Key takeaways

  • Claude tends to search when a question involves a specific business or facts that change.
  • Anthropic runs three bots with separate robots.txt names: ClaudeBot, Claude-SearchBot and Claude-User.
  • Blocking ClaudeBot is a training choice. Blocking the other two may cost you visibility in Claude.
  • Since October 30, 2026, Anthropic has begun honoring a new machine-readable no AI training signal.
  • Short, plain, dated answers near the top of a page are easier for Claude to quote and cite.

Start your 30-minute AI assessment.

When does Claude search the web instead of answering from memory?

Claude searches when a question depends on information that's current, changing or outside its training data. Anthropic's developer documentation lists recent news, current prices, direct requests to look something up, and details about specific organizations, people or products that might have changed. A customer asking about a local company falls in that last group, which is where your website comes in.

Anthropic's documentation for Claude's web search tool spells out the pattern. It says "Claude searches when the request depends on information that is current, changing, or outside its training data." One of the examples it gives is "Information about specific organizations, people, or products that might have changed."

It also lists what Claude answers without searching: established facts, math, creative writing, and analysis of content already in the conversation.

For an Orange County business, that split matters. A general question about how a heat pump works may never touch your site. A question asking which companies service commercial HVAC in Irvine, and whether they offer maintenance plans, is about specific companies, so it can trigger a search, and the answer is built from the pages Claude finds.

One honest caveat: that documentation describes the web search tool developers use. Anthropic doesn't publish the same level of detail for the Claude apps, so treat it as the best official guide to the pattern, not a promise about every chat.

If answer engines are new to you, What is answer engine optimization? covers the basics.

Which Anthropic crawlers visit your website, and what does each do?

Anthropic names three bots in its Help Center. ClaudeBot collects public web content that could be used to train its models. Claude-SearchBot crawls pages to improve Claude's search results. Claude-User fetches a page when a person asks Claude something that needs it. Each one has its own name in robots.txt, so you can allow or block them separately.

Here's how Anthropic's Help Center describes each one:

  • ClaudeBot collects "web content that could potentially contribute to their training." It's the training crawler.
  • Claude-SearchBot "navigates the web to improve search result quality for users." It's the search indexer.
  • Claude-User covers live lookups. Anthropic says "When individuals ask questions to Claude, it may access websites" using this agent.

The same page points to a fourth identifier, Claude-Agent, for browsers Anthropic hosts so Claude can complete tasks for users, including in its Claude Duet preview. That one is about Claude acting on a site, such as filling in a form, rather than reading it for an answer.

Anthropic says its bots honor robots.txt and support the Crawl-delay setting if you want to slow them down. It also warns that blocking its IP addresses instead "may not work correctly or persistently guarantee an opt-out," because that stops the bots from reading your robots.txt file at all.

That warning matters if a hosting company or firewall service blocks unfamiliar bots for you. Ask whoever runs your site whether Anthropic's bots are being turned away before they ever see your rules.

Should you block ClaudeBot but let Claude's search bots in?

For most service businesses, letting Claude-SearchBot and Claude-User in is the safer default, because Anthropic says blocking either one may reduce your visibility in Claude's answers. ClaudeBot is a separate decision about training. Blocking it signals that your future pages should be left out of training data, and Anthropic describes that effect only for training, not for search.

Anthropic's own wording on what each block does:

  • Blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."
  • Blocking Claude-User "may reduce your site's visibility for user-directed web search."
  • Blocking ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets."

Note the word "future." A ClaudeBot block covers pages from now on.

If you want to stay out of training but keep showing up in Claude's search answers, a robots.txt file can name each bot separately. A sketch your web developer can adapt:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Anthropic says to add rules for every subdomain you want covered, such as a separate booking or portal subdomain.

If you have no strong view on training, you can leave all three alone. Many small firms never name Anthropic's bots at all, and that's a valid choice. The point is to decide on purpose rather than inherit a blanket block from a template or plugin.

What does Anthropic's new no AI training signal change for you?

Since October 30, 2026, Anthropic has begun recognizing a machine-readable signal that lets site owners say whether their content may be used to train AI models. It follows draft work from the IETF, the group behind many internet standards. If your robots.txt names ClaudeBot, Anthropic follows that rule first. If it doesn't, Anthropic follows the general signal.

Anthropic's Help Center says it will "honor a new preference signal that lets website owners say" no to AI training, and that it's rolling this out "across all of our services over time and globally."

It's open about the setting being new: "Because standards in this space are still evolving, the way we read and apply these settings may change too."

What it means in practice:

  • It's about training only. The page ties the signal to training, not to Claude-SearchBot or Claude-User.
  • Named rules win. If your robots.txt names ClaudeBot, that rule takes priority over the general signal for the same path.
  • No action is required. If you're happy with your current setup, nothing breaks.

For most Orange County businesses, the simplest route is still the ClaudeBot line in robots.txt, because Anthropic says it treats that as the more specific instruction. The general signal is more useful if you want one setting that speaks to several AI companies at once and your website platform supports it.

If you're unsure what your site sends today, the glossary explains robots.txt and related terms in plain words.

What makes a page easy for Claude to quote in an answer?

Anthropic's web search documentation says citations are always on, and each citation can carry up to 150 characters of the text Claude used. That favors pages that state a fact in one plain sentence near the top: what you do, where, for whom, and when the page was last updated. Long intros, slogans and details hidden in images give Claude less to quote.

Two details from Anthropic's documentation are worth knowing. It says "Citations are always enabled for web search," and each search result includes a page age, described as "When the site was last updated."

Neither detail tells you how Claude ranks pages, and Anthropic doesn't publish that. But they point to habits that help any answer engine:

  • Answer first. Open each service page with one sentence a customer could repeat, such as: We repair and replace commercial rooftop HVAC units across central Orange County.
  • One page per service. A single page listing twelve services gives Claude little to cite for any one of them.
  • Name the towns. Write out the cities you serve instead of a vague phrase about the region.
  • Keep facts in text. Hours, service areas and price ranges inside an image can't be quoted.
  • Show a date. A visible last reviewed date helps a reader, and a stale page is easy to skip.
  • Match your listings. Use the same business name, phone and address on your site and your profiles.

The contractor guide to AI recommendations shows the same signals for trades firms, and How Orange County businesses get found by AI agents covers business details that agents read.

What should an Orange County business change first for Claude?

Start with access, then content. Check that your robots.txt and any firewall or hosting bot settings let Claude-SearchBot and Claude-User reach your pages. Decide on purpose whether ClaudeBot may collect your pages for training. Then rewrite the top paragraph of your three most important service pages so each one answers a real customer question in a sentence.

A short order of work for a 10 to 200 person company:

  1. Open your robots.txt. It lives at yoursite.com/robots.txt. Look for the names ClaudeBot, Claude-SearchBot, Claude-User, or a rule that blocks every bot.
  2. Ask about bot blocking. If a hosting company, security plugin or CDN filters bots, ask whether Anthropic's bots get through.
  3. Make the training call. Leave ClaudeBot alone or add a block, and write down who decided and why.
  4. Fix three pages. Pick the services that bring in the most revenue and give each page a plain first sentence, the towns you cover, and a review date.
  5. Recheck quarterly. Bot names and settings change. Anthropic's Help Center page is the place to confirm the current list.

If you'd like a second set of eyes on the content side, our Orange County page explains how we work with local firms, and Train an AI assistant on your business covers getting your facts written down in one place.

Worked example (hypothetical, not a client case). A 40-person property management company in Tustin finds that its website host blocks all unfamiliar bots by default, including Anthropic's. The owner asks the host to let Claude-SearchBot and Claude-User through, chooses to block ClaudeBot, and has the office manager rewrite the first paragraph of the owner services page to name the cities and property types the company manages. No results are claimed.

More on this

Share this post

Sebastian Alidad

Founder, Preferred AI Partners

Sebastian is a founder of Preferred AI Partners and runs the AI audit call.