Skip to main content
A marketer checks how AI tools read a company website on a laptop
All postsGetting started

Can AI search read your site?

You can check this in about an hour. Read your robots.txt file, ask your host whether a firewall blocks search bots, confirm Google has indexed your key pages,

Quick answer

You can check this in about an hour. Read your robots.txt file, ask your host whether a firewall blocks search bots, confirm Google has indexed your key pages, then ask ten real customer questions in ChatGPT, Perplexity, Claude and Google and write down what each says about you. Fix the facts that are wrong or missing on your own pages first.

Key takeaways

  • Search crawlers, training crawlers and user-triggered fetchers are different things, and you can allow some and block others.
  • A blocked crawler cannot read your page, and a firewall can block one even when robots.txt allows it.
  • Google says no special file or markup is needed to appear in its AI features.
  • Test with ten real customer questions and record what each tool says.
  • The 30-minute assessment names your top two or three AI use cases.

Start your 30-minute AI assessment.

Which AI crawlers should your website allow?

Allow the search crawlers if you want to appear in AI answers, and decide separately about training crawlers. OpenAI's OAI-SearchBot supports ChatGPT search, Perplexity's PerplexityBot supports its search, and Anthropic's Claude-SearchBot supports its search quality. GPTBot and ClaudeBot relate to model training. Each can be set on its own.

The companies publish the names of their crawlers and what each does. As of 1 November 2026, the vendor pages say:

| Company | Crawler | What the vendor says it is for | robots.txt | |---|---|---|---| | OpenAI | OAI-SearchBot | Surfacing sites in ChatGPT search results | Honors it | | OpenAI | GPTBot | Collecting content that may train models | Honors it | | OpenAI | ChatGPT-User | Actions a user triggers in ChatGPT | Rules may not apply | | Perplexity | PerplexityBot | Surfacing sites in Perplexity search, not training | Honors it | | Perplexity | Perplexity-User | Fetching a page when a user asks | Generally ignores it | | Anthropic | ClaudeBot | Collecting content that may train models | Honors it | | Anthropic | Claude-SearchBot | Improving the quality of search results | Honors it | | Anthropic | Claude-User | Retrieval a user directs | Honors it |

OpenAI says each of its settings is independent, so you can allow OAI-SearchBot and disallow GPTBot (OpenAI). Perplexity describes its two bots in its guide (Perplexity). Anthropic's support page says its three bots honor robots.txt (Anthropic, dated 7 April 2026). Google offers a separate token, Google-Extended, to limit use of your content for training and grounding (Google). These pages change, so check the vendor's page before you edit anything.

How do you check robots.txt and your firewall?

Type your domain followed by /robots.txt into a browser and read it. Look for a Disallow line that blocks everything, or one that names a search bot. Then ask your host or website company whether a firewall or content delivery network blocks automated visitors, because that can stop a crawler even when robots.txt allows it.

Three checks, in order:

  1. Read robots.txt. A line pair of User-agent: * and Disallow: / blocks every well-behaved crawler from the whole site. A line naming a bot with Disallow: / blocks that bot.
  2. Ask about bot protection. Many hosts and website builders switch on bot blocking by default. Ask in writing: "Are OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot allowed through our firewall and CDN?"
  3. Check the page itself. Make sure it is not marked noindex and that the answer to a customer question appears as text on the page, not only inside an image or a PDF.

Google says that for its AI features, a page needs to be indexed and eligible to show with a snippet, and that crawling must be allowed in robots.txt and by any CDN or hosting service (Google). The same logic applies to the other tools: a bot that cannot fetch a page cannot read it.

Worked example (hypothetical, not a client case). An HVAC company in Tustin finds its robots.txt allows everything. Its host's security setting, however, returns an error to unknown bots. After the host adjusts the setting, the company's service pages load for the search crawlers. No results are claimed.

What does Google say you need to appear in AI Overviews?

Google says there are no additional requirements. A page needs to be indexed and eligible to appear with a snippet, and the usual search guidance applies. Google adds that you do not need to create new machine readable files, AI text files or markup to appear in these features. Structured data should match the visible text.

Google's page on AI features and your website says AI Overviews and AI Mode use the same foundational search practices, with no extra technical requirement. It also states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features" (Google).

That makes the work plain. Be crawlable. Be indexed. Put the useful answer in readable text. If you add structured data, it has to match what visitors can see on the page.

For the other AI tools, the vendors' pages describe their crawlers but do not give a ranking recipe, so treat any promise of guaranteed placement in an AI answer with suspicion.

How do you test what AI tools say about your business?

Write ten questions your customers really ask, such as who does a job in your city, what it costs and how long it takes. Ask each one in ChatGPT, Perplexity, Claude and Google. For every answer, record whether you are named, whether the facts are right, and which sources it cites. Repeat the test monthly.

Use a simple table with these columns: question, tool, were we named, were the facts right, sources cited, and what to fix. Ask in plain language the way a customer would, and include a few that name your city. Ask the same question twice, because answers vary from one run to the next.

A few cautions. Do not read too much into one answer. A tool that names a competitor does not mean you did something wrong. A tool that states a wrong price or the wrong service area is useful information: it tells you which fact is unclear or missing on your own pages.

The related guide How Can Orange County Businesses Get Found by AI Agents? covers the broader picture of being found.

What should you fix first when an AI answer is wrong?

Fix the source, not the answer. Find the page or listing the tool cited, correct the fact there, and state it plainly on your own site: service area, hours, prices or price ranges, licenses and what you do not do. Then retest after a few weeks. Start with the facts customers ask about most.

Typical causes of a wrong answer: an old directory listing, a page that gives a price in an image only, a service area written as a vague phrase like "the surrounding area," or two pages on your site that disagree. A one-page fact sheet with your zip codes, hours, services, standard prices and the words customers use is a good base. The same sheet helps staff and any AI assistant you build later; see How do you train an AI assistant on your own business?.

Add the fixes to your own plan. If you do not know which fix matters most, the tests in What processes should be automated first? help you choose where to spend time.

Who helps Orange County businesses check their AI visibility?

Preferred AI Partners is a fractional AI partner in Irvine that works with businesses of 10 to 200 people. It can walk through this check with you and point out the facts that are unclear. The 30-minute assessment looks at how customers find you and names where to start.

See the Orange County page and the services page. Ask any provider that sells AI visibility work to show you the specific pages it will change and how it will measure the result.

More on this

Share this post

Sebastian Alidad

Founder, Preferred AI Partners

Sebastian is a founder of Preferred AI Partners and runs the AI audit call.