# AI crawler access

Paste a URL. The check reads robots.txt the way RFC 9309 says a crawler must, and reports what each documented AI crawler may fetch — at the home page and at the path you gave. There is no score: a blocked training crawler and a blocked search crawler are not the same event, and one number would hide which one happened.

## What the check reads

Three requests, all to the domain you name: robots.txt, llms.txt, and a HEAD of the home page. Nothing else is fetched, no page content is read, and no user agent is faked — the requests go out as seodraft-site-check.

robots.txt is parsed under RFC 9309: groups of consecutive user-agent lines merge, the longest matching rule wins, a tie goes to allow, and `*` and `$` work. A 4xx means no file and no restrictions. A 5xx, a 429, a timeout or a dead connection means the file is unreadable, which the RFC says a crawler must read as the whole site being disallowed.

Every crawler is evaluated twice: at the home page and at a content path. The content path is the URL you gave when it has one, because a site whose home is open and whose /blog/ is closed is a blocked site for anything that would cite an article.

## The crawlers it checks

One row per crawler, with the operator's own documentation behind each. A crawler whose operator does not document its token and what blocking it does is not on the list.

### Search and answers

The crawlers behind an AI product's citations. Blocked, the product cannot quote or link the page.

| token | operator | product | obeys robots.txt | what blocking it does | docs |
|---|---|---|---|---|---|
| `Googlebot` | Google | Google Search, AI Overviews, AI Mode | yes | Blocked means out of Google Search and every Search feature, AI Overviews included. | [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) |
| `OAI-SearchBot` | OpenAI | ChatGPT search | yes | Blocked pages are not shown in ChatGPT search answers. | [OpenAI](https://developers.openai.com/api/docs/bots) |
| `Claude-SearchBot` | Anthropic | Claude search | yes | Blocked pages are not indexed for Claude's search results. | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| `PerplexityBot` | Perplexity | Perplexity search | yes | Blocked pages are not surfaced or linked in Perplexity's results. | [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| `Applebot` | Apple | Siri, Spotlight, Safari search | yes | Blocked pages drop out of Apple's search features and the web answers in Siri and Search. | [Apple](https://support.apple.com/en-us/119829) |
| `meta-webindexer` | Meta | Meta AI | yes | Blocked pages are not cited or linked in Meta AI's answers. | [Meta](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) |

### Fetched when a person asks

Someone pastes a URL into a chat and the product opens it. Some of these ignore robots.txt by design, so a rule for them may not hold.

| token | operator | product | obeys robots.txt | what blocking it does | docs |
|---|---|---|---|---|---|
| `Claude-User` | Anthropic | Claude (user request) | yes | Blocked means Claude cannot open the page when a user asks about it. | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| `ChatGPT-User` | OpenAI | ChatGPT (user request) | no | User-initiated: OpenAI says robots.txt rules may not apply. It does not decide search inclusion. | [OpenAI](https://developers.openai.com/api/docs/bots) |
| `Perplexity-User` | Perplexity | Perplexity (user request) | no | User-initiated: Perplexity says it generally ignores robots.txt. | [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |

### Model training

Training only. Blocking these does not remove the site from the products above, by each operator's own documentation.

| token | operator | product | obeys robots.txt | what blocking it does | docs |
|---|---|---|---|---|---|
| `GPTBot` | OpenAI | OpenAI model training | yes | Blocked means the content is not used to train OpenAI's models. ChatGPT search is unaffected. | [OpenAI](https://developers.openai.com/api/docs/bots) |
| `ClaudeBot` | Anthropic | Anthropic model training | yes | Blocked means future content is excluded from Anthropic's training data. Claude search is unaffected. | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| `Google-Extended` | Google | Gemini training and grounding | yes | Blocked means no Gemini training and no grounding in Gemini Apps. Google Search is unaffected. | [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) |
| `Applebot-Extended` | Apple | Apple foundation model training | yes | Blocked means no training of Apple's foundation models. Apple search is unaffected. | [Apple](https://support.apple.com/en-us/119829) |
| `meta-externalagent` | Meta | Meta model training | yes | Blocked means Meta does not crawl the content to train its models. | [Meta](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) |
| `CCBot` | Common Crawl | Common Crawl open dataset | yes | Blocked means the pages stay out of Common Crawl, a dataset many models train on. | [Common Crawl](https://commoncrawl.org/ccbot) |

## What it cannot see

- Only robots.txt. A firewall, a WAF rule or a CDN policy can block a crawler that robots.txt allows, and none of that is visible from outside.
- Crawlers that ignore robots.txt. Two of the user-initiated fetchers are documented by their own operators as not bound by it; the check marks them and never reports them as blocked.
- Whether the crawler ever came. robots.txt is permission, not traffic. Server logs answer that question, this does not.
- Whether a product cites you. Permission is the floor, not the outcome.

## API

Request: `POST /api/ai-access` with `{ "url": "example.com/blog/a-post" }`.

```
curl -s https://seodraft.app/api/ai-access \
  -H "Content-Type: application/json" \
  -d '{"url":"example.com/blog/a-post"}'
```

### Response

- `origin`, `checkedAt`, and `paths` — the two paths every crawler was evaluated at: `/` and the content path (the URL's own path when it has one).
- `robots` — `availability` (`parsed` | `unavailable` | `unreachable`), `status`, `sitemaps`, and `contentSignal` as declared for `*`.
- `llmsTxt` — `present`, `status`, `hasH1`, `title`, `hasSummary`, `sections`; `null` when the request itself failed.
- `cdn` — `"cloudflare"` or `null`, from the `server` and `cf-ray` headers.
- `bots` — one entry per crawler: `token`, `operator`, `product`, `category`, `honorsRobots`, `allowedHome`, `allowedContent`, `matchedBy`, `rule`, `effect` (en and es), `docsUrl`.
- `findings` — `{ severity: "critical" | "warning" | "info", code, bots }`, in reading order. Codes: `googlebot-blocked`, `robots-unreachable`, `search-bots-blocked`, `user-bots-blocked`, `training-bots-blocked`, `cloudflare-edge`, `llms-txt-missing`.
- There is no score field, and there never will be one.

A URL that cannot be checked returns HTTP 400 and `{ degraded: true, reason, message }` with reason `invalid-url`, `blocked-target` or `dns-failed`. Rate limits answer 429; a database that cannot count answers 503, because every call makes outbound requests.

## For agents and scripts

POST a URL and read the same report back as JSON: the per-crawler verdicts, the findings, what robots.txt declares, and llms.txt. The Markdown twin of this page carries the crawler registry and the contract.

- [This page as Markdown](https://seodraft.app/tools/ai-crawlers.md)
- JSON endpoint: `POST https://seodraft.app/api/ai-access`

## Questions

### Does it give a score?

No. It reports which crawler may read which path, and what each operator says blocking it does.

### Should I unblock the training crawlers?

That is your decision and the check does not push it either way. OpenAI, Anthropic, Google and Apple all document that their training crawler is separate from the one behind their search and answers, so blocking training leaves citation untouched.

### Why is Googlebot on a list of AI crawlers?

AI Overviews and AI Mode are built on Google Search's index, and Googlebot is what fills it. Blocking Googlebot removes the site from both.

### Does it change anything on my site?

No. It reads three files and reports. The robots.txt snippet is text for you to paste, and nothing here can edit your server.