seodraft
Start · $5/mo

AI crawler access

Paste a URL. The check reads robots.txt the way RFC 9309 says a crawler must, and reports what each documented AI crawler may fetch — at the home page and at the path you gave. There is no score: a blocked training crawler and a blocked search crawler are not the same event, and one number would hide which one happened.

What the check reads

Three requests, all to the domain you name: robots.txt, llms.txt, and a HEAD of the home page. Nothing else is fetched, no page content is read, and no user agent is faked — the requests go out as seodraft-site-check.

robots.txt is parsed under RFC 9309: groups of consecutive user-agent lines merge, the longest matching rule wins, a tie goes to allow, and `*` and `$` work. A 4xx means no file and no restrictions. A 5xx, a 429, a timeout or a dead connection means the file is unreadable, which the RFC says a crawler must read as the whole site being disallowed.

Every crawler is evaluated twice: at the home page and at a content path. The content path is the URL you gave when it has one, because a site whose home is open and whose /blog/ is closed is a blocked site for anything that would cite an article.

The crawlers it checks

One row per crawler, with the operator's own documentation behind each. A crawler whose operator does not document its token and what blocking it does is not on the list.

Search and answers

The crawlers behind an AI product's citations. Blocked, the product cannot quote or link the page.

  • Googlebot Google · Google Search, AI Overviews, AI Mode — Blocked means out of Google Search and every Search feature, AI Overviews included. operator docs
  • OAI-SearchBot OpenAI · ChatGPT search — Blocked pages are not shown in ChatGPT search answers. operator docs
  • Claude-SearchBot Anthropic · Claude search — Blocked pages are not indexed for Claude's search results. operator docs
  • PerplexityBot Perplexity · Perplexity search — Blocked pages are not surfaced or linked in Perplexity's results. operator docs
  • Applebot Apple · Siri, Spotlight, Safari search — Blocked pages drop out of Apple's search features and the web answers in Siri and Search. operator docs
  • meta-webindexer Meta · Meta AI — Blocked pages are not cited or linked in Meta AI's answers. operator docs

Fetched when a person asks

Someone pastes a URL into a chat and the product opens it. Some of these ignore robots.txt by design, so a rule for them may not hold.

  • Claude-User Anthropic · Claude (user request) — Blocked means Claude cannot open the page when a user asks about it. operator docs
  • ChatGPT-User OpenAI · ChatGPT (user request) — User-initiated: OpenAI says robots.txt rules may not apply. It does not decide search inclusion. operator docs
  • Perplexity-User Perplexity · Perplexity (user request) — User-initiated: Perplexity says it generally ignores robots.txt. operator docs

Model training

Training only. Blocking these does not remove the site from the products above, by each operator's own documentation.

  • GPTBot OpenAI · OpenAI model training — Blocked means the content is not used to train OpenAI's models. ChatGPT search is unaffected. operator docs
  • ClaudeBot Anthropic · Anthropic model training — Blocked means future content is excluded from Anthropic's training data. Claude search is unaffected. operator docs
  • Google-Extended Google · Gemini training and grounding — Blocked means no Gemini training and no grounding in Gemini Apps. Google Search is unaffected. operator docs
  • Applebot-Extended Apple · Apple foundation model training — Blocked means no training of Apple's foundation models. Apple search is unaffected. operator docs
  • meta-externalagent Meta · Meta model training — Blocked means Meta does not crawl the content to train its models. operator docs
  • CCBot Common Crawl · Common Crawl open dataset — Blocked means the pages stay out of Common Crawl, a dataset many models train on. operator docs

What it cannot see

  • Only robots.txt. A firewall, a WAF rule or a CDN policy can block a crawler that robots.txt allows, and none of that is visible from outside.
  • Crawlers that ignore robots.txt. Two of the user-initiated fetchers are documented by their own operators as not bound by it; the check marks them and never reports them as blocked.
  • Whether the crawler ever came. robots.txt is permission, not traffic. Server logs answer that question, this does not.
  • Whether a product cites you. Permission is the floor, not the outcome.

For agents and scripts

POST a URL and read the same report back as JSON: the per-crawler verdicts, the findings, what robots.txt declares, and llms.txt. The Markdown twin of this page carries the crawler registry and the contract.

POST /api/ai-access

curl -s https://seodraft.app/api/ai-access \
  -H "Content-Type: application/json" \
  -d '{"url":"example.com/blog/a-post"}'

Questions

Does it give a score?

No. It reports which crawler may read which path, and what each operator says blocking it does.

Should I unblock the training crawlers?

That is your decision and the check does not push it either way. OpenAI, Anthropic, Google and Apple all document that their training crawler is separate from the one behind their search and answers, so blocking training leaves citation untouched.

Why is Googlebot on a list of AI crawlers?

AI Overviews and AI Mode are built on Google Search's index, and Googlebot is what fills it. Blocking Googlebot removes the site from both.

Does it change anything on my site?

No. It reads three files and reports. The robots.txt snippet is text for you to paste, and nothing here can edit your server.