# Audit AI Crawler Permissions Across Robots and WAF

> Source: CitedIndex — https://citedindex.com/guides/audit-ai-search-crawler-permissions/
> By: CitedIndex Editorial, Research desk
> Claims last verified: 2026-09-12
> Last updated: 2026-09-12
> Covers: Separate intended search, training and user-requested fetch permissions, Inspect deployed robots.txt alongside CDN/WAF responses and origin evidence, Use official live IP references without user-agent-only trust or broad firewall bypasses, Diagnose a hypothetical OAI-SearchBot permission and WAF challenge mismatch, Distinguish PerplexityBot, Perplexity-User and Google's Google-Extended product token, Record evidence limits and maintain one canonical audit procedure

A documentation-based audit of search, training and user-request permissions: compare robots.txt with CDN/WAF evidence before changing access.

Audit permissions in three separate categories—search, training and user-requested fetching—then compare your intended policy with the deployed robots.txt file and the CDN/WAF evidence for actual requests. Record contradictions before changing either layer. Treat crawler access, content-use preferences and observed answer citations as separate audit records.

This is a documentation-based walkthrough. Account-level steps were not exercised by CitedIndex; the configuration and logs below are hypothetical, not observations from a live site. Source-check date: 12 September 2026, using the sources linked in this guide. The live IP feeds themselves were not fetched for this guide.

## 1. Write the intended policy before inspecting traffic

Create one audit row per hostname, path scope and provider purpose. For each row, choose an intended outcome: allow, deny or unresolved. Keep unresolved decisions visible rather than silently treating them as permission.

- **Search:** Which public pages do you want available to the provider's search crawler?
- **Training or other documented content use:** Which uses do you intend to permit? Record the provider's actual control scope rather than assuming every provider offers the same training switch.
- **User-requested fetching:** Should a request made on behalf of a user reach those pages?
- **Protected paths:** Identify account, administrative and other restricted routes that must remain outside any proposed crawler exception.

OpenAI documents OAI-SearchBot and GPTBot settings as independent, allowing a site to permit search crawling while disallowing training crawling. [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots.md)

### Purpose, control, verification and limitation matrix

The verification column contains suggested audit actions, not checks performed by CitedIndex. Use the linked documentation to interpret each control.

| Agent or token | Documented purpose | Control to inspect | Suggested verification | Limitation to retain |
| --- | --- | --- | --- | --- |
| OAI-SearchBot | OAI-SearchBot surfaces websites in ChatGPT search results. [Source](https://developers.openai.com/api/docs/bots.md) | Inspect its robots.txt group and the edge rules applied to its published IP ranges. | Compare the applicable robots rule with identified searchbot requests and their returned content. | Do not mark citation or traffic as established by access. |
| GPTBot | GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. [Source](https://developers.openai.com/api/docs/bots.md) | Inspect its robots.txt group separately from OAI-SearchBot. | Record the deployed directive and any observed requests separately from searchbot traffic. | Do not label this audit as evidence of training-data deletion. |
| ChatGPT-User | ChatGPT-User supports certain user actions in ChatGPT and Custom GPTs and is not an automatic web crawler. [Source](https://developers.openai.com/api/docs/bots.md) | Inspect edge access separately from automatic search-crawl preferences. | Compare the claimed agent and source IP with the provider reference; record any known user-request context. | Do not substitute this row for the search-crawler row. |
| PerplexityBot | PerplexityBot is designed to surface and link websites in Perplexity search results, not to crawl content for AI foundation models. [Source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md) | Inspect its robots.txt group and matching edge rules. | Compare both the agent string and source IP with the current provider references. | Do not classify this as a training crawler. |
| Perplexity-User | Perplexity-User supports user actions and is not used for web crawling or collecting content to train AI foundation models. [Source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md) | Inspect user-fetch access at the edge separately from PerplexityBot. | Use its own published IP reference and request logs. | Do not assume the searchbot's robots behavior applies. |
| Googlebot | Googlebot crawl preferences affect Google Search, including Discover and all Google Search features. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) | Inspect Googlebot robots rules and edge handling together. | Use Google's common-crawler identity references when reviewing requests. | Review Search impact before changing this row. |
| Google-Extended | Google-Extended is a standalone product token, not a separate HTTP request user-agent. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) | Inspect the Google-Extended robots.txt group as a content-use preference. | Save the deployed directive; do not require a Google-Extended HTTP log entry as proof. | Do not invent a separate firewall identity for this token. |

Google-Extended controls whether crawled content may be used to train future Gemini models powering Gemini Apps and the Vertex AI API for Gemini, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. [Google's documented scope](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

Google says Google-Extended does not affect inclusion or ranking in Google Search. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) If the decision is specifically about Google AI Search rather than cross-provider crawl access, use the separate guide to [Google AI Search content controls](/guides/google-ai-search-content-controls).

## 2. Capture the deployed robots.txt evidence

1. For each hostname and scheme in scope, retrieve its public robots.txt through the same public edge that serves the site. Save the retrieval time, requested URL, redirect chain, status, headers and body. Keep the configuration repository's version alongside the public response, not instead of it.
2. Read the complete file. Record the named groups, wildcard group and path directives relevant to each audit row. Flag overlapping or ambiguous rules for interpretation against the provider's documentation rather than guessing.
3. Compare the deployed directives with the intended policy. Give each row a finding such as aligned, policy mismatch, unavailable evidence or unresolved rule interpretation.
4. Include robots.txt requests in the edge-log review as well as content requests. Keep response bodies or safe diagnostic excerpts so that the audit does not rely on status codes alone.

Google says its common crawlers always obey robots.txt when crawling automatically. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) Keep the qualification “automatically” in your audit notes; do not extend this statement to every Google user-triggered fetcher.

### User-requested fetching needs a separate decision

OpenAI says robots.txt rules may not apply to ChatGPT-User because its actions are initiated by a user. [Source](https://developers.openai.com/api/docs/bots.md) ChatGPT-User is not used to determine whether content may appear in Search. [Source](https://developers.openai.com/api/docs/bots.md)

Perplexity says Perplexity-User generally ignores robots.txt because a user requested the fetch. [Source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md) Preserve the providers' different wording: “may not apply” is not the same statement as “generally ignores.” For a deny policy on user-requested access, review the site's enforceable access controls rather than treating a robots directive as sufficient evidence.

## 3. Compare identity, edge decisions and origin evidence

Suggested audit fields are: timestamp and timezone, hostname, path, source IP as recorded by the trusted edge, full user-agent, request ID, matched rule ID, action, response status, response type, origin disposition, and the provider IP-reference snapshot used for comparison. Record missing fields explicitly.

Perplexity recommends combining user-agent matching with IP-address verification when configuring WAF rules. [Source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md) Apply a conservative audit rule: do not label traffic legitimate from its user-agent string alone, and do not test authenticity by merely sending a request with that string from your own machine.

### Use live provider references, not copied IP lists

OpenAI publishes separate IP references for OAI-SearchBot, GPTBot and ChatGPT-User. [OpenAI's endpoint references](https://developers.openai.com/api/docs/bots.md)

- OAI-SearchBot: `https://openai.com/searchbot.json`
- GPTBot: `https://openai.com/gptbot.json`
- ChatGPT-User: `https://openai.com/chatgpt-user.json`

Perplexity directs WAF users to the most current IP ranges in its official JSON endpoints because those addresses are updated regularly. [Perplexity's endpoint references](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md)

- PerplexityBot: `https://www.perplexity.com/perplexitybot.json`
- Perplexity-User: `https://www.perplexity.com/perplexity-user.json`

Google says its common crawlers generally use the ranges in common-crawlers.json, with reverse-DNS hostname masks matching crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. [Google's common-crawler reference](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

For Google, follow the `common-crawlers.json` reference from the official page. For every provider, save the reference and retrieval time used in your assessment; do not turn this guide into a fixed IP allowlist. Treat Google's hostname masks as documented identity context, not a user-agent-only trust rule.

1. Group observed traffic by intended purpose, not merely provider name. Keep unidentified requests in an unresolved bucket.
2. For each identifiable request, compare the applicable robots policy with the edge action and the actual response content. Inspect redirects, challenge responses and access denials explicitly.
3. Correlate edge and origin records by request ID where available. If origin evidence is missing, record that gap rather than assigning a cause from absence alone.
4. List each contradiction with its evidence and owner before proposing a rule change.

The Perplexity WAF examples describe Allow rules that can bypass other security rules. [Configuration examples](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md) This walkthrough does not recommend copying that broad bypass. As a suggested security review, identify the particular blocking or challenge rule first, then consider only a narrowly scoped adjustment for the intended agent, current provider ranges and approved public paths. Retain unrelated protections and review rule ordering with the CDN/WAF owner.

## 4. Worked example: robots permits search, but the WAF challenges it

**Hypothetical policy:** permit OpenAI search crawling of public pages, disallow GPTBot training crawling, and leave user-requested access unresolved pending a separate decision. This illustrative excerpt is not a replacement robots.txt file:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /
```

OpenAI says disallowing GPTBot indicates that a site's content should not be used to train generative AI foundation models. [Source](https://developers.openai.com/api/docs/bots.md)

**Illustrative logs only:** the following records assume the auditor has separately compared the edge-recorded source IP with a saved, current searchbot.json response. Agent strings are shortened for readability; no real request, IP lookup or account test is represented.

```
time=T1 request_id=R1 path=/robots.txt
agent=OAI-SearchBot source_ip=EXAMPLE_SEARCH_IP
identity_note=matches_saved_searchbot_feed
edge_action=pass status=200 body=expected_robots_file

time=T2 request_id=R2 path=/public/article
agent=OAI-SearchBot source_ip=EXAMPLE_SEARCH_IP
identity_note=matches_saved_searchbot_feed
edge_action=managed_challenge rule_id=EXAMPLE_RULE
status=403 body=challenge_page origin_forwarded=false
```

**Hypothetical diagnosis:** the saved robots response aligns with the search permission, but the content request receives a WAF challenge instead of the article. The contradiction is between intended search access and the edge response, not between the OAI-SearchBot and GPTBot directives.

Before accepting that diagnosis on a real site, collect:

- The complete public robots.txt response and the relevant rules for the article path.
- The full user-agent and trusted edge source IP, plus the provider-feed snapshot used to assess identity. A note saying “searchbot” is not enough.
- The edge event identifying the challenge rule, its deployed version and the request ID.
- The returned challenge content or diagnostic evidence distinguishing it from the article.
- The edge forwarding disposition and any matching origin record, with evidence gaps marked.

Record the finding before changing anything: `Search intended: allow; robots evidence: allow; edge evidence: challenge; GPTBot policy: disallow; proposed owner: CDN/WAF team.` If identity remains unresolved, change the finding to “claimed searchbot challenged; identity unresolved” and do not create an exception from the agent string alone.

A suggested next step is to review a narrow adjustment to the identified challenge rule while leaving GPTBot's directive intact. After any approved change, collect fresh edge and response evidence for the affected public page and check that protected routes remain protected. Do not present a local agent-string request as proof of provider access.

## 5. Close the audit without overstating the result

OpenAI says that when both bots are allowed, it may use the results of one crawl for both search and training to avoid duplicate crawling. [Source](https://developers.openai.com/api/docs/bots.md) Audit implication: do not infer a complete content-use history by counting OAI-SearchBot and GPTBot visits.

OpenAI says its search systems can take approximately 24 hours to adjust after a robots.txt update. [Source](https://developers.openai.com/api/docs/bots.md) Perplexity says its systems may take up to 24 hours to reflect changes to crawler settings. [Source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md) Use these as documented adjustment windows, not promises of a crawl, citation or traffic.

Finish with a ledger containing intended policy, deployed directive, identity evidence, observed response, mismatch, proposed action, owner and follow-up evidence. Use “no observation” when appropriate; do not turn it into a pass or a failure.

**Evidence boundaries:** this audit can document a directive and an observed access outcome; do not use it to claim guaranteed inclusion, citation, referral traffic or deletion from training data. For measurement, keep a bot visit, brand mention, linked citation, referral visit, API response and consumer-interface observation in separate fields. Do not count a bot visit as a citation, a brand mention as a linked citation, or an API observation as a consumer-interface observation. For answer-level evidence, use [the citation-support audit](/guides/audit-ai-answer-citation-support).

**Suggested maintenance:** review this educational procedure monthly. Recheck the relevant live provider references when assessing traffic or proposing an access change. If a provider changes the meaning of a token, its robots behavior or its identity guidance, review and update this canonical guide rather than publishing a date-stamped duplicate. Treat this source check as a baseline, not evidence of a newly announced change.

## Sources

- [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots.md)
- [Perplexity: Crawlers and WAF configuration](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md)
- [Google: List of common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)