Crawler owner documentation, read 2026-11-02 for the UK news robots.txt monthly snapshot (jacobwhitmore.com). Text is each page's main content (document innerText) as read in Chrome. Requested URL, final URL after redirects and read time (UTC) are given. ### CRAWLER: GPTBot | URL: https://platform.openai.com/docs/bots | FINAL URL: https://developers.openai.com/api/docs/bots | READ: 2026-11-02T09:20:31.727Z | PAGE TEXT: see section openai below ### CRAWLER: OAI-SearchBot | URL: https://platform.openai.com/docs/bots | FINAL URL: https://developers.openai.com/api/docs/bots | READ: 2026-11-02T09:20:31.727Z | PAGE TEXT: see section openai below ### CRAWLER: ChatGPT-User | URL: https://platform.openai.com/docs/bots | FINAL URL: https://developers.openai.com/api/docs/bots | READ: 2026-11-02T09:20:31.727Z | PAGE TEXT: see section openai below ### CRAWLER: ClaudeBot | URL: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | FINAL URL: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | READ: 2026-11-02T09:20:38.197Z | PAGE TEXT: see section anthropic below ### CRAWLER: Claude-SearchBot | URL: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | FINAL URL: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | READ: 2026-11-02T09:20:38.197Z | PAGE TEXT: see section anthropic below ### CRAWLER: Claude-User | URL: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | FINAL URL: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | READ: 2026-11-02T09:20:38.197Z | PAGE TEXT: see section anthropic below ### CRAWLER: anthropic-ai | URL: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | FINAL URL: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | READ: 2026-11-02T09:20:38.197Z | PAGE TEXT: see section anthropic below ### CRAWLER: Google-Extended | URL: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers | FINAL URL: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers | READ: 2026-11-02T09:20:46.241Z | PAGE TEXT: see section google below ### CRAWLER: PerplexityBot | URL: https://docs.perplexity.ai/guides/bots | FINAL URL: https://docs.perplexity.ai/docs/resources/perplexity-crawlers | READ: 2026-11-02T09:20:52.747Z | PAGE TEXT: see section perplexity below ### CRAWLER: Perplexity-User | URL: https://docs.perplexity.ai/guides/bots | FINAL URL: https://docs.perplexity.ai/docs/resources/perplexity-crawlers | READ: 2026-11-02T09:20:52.747Z | PAGE TEXT: see section perplexity below ### CRAWLER: CCBot | URL: https://commoncrawl.org/ccbot | FINAL URL: https://commoncrawl.org/ccbot | READ: 2026-11-02T09:21:15.044Z | PAGE TEXT: see section commoncrawl below ### CRAWLER: Applebot-Extended | URL: https://support.apple.com/en-us/119829 | FINAL URL: https://support.apple.com/en-us/119829 | READ: 2026-11-02T09:21:21.800Z | PAGE TEXT: see section apple below ### CRAWLER: Bytespider | URL: none | READ: 2026-11-02 | No owner documentation page was found for this crawler on this date. Class left as published. ### CRAWLER: Meta-ExternalAgent | URL: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers | FINAL URL: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers | READ: 2026-11-02T09:21:30.675Z | PAGE TEXT: see section meta below ### CRAWLER: Amazonbot | URL: https://developer.amazon.com/amazonbot | FINAL URL: https://developer.amazon.com/docs/amazonbot/amazonbot.html | READ: 2026-11-02T09:21:38.731Z | PAGE TEXT: see section amazon below ================ SECTION openai | Overview of OpenAI Crawlers | https://developers.openai.com/api/docs/bots | read 2026-11-02T09:20:31.727Z ================ Copy Page Overview of OpenAI Crawlers OpenAI uses web crawlers (“robots”) and user agents to perform actions for its products, either automatically or triggered by user request. OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI. Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models. If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling. For search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust. User agent Description & details OAI-SearchBot OAI-SearchBot is for search. OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below. Example user-agent string (the version number may change): Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot When fetching robots.txt files, we may use a user-agent string with an additional robots.txt marker. The marker helps site owners more easily distinguish requests for the robots.txt file from requests for other resources, especially when their logs do not include paths: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot Published IP addresses: https://openai.com/searchbot.json OAI-AdsBot OAI-AdsBot is used to validate the safety of web pages submitted as ads on ChatGPT. When you submit an ad, OpenAI may visit the landing page to ensure it complies with our policies. We may also use content from the landing page to determine when it’s most relevant to show the ad to users. OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models. Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot Published IP addresses: https://openai.com/adsbot.json GPTBot GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models. Example user-agent string (the version number may change): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot When fetching robots.txt files, we may add a robots.txt marker to the user-agent string to help site owners distinguish those requests from requests for other resources, especially when logs do not include paths. Published IP addresses: https://openai.com/gptbot.json ChatGPT-User OpenAI also uses ChatGPT-User for certain user actions in ChatGPT and Custom GPTs. When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent. ChatGPT users may also interact with external applications via GPT Actions. ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. ChatGPT-User is not used to determine whether content may appear in Search. Please use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl. Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot Published IP addresses: https://openai.com/chatgpt-user.json Subscribe to the RSS feed for updates to this page. ================ SECTION anthropic | Does Anthropic crawl data from the web, and how can site owners block the crawler? | Claude Help Center | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | read 2026-11-02T09:20:38.197Z ================ All Collections Privacy and legal Does Anthropic crawl data from the web, and how can site owners block the crawler? Does Anthropic crawl data from the web, and how can site owners block the crawler? April 7, 2026 Copy for LLM As per industry standard, Anthropic uses a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users’ direction. Anthropic uses different robots to enable website owner transparency and choice. Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t. Bot Use What happens when you disable it ClaudeBot ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training. When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets. Claude-User Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent. Claude-User allows site owners to control which sites can be accessed through these user-initiated requests. Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search. Claude-SearchBot Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results. As part of our mission to build safe and reliable frontier systems and advance the field of responsible AI development, we’re sharing the principles by which we collect data as well as instructions on how to opt out of our crawling going forward: Our collection of data should be transparent. Anthropic uses the Bots described above to access web content. Our crawling should not be intrusive or disruptive. We aim for minimal disruption by being thoughtful about how quickly we crawl the same domains and respecting Crawl-delay where appropriate. Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt. Anthropic’s Bots respect anti-circumvention technologies (e.g., we will not attempt to bypass CAPTCHAs for the sites we crawl.) To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt. An example of this might be: User-agent: ClaudeBot Crawl-delay: 1 To block a Bot from your entire website, add this to the robots.txt file in your top-level directory. Please do this for every subdomain that you wish to opt out from. An example of this is: User-agent: ClaudeBot Disallow: / Opting out of being crawled by Anthropic Bots requires modifying the robots.txt file in the manner above. Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file. If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic. You can learn more about our data handling practices and commitments at our Help Center. If you have further questions, or believe that our Bots may be malfunctioning, please reach out to claudebot@anthropic.com. Please reach out from an email that includes the domain you are contacting us about, as it is otherwise difficult to verify reports. You can be notified of substantial changes to this article by clicking here and completing the form: Subscribe to updates Did this answer your question? Disappointed Reaction Neutral Reaction Smiley Reaction Related Articles Report, block, and remove content from Claude How to get support Report, block, and remove content from Claude ================ SECTION google | Google's common crawlers | Google Crawling Infrastructure  |  Crawling infrastructure  |  Google for Developers | https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers | read 2026-11-02T09:20:46.241Z ================ Home Crawling infrastructure Docs Was this helpful? Send feedback On this page Googlebot Googlebot Image Googlebot Video Googlebot News Google StoreBot List of Google's common crawlers Google's common crawlers are used to find information for building Google's search indexes, perform other product specific crawls, and for analysis. They always obey robots.txt rules when crawling automatically. The general technical properties of Google's crawlers also apply to the common crawlers. The common crawlers generally crawl from the IP ranges published in the common-crawlers.json object, and the reverse DNS mask of their hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. The following list shows the common crawlers, their user agent strings as they appear in the HTTP requests, their user agent tokens for the User-agent: line in robots.txt, and the products that are affected by crawl preferences for the crawler. Some crawlers have more than one user agent token; you need to match only one crawler token for a rule to apply. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about. Caution: The HTTP user agent string can be spoofed. Learn how to verify if a visitor is a Google crawler. Googlebot User-Agent in HTTP requests Googlebot Smartphone Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot Desktop Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36 Rarely: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Googlebot/2.1 (+http://www.google.com/bot.html) robots.txt User-agent token in robots.txt Googlebot Example robots.txt group user-agent: Googlebot allow: /archive/1Q84 disallow: /archive Affected products Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Video mode, Google News, and Discover. Googlebot Image User-Agent in HTTP requests Googlebot-Image/1.0 robots.txt User-agent token in robots.txt Googlebot-Image Googlebot Example robots.txt group user-agent: Googlebot-Image allow: /archive/1Q84 disallow: /archive/moons.jpg Affected products Crawling preferences addressed to the Googlebot-Image user agent affect Google Images, Discover, Video mode, and all features in Google Search where images, logos, and favicons are presented. Googlebot Video User-Agent in HTTP requests Googlebot-Video/1.0 robots.txt User-agent token in robots.txt Googlebot-Video Googlebot Example robots.txt group user-agent: Googlebot-Video allow: /archive/1Q84 disallow: /archive/ Affected products Crawling preferences addressed to the Googlebot-Video user agent affect video-related Google Search features and other products dependent on videos. Googlebot News User-Agent in HTTP requests Googlebot-news doesn't have a separate HTTP request user agent string. Crawling is done with various Googlebot user agent strings. robots.txt User-agent token in robots.txt Googlebot-News Googlebot Example robots.txt group user-agent: Googlebot-News allow: /archive/1Q84 disallow: /archive/ Affected products Crawling preferences addressed to the Googlebot-News user agent affect the Google News product, including news.google.com and the Google News app. Google StoreBot User-Agent in HTTP requests Desktop agent Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36 Mobile agent Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 robots.txt User-agent token in robots.txt Storebot-Google Example robots.txt group user-agent: Storebot-Google allow: /archive/1Q84 disallow: /archive/konbini Affected products Crawling preferences addressed to the Storebot-Google user agent affect all surfaces of Google Shopping (for example, the Shopping tab in Google Search and Google Shopping). Google-InspectionTool User-Agent in HTTP requests Desktop agent Mozilla/5.0 (compatible; Google-InspectionTool/1.0) Mobile agent Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0) robots.txt User-agent token in robots.txt Google-InspectionTool Googlebot Example robots.txt group user-agent: Google-InspectionTool allow: /archive/1Q84 disallow: /archive/ Affected products Crawling preferences addressed to the Google-InspectionTool user agent affect Search testing tools such as the Rich Result Test and URL inspection in Search Console. It has no effect on Google Search or other products. GoogleOther User-Agent in HTTP requests Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36 robots.txt User-agent token in robots.txt GoogleOther Example robots.txt group user-agent: GoogleOther allow: /archive/1Q84 disallow: /archive/ Affected products Crawling preferences addressed to the GoogleOther user agent don't affect any specific product. GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. GoogleOther-Image User-Agent in HTTP requests GoogleOther-Image/1.0 robots.txt User-agent token in robots.txt GoogleOther-Image GoogleOther Example robots.txt group user-agent: GoogleOther-Image allow: /archive/1Q84 disallow: /archive/moon.jpg Affected products Crawling preferences addressed to the GoogleOther-Image user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Image is the version of GoogleOther optimized for fetching publicly accessible image URLs. GoogleOther-Video User-Agent in HTTP requests GoogleOther-Video/1.0 robots.txt User-agent token in robots.txt GoogleOther-Video GoogleOther Example robots.txt group user-agent: GoogleOther-Video allow: /archive/1Q84 disallow: /archive Affected products Crawling preferences addressed to the GoogleOther-Video user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Video is the version of GoogleOther optimized for fetching publicly accessible video URLs. Google-CloudVertexBot User-Agent substring in HTTP requests Google-CloudVertexBot robots.txt User-agent token in robots.txt Google-CloudVertexBot Googlebot Example robots.txt group user-agent: Google-CloudVertexBot allow: /archive/1Q84 disallow: /archive/ Affected products Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents. It has no effect on Google Search or other products. Google-Extended User-Agent in HTTP requests Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity. robots.txt User-agent token in robots.txt Google-Extended Example robots.txt group user-agent: Google-Extended allow: /archive/1Q84 disallow: /archive/ Affected products Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI. Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. Note: Google intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: __usage_ai-training") in early 2027. A note about Chrome/W.X.Y.Z in user agents The string Chrome/W.X.Y.Z in the user agent strings in the list is a placeholder that represents the version of the Chrome browser used by that user agent: for example, 41.0.2272.96. This version number increases over time to match the latest Chromium release version used by Googlebot. If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number. Was this helpful? Send feedback Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. Last updated 2026-10-30 UTC. ================ SECTION perplexity | Perplexity Crawlers - Perplexity | https://docs.perplexity.ai/docs/resources/perplexity-crawlers | read 2026-11-02T09:20:52.747Z ================ On this page WAF Configuration Cloudflare WAF AWS WAF IP Address Sources Best Practices Resources Perplexity Crawlers Copy page We strive to improve our service every day by delivering the best search experience possible. To achieve this, we collect data using web crawlers (“robots”) and user agents that gather and index information from the internet, operating either automatically or in response to user requests. Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity. Each setting works independently, and it may take up to 24 hours for our systems to reflect changes. User Agent Description PerplexityBot PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models. To ensure your site appears in search results, we recommend allowing PerplexityBot in your site’s robots.txt file and permitting requests from our published IP ranges listed below. Example user-agent string (the version number may change): Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) Published IP addresses: https://www.perplexity.ai/perplexitybot.json Perplexity‑User Perplexity-User supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response. Perplexity-User controls which sites these user requests can access. It is not used for web crawling or to collect content for training AI foundation models. Example user-agent string (the version number may change): Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Published IP addresses: https://www.perplexity.ai/perplexity-user.json Since a user requested the fetch, this fetcher generally ignores robots.txt rules. ​ WAF Configuration If you’re using a Web Application Firewall (WAF) to protect your site, you may need to explicitly allow Perplexity’s bots to ensure they can access your content. Below are configuration guidelines for popular WAF providers. ​ Cloudflare WAF To configure Cloudflare WAF to allow Perplexity bots: 1 Navigate to security rules In your Cloudflare dashboard, go to Security → Security rules. 2 Create a new rule Select Create rule → Custom rules. 3 Configure rule conditions Each bot has its own IP ranges, so match each User-Agent against its own IP ranges. Select Edit expression and enter: (http.user_agent contains "PerplexityBot" and ip.src in {}) or (http.user_agent contains "Perplexity-User" and ip.src in {}) Replace each placeholder with the current ranges from the official endpoints listed below, separated by spaces, for example {192.0.2.0/24 198.51.100.7/32}. When the published ranges change, update them in this rule. On paid plans, you can instead store each bot’s ranges in an IP list and reference it, for example ip.src in $perplexitybot. You then update the lists rather than the rule. Free plans allow only one custom list, which isn’t enough for both bots. 4 Set rule action Set the action to Skip and select the security features these requests should bypass, such as All remaining custom rules, All managed rules, All rate limiting rules, or All Super Bot Fight Mode rules. 5 Order the rule Place this rule above any custom rules that block or challenge requests. A Skip rule only bypasses rules that run after it. Cloudflare’s Bot Fight Mode can’t be skipped with WAF custom rules. If Bot Fight Mode is enabled, it can still challenge Perplexity bots that match this rule. ​ AWS WAF For AWS WAF configuration, create IP sets and rules that match them together with the User-Agent header: 1 Create IP sets In the AWS WAF console, create an IP set for PerplexityBot and one for Perplexity-User using the IP addresses from the official endpoints listed below. 2 Create allow rules For each bot, create a rule with an AND statement that combines: An IP set match statement that uses the bot’s IP set A string match statement on the User-Agent header that contains PerplexityBot or Perplexity-User Set the rule action to Allow. 3 Add the rules to your web ACL Add these rules to your web ACL and give them lower priority numbers than your blocking rules, so AWS WAF evaluates them first. ​ IP Address Sources Always use the most current IP ranges from the official JSON endpoints. These addresses are updated regularly and should be the source of truth for your WAF configurations. PerplexityBot IP addresses: https://www.perplexity.ai/perplexitybot.json Perplexity-User IP addresses: https://www.perplexity.ai/perplexity-user.json Set up automated processes to periodically fetch and update your WAF rules with the latest IP ranges from these endpoints to ensure continuous access for Perplexity bots. ​ Best Practices When configuring WAF rules for Perplexity bots, combine both User-Agent string matching and IP address verification for enhanced security while ensuring legitimate bot traffic can access your content. Changes to WAF configurations may take some time to propagate. Monitor your logs to ensure the rules are working as expected and that legitimate Perplexity bot traffic is being allowed through. Was this page helpful? Yes No Get in Touch Previous ⌘I x discord website ================ SECTION commoncrawl | Common Crawl - CCBot | https://commoncrawl.org/ccbot | read 2026-11-02T09:21:15.044Z ================ The Data Resources Community About Search Contact Us CCBot Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone. Enabling free access to web crawl data encourages collaboration and interdisciplinary research, as organizations, academia, and non-profits can work together to address complex challenges. Collaborating using Open Data accelerates progress and helps find solutions to pressing global issues, such as climate change, public health, and social equality. By embracing Open Data, we promote an inclusive and thriving knowledge ecosystem, where the collective intelligence of the global community can lead to transformative discoveries and positive societal impact. CCBot identifies itself in its UserAgent string as: CCBot/2.0 (https://commoncrawl.org/faq/) To prevent Common Crawl from crawling your website, include the following in your robots.txt: User-agent: CCBot Disallow: / Please note that we are aware of crawlers falsely identifying themselves as CCBot. We recommend verifying UserAgent strings to ensure authenticity. ‍ CCBot is now run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.) This allows webmasters to verify whether a logged request stems from the real CCBot, for example:‍ $> host 18.97.14.84 84.14.97.18.in-addr.arpa domain name pointer 18-97-14-84.crawl.commoncrawl.org. $> host 18-97-14-84.crawl.commoncrawl.org 18-97-14-84.crawl.commoncrawl.org has address 18.97.14.84 $> dig -x 18.97.14.84 ;; ANSWER SECTION: 84.14.97.18.in-addr.arpa. 276   IN      PTR     18-97-14-84.crawl.commoncrawl.org. $> dig 18-97-14-84.crawl.commoncrawl.org A ;; ANSWER SECTION: 18-97-14-84.crawl.commoncrawl.org. 275 IN A     18.97.14.84 These IP ranges (v4 and v6) are also provided as JSON at https://index.commoncrawl.org/ccbot.json. Read about our Opt-out Registry on our blog. Please see our FAQ for further information. The Data Overview CDXJ Index URL Index Web Graphs Latest Crawl Crawl Stats Graph Stats Errata Resources Get Started Blog Examples CCBot Infra Status Opt-Out Ledger FAQ Community Research Papers Mailing List Archive Hugging Face Discord Collaborators About About Team Jobs Privacy Policy Terms of Use © 2026 Common Crawl ================ SECTION apple | About Applebot - Apple Support | https://support.apple.com/en-us/119829 | read 2026-11-02T09:21:21.800Z ================ Apple Store Mac iPad iPhone Watch Vision AirPods Home Entertainment Accessories Support 0 + About Applebot Learn about Applebot, the web crawler for Apple. The data crawled by Applebot is used to power various features, such as the search technology integrated into many user experiences in Appleʼs ecosystem including Spotlight, Siri, and Safari. Enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world in these products. The data crawled by Applebot may also be used to help train Apple foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools. Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file. Applebot crawled data may be used to provide additional context and up-to-date content when AI models are used to generate output for display in Apple products and services. For example, answering broad world knowledge questions in Siri and Search that may include links to sources and websites used to help generate the answer. Web publishers can opt out of their content being used in these broad world knowledge answers by applying the nosnippet meta tag to specific content. Even if you disallow Applebot-Extended and tag website content with the nosnippet meta tag, your website instructions may still allow Applebot to crawl your webpages. Your content will remain discoverable through Spotlight, Siri, and Safari, as well as other system-wide features on Apple devices. Identifying Applebot Traffic coming from Applebot is generally identified by using reverse DNS in the *.applebot.apple.com domain. Another way is to match the IP address with a CIDR prefix contained in the following JSON file: Applebot IP CIDRs. Reverse DNS The host command can be used to determine if an IP address is part of Applebot. These examples show the host command and its result: $ host 17-58-101-179.applebot.apple.com 17-58-101-179.applebot.apple.com has address 17.58.101.179. The host command can also be used to verify that the DNS points to the same IP address: $ host 17.58.101.179 179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com. User agents A user agent helps webmasters identify crawler traffic, so that they can get accurate access log reports of crawler activity and control access to the site via robots.txt. Applebot powers several user agents, including Search and Podcasts. Search For search web crawling and rendering, Applebot uses the following format: The user-agent string contains ”Applebot” and other information. The following is the general format: Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko)Version/Safari_version [Mobile/Mobile_version] Safari/WebKit_version (Applebot/Applebot_version; +http://www.apple.com/go/applebot) Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko)Version/Safari_version [Mobile/Mobile_version] Safari/WebKit_version (Applebot/Applebot_version; +http://www.apple.com/go/applebot) Example for desktop: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) Example for mobile: Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot) Occasionally, Applebot will update the browser version that it advertises while remaining in the general format above. Special crawlers Apple online properties iTMS traffic may come from applebot.apple.com hosts, and will be identified by the following user agent: User-Agent: iTMS The iTMS user agent does not follow robots.txt, as it is not a general search crawler. The iTMS user agent only crawls URLs associated with registered content on Apple Podcasts. Customizing robot.txt rules Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot. In this example, Applebot doesn't try to crawl documents that are under /private/ or /not-allowed/: User-agent: Applebot Allow: / Disallow: /private/ User-agent: * Disallow: /not-allowed/ If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions. Applebot does not follow crawl-delay. Applebot is engineered for efficiency and will adjust to minimize the impact on site owners. For example, to avoid overloading site servers, Applebot’s crawl rate adjusts automatically when a site slows down or returns errors. Apple also caches crawled content to reduce unnecessary crawling. Identifying content that doesn't need to be crawled lowers infrastructure costs for site owners and makes the internet more efficient as a whole. Rendering and robot rules Applebot may render the content of your website within a browser. If javascript, CSS, and other resources are blocked via robots.txt, it may not be able to render the content properly. This includes XHR, JS, and CSS that the page might require. In order for Applebot to index the best content for the page, make sure that everything needed for a user to render the page is available to Applebot. Alternatively, make sure that the website renders cleanly, even if all of the resources are not available. This is often referred to as graceful degradation. Customizing indexing rules for Applebot Applebot supports robots meta tags in HTML documents. To specify robots rules in meta tags, put the tags in the section of the document: ... ... Applebot also supports the following directives: noindex: Applebot won't index this page, and it won't appear in Spotlight or Siri Suggestions. nosnippet: Applebot won't generate a description or web answer for the page. Any suggestions to visit this URL will only include the page's title. Apple will not use data tagged nosnippet as additional context and up-to-date content when AI models are used to generate output for display in Apple products and services. nofollow: Applebot won't follow any links on the page. none: Applebot won't index, snippet, or follow links on the page, as described above. all: Applebot provides the document for suggestions and snippets the contents so that a short description of the page can appear next to a representative image. Applebot may follow links on the page to provide more suggestions. To put multiple directives in a single meta tag, use a comma-separated list or multiple meta tags. Example: Using the X-Robots-Tag HTTP header Applebot also supports indexing directives delivered via the X-Robots-Tag HTTP response header. This is useful for non-HTML resources (such as PDFs or images) where meta tags can’t be used, or when you want to apply directives at the server level.To apply a directive to Applebot: X-Robots-Tag: applebot: nosnippet Marking paywalled content Applebot supports the schema.org isAccessibleForFree property to identify pages that are behind a paywall, metered access, or subscription. Add the property to your page-level structured data in JSON-LD: Pages marked isAccessibleForFree: false are eligible to appear in search results, but Applebot will not use that content as additional context when AI models are used to generate output for display in Apple products and services.This signal applies at the page level. Section-level markup using hasPart is not supported. To opt out of having your content used to train Apple’s foundation models, use Applebot-Extended described in the next section. Applebot-Extended and controlling data usage In addition to following all robots.txt rules and directives, Apple has a secondary user agent, Applebot-Extended, that gives web publishers additional controls over how their website content can be used by Apple. With Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s general purpose foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools. You can add a rule in robots.txt to disallow Applebot-Extended, as follows: User-agent: Applebot-Extended Disallow: /private/ Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results. Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent. Allowing Applebot-Extended will help improve the capabilities and quality of Apple’s generative AI models over time. About search rankings Apple Search may take the following factors into account when ranking web search results: Aggregated user engagement with search results Relevancy and matching of search terms to webpage topics and content Number and quality of links from other pages on the web User location based signals (approximate data) Webpage design characteristics Search results may use the factors above with no (pre-determined) importance of ranking. Users of Search are subject to the privacy policy in Siri Suggestions, Search & Privacy. Site rules for Applebot-Extended are not considered in ranking for Search. Contact us If you have questions or concerns, please contact us at applebot@apple.com. Information about products not manufactured by Apple, or independent websites not controlled or tested by Apple, is provided without recommendation or endorsement. Apple assumes no responsibility with regard to the selection, performance, or use of third-party websites or products. Apple makes no representations regarding third-party website accuracy or reliability. Contact the vendor for additional information. Published Date: September 04, 2026 Helpful? Yes No Apple Footer  Apple Support About Applebot United States Copyright © 2026 Apple Inc. All rights reserved. Privacy Policy Terms of Use Sales and Refunds Site Map ================ SECTION meta | Meta Web Crawlers | Developer Documentation | https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers | read 2026-11-02T09:21:30.675Z ================ Docs Overview Meta Web Crawlers Was this helpful? Meta Web Crawlers Updated: May 21, 2026 Copy for LLM View as Markdown Meta uses web crawlers (software which fetches content from websites or web apps) for a number of different purposes. This page lists the User Agent (UA) strings that identify Meta’s most common web crawlers and what each of those crawlers are used for. We make it easy for site managers and content owners to indicate their preferences by using industry-standard practices like robots.txt rather than non-standard formats like NoAI tags. This page provides guidance on how to configure your robots.txt file so that our crawlers interact properly with your site. FacebookExternalHit The primary purpose of FacebookExternalHit is to crawl the content of an app or website that was shared on one of Meta’s family of apps, such as Facebook, Instagram, or Messenger. The link might have been shared by copying and pasting or by using the Facebook social plugin. This crawler gathers, caches, and displays information about the app or website such as its title, description, and thumbnail image. The specific UA string that you will see in your log files will be similar to one of the following: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) facebookexternalhit/1.1 Note that the FacebookExternalHit crawler might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content. Crawler Requirements Your server must use gzip and deflate encodings. Any Open Graph properties need to be listed before the first 1 MB of your website or app, or it will be cutoff. Ensure that the content can be crawled by the crawler within a few seconds or Facebook will be unable to display the content. Your app or website should either generate and return a response with all required properties according to the bytes specified in the Range header of the crawler request or it should ignore the Range header altogether. Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler. Troubleshooting If your app or website content is not available at the time of crawling, you can force a crawl once it becomes available either by passing the URL through the Sharing Debugger tool or by using the Sharing API. You can simulate a crawler request with the following code: curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL" Meta-WebIndexer The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users. In doing so, Meta analyzes online content to enhance the relevance and accuracy of Meta AI. Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI’s responses. The specific UA string that you will see in your log files will be similar to one of the following: meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers) meta-webindexer/1.1 Meta-ExternalAds The Meta-ExternalAds crawler crawls the web for use cases such as improving advertising and other business-related products and services. The specific UA string that you will see in your log files will be similar to one of the following: meta-externalads/1.1 (+/documentation/sharing/webmasters/web-crawlers) meta-externalads/1.1 Meta-ExternalAgent The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. The specific UA string that you will see in your log files will be similar to one of the following: meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers) meta-externalagent/1.1 Meta-ExternalFetcher The Meta-ExternalFetcher crawler fetches individual links at a user’s request and supports product functions such as evaluating and improving agentic AI capabilities—including helping AI navigate websites to complete tasks for users. Accordingly, this crawler may bypass robots.txt rules. The specific UA string that you will see in your log files will be similar to one of the following: meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers) meta-externalfetcher/1.1 The robots.txt file By configuring the robots.txt file on your website, you can specify to the Meta web crawlers how you would prefer them to interact with your site. In order to block these crawlers, add a disallow for the relevant crawler to robots.txt. User-agent: meta-externalagent Allow: / # Allow everything Disallow: /private/ # Disallow a specific directory User-agent: meta-externalads Allow: / # Allow everything Disallow: /private/ # Disallow a specific directory The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user. Also, the FacebookExternalHit crawler might bypass robots.txt when performing security or integrity checks. Please allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache the contents of robots.txt for up to 24 hours. Contact us If you have questions or concerns, please contact us at webmasters@meta.com (Meta Web Masters). Did you find this page helpful? ================ SECTION amazon | Amazon Web Search crawlers | Amazonbot | https://developer.amazon.com/docs/amazonbot/amazonbot.html | read 2026-11-02T09:21:38.731Z ================ Alexa Amazon Appstore Ring AWS Documentation Alexa Amazon Appstore Ring AWS Documentation Console Sign in Fire OS Developer Docs Home Develop Test Publish Monetize Engage users Device specifications Resources Amazon Web Search crawlers This page describes how webmasters can control Amazonbot, Amzn-SearchBot, and Amzn-User interactions with their site. Each user agent setting is independent of the others, and may take ~24 hours for our systems to reflect changes. Amazonbot Amazonbot is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models. Example User Agent String: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36 Published IP Addresses: https://developer.amazon.com/amazonbot/ip-addresses/ Amzn-SearchBot Amzn-SearchBot is used to improve search experiences in Amazon products and services. By permitting Amzn-SearchBot access to your website, your content is eligible to appear in search experiences such as Alexa. If robots.txt files don’t mention Amzn-SearchBot but allow other search bots, Amzn-SearchBot will crawl in accordance with the robots.txt directives given to other search bots. Amzn-SearchBot does not crawl content for generative AI model training. Example User Agent String: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36 Published IP Addresses: https://developer.amazon.com/amazonbot/searchbot-ip-addresses/ Amzn-User Amzn-User supports user actions, such as responding to Alexa queries that require up-to-date information. For example, when a customer asks a question, Amzn-User may fetch live information from the web to provide accurate answers on the user’s behalf. Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives.Amzn-User does not crawl content for generative AI model training. Example User Agent String: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36 Published IP Addresses: https://developer.amazon.com/amazonbot/live-ip-addresses/ Our Approach to Robots.txt Automated crawling from these listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives. They will fetch host-level robots.txt files or use a cached copy from the last 30 days. When a file can’t be fetched, they will behave as if it does not exist. These user agents attempt to read robots.txt files at the host level (for example example.com), so they look for robots.txt at example.com/robots.txt. If a domain has multiple hosts, then they will honor robots rules exposed under each host. For example, if there is also a site.example.com host, they will look for robots.txt at site.example.com/robots.txt. When these user agents access web pages they respect the link-level rel=nofollow directive, and page level robots meta tags of noarchive (do not use the page for model training), noindex (do not index the page) and none (do not index the page). They do not support the crawl-delay directive. Contact Us If you are a content owner or publisher and have questions, please contact us at amazonbot@amazon.com. Always include any relevant domain names in your message. Explore more If you allow Amazonbot on your robots.txt, you may be eligible for benefits with Amazon Content Partners. Amazon Content Partners gives independent content creators a +1% affiliate commission boost on eligible sales, free hosting credits, and AI traffic management tools — all in one program. Learn more at contentpartners.amazon.com. Last updated: Oct 26, 2026 Back to Top Follow us: Resources Appstore developer blog Technical documentation Amazon Developer Huddle Supported devices Amazon Fire TV Fire tablets Other services & APIs Small Business Accelerator Program Developer Promotions Console Alexa for video publishers Login with Amazon Frustration-Free Setup Amazon Incentives API Amazon Merch on Demand Amazon Music Fire TV Partners Support Appstore Developer Community FAQs Contact us Legal Terms & agreements Privacy Notice © 2010-2026, Amazon.com, Inc. or its affiliates. All Rights Reserved. English Terms Amazon Developer Blog Contact Us Welcome! Ask the DevAssistant