Every WordPress site owner already knows Googlebot. You have seen it in your logs, you have a mental model of how often it visits, and you have a plugin or two that tells you when it last crawled a given page. What almost nobody has a mental model for yet is the newer set of visitors showing up in those same logs: GPTBot, ClaudeBot, PerplexityBot, and a handful of others that behave nothing like a search engine crawler and are not indexing your site for a results page. They are reading it for something else entirely, and most site owners have no idea it is happening.
This is not a hypothetical shift. Millions of people now ask ChatGPT, Claude, and Perplexity questions that used to go to a search box, and increasingly those assistants answer by reading and citing real web pages, sometimes fetched live, sometimes pulled from a training snapshot collected weeks or months earlier. Every one of those interactions traces back to a crawler with its own name, its own purpose, and its own user agent string, hitting your server exactly the way Googlebot does. The difference is that almost no WordPress owner has gone looking for it.
That gap matters for a practical reason, not a philosophical one. If AI assistants are already fetching your content to answer questions, you are already part of the AI search ecosystem whether you planned for it or not. You can either understand what is happening in your own logs and make deliberate decisions about it, or you can leave it to chance: unaware of which bots are hitting 404s, unaware of which ones your security plugin quietly blocked six months ago, unaware of whether the content you are proudest of is being read at all. None of that requires guesswork. It requires knowing what to look for.
This guide is a field manual, not a listicle. The goal is to leave you able to open your own server logs, or a tool like Rankwyn's AI traffic insights, and immediately answer three questions: who is visiting, whether that activity looks healthy, and what you want to do about each bot specifically. We will cover why AI crawler traffic is a genuinely different category from the search bots you already know, walk through a full identification table for the crawlers that matter today, show you exactly how to check your own logs, describe what healthy versus broken AI crawler activity looks like, and give you a concrete allow, throttle, or block framework with copy-paste robots.txt examples for each stance. Along the way we will flag the mistakes that trip up most site owners, including the ones your security plugin might already be making without telling you.
Why AI crawler traffic is not the same thing as search bot traffic
It is tempting to lump every non-human visitor into one bucket labeled bots, but AI crawlers and search indexing bots exist to do fundamentally different jobs, and understanding that difference changes how you should think about them.
A traditional search crawler like Googlebot or Bingbot exists to build and maintain an index. It visits your pages, extracts content and signals, stores a processed version, and later matches that stored version against a searcher's query. The crawl and the moment someone sees your page in results are separated, often by days or weeks. Your relationship with that bot is mediated entirely through rankings: crawl frequency, indexing status, and position in results are the feedback loop you have spent years learning to read.
AI crawlers split into two categories that behave very differently from that model and from each other. The first category is bulk or periodic crawling for model training or index building, the closest analog to what Googlebot does. GPTBot from OpenAI and ClaudeBot from Anthropic largely fall here: they crawl in sweeps, sometimes far apart, to gather text that either trains a future model version or feeds a retrieval index the assistant can search later. This crawling is not tied to any single user's question in the moment it happens.
The second category is live retrieval, sometimes called retrieval-augmented generation or RAG, where a bot fetches a specific page in real time because a user just asked a question that requires current information. ChatGPT-User and Claude-User exist for exactly this: when someone asks ChatGPT or Claude to look something up right now, a browsing-mode fetch goes out, reads the page, and the answer that comes back to the user is built partly from what that fetch found. PerplexityBot operates closer to a hybrid, since Perplexity's entire product is built around answering queries with live-feeling citations, so its crawling supports both an index and near-real-time answer construction.
This distinction matters practically for three reasons. First, the timing is different: a bulk training crawl might visit once a month, while a live retrieval fetch is directly triggered by a user's question and can happen at any hour with no pattern at all. Second, the stakes are different: blocking a live retrieval bot means a real user asking a real question right now does not get to see your page as a source, which is a more immediate loss than being excluded from a future model's training data. Third, the signal is different: if you see ChatGPT-User hitting a specific page, you have direct evidence that someone, somewhere, asked a question that led an AI assistant to your content in that exact moment. That is a citation event you can trace, in a way that a Googlebot crawl never gives you.
Search indexing and AI grounding are increasingly separate systems, even from the same company
Google runs Googlebot for regular search indexing and a separate crawler, Google-Extended, that controls whether your content can be used for Gemini and AI Overviews training and grounding. Blocking one does not block the other, and conflating them is one of the most common mistakes site owners make when editing robots.txt.
The AI crawler identification table: who is who, and what they actually do
Before you can decide anything, you need to know what you are looking at. Below is a working reference for the AI crawlers most likely to show up in a WordPress access log today. This list will grow and some of these bots will change behavior over time, so treat it as a snapshot rather than a permanent fact, and recheck it periodically, since the AI crawler landscape is evolving faster than the search crawler landscape ever did.
| Bot name | Operated by | Purpose |
|---|---|---|
| GPTBot | OpenAI | Bulk crawling for model training data collection, used to improve future ChatGPT and GPT model versions. Not tied to a specific user query. |
| ChatGPT-User | OpenAI | Real-time browsing fetch, triggered when a user asks ChatGPT to look something up live. Directly tied to a specific in-the-moment user question. |
| ClaudeBot | Anthropic | Crawls the web to build and refresh the index and training material Claude draws on. Operates in sweeps rather than per-query fetches. |
| Claude-User | Anthropic | Real-time fetch when a user's conversation with Claude requires reading a specific live page, similar in function to ChatGPT-User. |
| PerplexityBot | Perplexity | Crawls and indexes content for Perplexity's answer engine, which is built around citing sources directly in its responses. |
| Google-Extended | A separate control from regular Googlebot indexing. Governs whether your content can be used for Gemini and AI Overviews training and grounding. | |
| Bingbot | Microsoft | Primarily the standard Bing search indexing crawler, but its crawl data also powers Copilot's web grounding for cited answers. |
| CCBot | Common Crawl | A general-purpose web archive crawler, not affiliated with any single AI assistant. Many AI labs, including some not listed here, train on the Common Crawl dataset rather than crawling the web themselves. |
A few things are worth pulling out of that table explicitly, because they change how you should treat each row. GPTBot and ChatGPT-User are both operated by OpenAI, but they are not the same bot with two names, they are two functionally distinct crawlers with two distinct user agent strings, and blocking one does not block the other. The same split exists between ClaudeBot and Claude-User. If your goal is to stop your content from training a future model but you still want it available when a user asks Claude or ChatGPT a live question about your page, you need to treat these as separate decisions, which is exactly what the framework later in this guide is built around.
CCBot deserves a special note because it is the one entry on this list that is not directly controlled by a company running a consumer AI product. Common Crawl is a nonprofit that publishes a enormous, freely available archive of crawled web pages, and a wide range of AI labs, both the big names and smaller research groups you have never heard of, use that archive as raw training material. Blocking CCBot does not target any one assistant. It is closer to opting out of a shared public dataset that many different models are trained from, which is a different kind of decision than blocking GPTBot specifically.
Google-Extended is worth a second look too, because it trips people up constantly. If you already block Googlebot from a section of your site through your existing robots.txt rules, that has zero effect on whether Google-Extended can use that same content for Gemini or AI Overviews. They are evaluated independently. If you want a section of your site out of search results entirely, you use noindex or disallow Googlebot. If you separately want it excluded from AI training and grounding, you need a distinct Google-Extended rule.
One more nuance: these user agent strings are not fixed forever. Companies do occasionally add new crawler names as products evolve, split existing bots into more specific ones, or retire old identifiers. Treat any list like this one as a living reference, and when in doubt, check the crawler's officially published documentation rather than relying on secondhand summaries, since misidentifying a bot in your robots.txt rules can silently let the wrong traffic through or block the traffic you actually wanted.
How to actually check your server logs for AI bot activity
Knowing the bot names is only useful once you can find them in your own traffic. This is the part most WordPress owners skip, not because it is technically difficult, but because nobody has told them exactly where to look. Here is the process in order of how most people should actually do it.
Start with whatever you already have access to, before installing anything new. Most hosting control panels, whether that is cPanel, Plesk, or a managed WordPress host's custom dashboard, expose raw access logs somewhere under a label like Raw Access Logs, Metrics, or Visitors. These logs record every single HTTP request that hits your server, including the user agent string sent by the client making the request. A human browser sends a user agent describing the browser and operating system. A crawler sends a user agent describing itself, and reputable AI crawlers identify themselves plainly, which is exactly what makes this approach work.
If your host gives you downloadable or searchable raw logs, the fastest path is a simple text search for the bot names from the table above. You are looking for the user agent field of each log line to contain a string like GPTBot, ClaudeBot, PerplexityBot, or CCBot. A single log line typically bundles the requesting IP address, a timestamp, the specific URL requested, the response status code, and the user agent string together on one line, and searching for the bot name will surface every request that bot made.
203.0.113.42 - - [12/Aug/2026:14:22:01 +0000] "GET /blog/wordpress-seo-checklist/ HTTP/1.1" 200 18432 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot"Reading a line like that: the IP address and timestamp tell you when and roughly where the request came from, the quoted GET line tells you which exact URL on your site was requested and confirms the request succeeded with a 200 status code, and the long string at the end containing GPTBot is the user agent, which is the part that identifies the crawler. If you see a 404 instead of a 200 in that position, the bot requested a page that does not exist on your site, which is worth investigating rather than ignoring, since it usually means an outdated link or a broken internal reference.
If digging through raw text logs is not something you want to do regularly, or your host does not expose them in an easily searchable form, a security or logging plugin is the next practical option. Many WordPress security plugins already log bot and crawler visits as part of their firewall or activity logging features, and some let you filter or search by user agent string directly inside the WordPress admin, which avoids needing shell or file-manager access to your server at all. The tradeoff is that general-purpose security plugins were not built specifically to recognize AI crawlers, so their bot lists can be incomplete or lump unfamiliar AI bots into a generic unknown-bot category rather than naming them.
This is the specific gap Rankwyn's AI traffic insights feature is built to close. Instead of digging through raw access logs or hoping a general security plugin happens to recognize the right user agent strings, it identifies known AI assistant and crawler hits by name directly inside your WordPress dashboard, on a dedicated insights screen built for exactly this purpose, separate from your regular analytics. It runs entirely on your own server, with no data sent to an external service and no API call required, so the same privacy boundary you already expect from your own site applies to this data too. For a site owner who wants an ongoing answer to who is visiting rather than a one-time manual log search, this is the more sustainable path, and it is the approach we would point most readers toward after they have done the manual log check once to understand what they are looking at.
Whichever method you use, do the check more than once. A single log snapshot tells you what happened in that window, not whether AI crawler activity on your site is increasing, decreasing, or absent entirely. The real value comes from comparing snapshots over time, particularly after you publish new content, add schema markup, or generate an llms.txt file, since that comparison is what tells you whether changes you make are actually having an effect on how AI systems engage with your site.
What healthy AI crawler activity looks like versus a site with a problem
Once you can see AI crawler activity in your logs, the next skill is knowing whether what you are seeing is normal or a sign something is broken. There is no single universal benchmark here, since crawl frequency varies by bot, by how large and how frequently updated your site is, and by factors outside your control, but there are clear qualitative patterns that separate a healthy pattern from a problem.
A healthy pattern on a reasonably well-optimized site typically shows a handful of recognizable characteristics. You see repeat visits from multiple distinct AI crawlers over time, not just one, which suggests your site is discoverable and not accidentally excluded from one ecosystem while being open to another. The pages being crawled skew toward your genuinely substantive content, meaning the crawler is finding and prioritizing your best pages rather than getting stuck on thin or low-value ones. Response codes on those requests are overwhelmingly 200, meaning the bot is successfully retrieving the pages it asks for. And when you compare activity before and after you publish something new or improve an existing page's structure and schema, you tend to see a corresponding uptick in that specific page's crawl activity within a reasonable window, which tells you the crawlers are actually re-checking your site rather than working from a stale snapshot indefinitely.
A site with a problem shows the opposite patterns, and they generally fall into three recognizable failure modes. The first is a crawler hitting a wall of 404s, repeatedly requesting URLs that no longer exist, which usually traces back to old internal links, an outdated sitemap, or a site restructuring that was not properly redirected. This wastes the crawler's attention on pages that cannot be read and signals a maintenance gap that is also hurting your regular search crawling. The second is a crawler being blocked entirely, where you see zero hits from a given bot in your logs despite the bot being active and crawling elsewhere on the web, which is a strong signal that something in your robots.txt, security plugin, firewall, or CDN is silently rejecting it, often without your knowledge. The third and subtler failure mode is a crawler simply ignoring your site, showing up rarely or not at all even though nothing is technically blocking it, which usually points to a discoverability problem rather than an access problem: thin content, a sitemap that is not being kept current, weak internal linking, or a site that has not given these systems a reason to prioritize it over competing pages covering the same topic.
Distinguishing between these three failure modes matters because the fix for each is completely different. Fixing 404s is a content and redirect maintenance task. Fixing a silent block is a configuration task, usually inside your security plugin, firewall, or CDN settings. Fixing a discoverability problem is a content and structure task, closer to traditional SEO and to the schema and llms.txt work covered in Rankwyn's broader generative engine optimization guide. Treating all three the same way, or worse, not noticing which one you actually have, is how site owners end up making changes that do not move the needle.
That number is a useful reality check on why this all matters beyond curiosity. When an AI system answers a query directly instead of sending the searcher to a list of links, the click-through rate to any individual site drops substantially, in some studies by more than half. That makes being one of the sources an AI system actually cites, rather than simply being crawled and ignored, a meaningfully bigger share of your total visibility than it used to be. Healthy crawler activity that never translates into citations is still a gap worth closing.
The allow, throttle, or block decision framework
Once you know who is visiting and whether it looks healthy, the actual decision comes down to a small number of stances you can take toward each category of bot, applied through robots.txt. There is no universally correct answer here, and reasonable site owners land in different places depending on their business model, but the framework below gives you a structured way to decide rather than guessing.
Start by splitting the crawlers from the identification table into two functional groups, because they warrant genuinely different treatment. The first group is search-and-cite bots: GPTBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, and Google-Extended. These are the crawlers most likely to result in your content being cited, quoted, or linked back to inside an AI-generated answer that a real person sees. Blocking these has a direct, visible cost: you lose the chance to be a cited source in a fast-growing share of how people find information. The second group is training-only bots, best represented by CCBot, which feed general-purpose datasets used by a wide range of models, without a clear, direct link back to any specific citation event you would ever see or benefit from.
With that split in mind, three stances cover most site owners. The first is allow everything, appropriate for a site whose growth strategy leans on being discovered and cited as widely as possible, such as a publisher, an informational or educational site, or a company that treats AI citations as a marketing channel similar to organic search. The second is allow citation bots but block training-only bots, appropriate for a site that wants the visibility benefit of being cited in AI answers today but is not interested in having its content folded anonymously into a general training dataset with no attribution path back. The third is block all AI bots, appropriate for a site with content that is commercially sensitive, paywalled, or simply not something the owner wants exposed to any AI system regardless of near-term visibility tradeoffs.
Here is the robots.txt block for the first stance, allowing everything:
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /Here is the second stance, allowing citation bots while blocking CCBot specifically:
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Disallow: /And here is the third stance, blocking all AI bots outright:
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /None of these three stances is permanent once chosen. It is reasonable to start conservatively, for example allowing citation bots while blocking training-only crawlers, and revisit the decision after a few months of watching your actual crawl data and any downstream referral traffic you can attribute to AI assistants. What matters more than picking the theoretically optimal stance on day one is making the decision deliberately, writing it into your robots.txt file explicitly, and revisiting it periodically as both the AI crawler landscape and your own site's content strategy evolve, rather than leaving the default state, which is usually whatever a security plugin or hosting provider decided for you without your input.
Two practical notes on applying any of these. First, robots.txt rules are matched by exact user agent name, so a typo in the bot name silently does nothing, it does not partially block or throw an error, it just fails to match and the rule is ignored. Second, you can apply these rules at a path level rather than site-wide, disallowing a specific directory like a members area or a paywalled section while leaving the rest of the site open, which is often the more precise choice than an all-or-nothing site-wide rule.
Robots.txt is a request, not a lock
Being straight about a real limitation here: robots.txt is a voluntary standard. Reputable operators like OpenAI, Anthropic, and Perplexity publish their crawlers' names specifically so they can be identified and honored, and they do generally respect Disallow rules for their named bots. But robots.txt has no technical enforcement mechanism, and a bot that does not want to identify itself, or one operated by someone with no interest in following the standard, can simply ignore your file entirely or spoof a different user agent. If a page truly cannot be exposed to any crawler under any circumstances, robots.txt alone is not sufficient, you need server-level access control, authentication, or a firewall rule that actually blocks the request rather than politely asking the requester not to make it.
What typically changes after adding an llms.txt file and improving schema
It helps to walk through what this looks like in practice, so here is an illustrative example, not a real named company, built from the pattern that shows up repeatedly across sites that make these changes deliberately and then check their logs afterward.
Picture a mid-sized WordPress site running a WooCommerce store alongside a blog of buying guides and comparison articles, the kind of site that gets a reasonable amount of organic search traffic but has never once looked at its logs for AI crawler activity. Before making any changes, a manual log check turns up a familiar, uneven picture: GPTBot shows up sporadically, ClaudeBot barely at all, and PerplexityBot not once in the sample window. There is no llms.txt file. Product pages have basic Product schema from WooCommerce, but the buying guides and comparison content, the pages best suited to being cited in an AI answer, have no structured markup describing their content at all.
The site owner makes two changes over the course of a month. First, they generate and publish an llms.txt file, giving AI systems a clean, structured entry point that summarizes what the site covers and links to its most important content, rather than making a crawler infer that structure from navigation menus and internal links. Second, they add FAQ and HowTo schema to the comparison and buying-guide content specifically, the pages most likely to directly answer the kind of question someone would type into ChatGPT or Perplexity.
In this kind of scenario, the pattern that typically shows up over the following weeks is not a single dramatic spike, it is a gradual broadening. PerplexityBot, previously absent, starts appearing, often because Perplexity's crawling is closely tied to actively answering live queries and a clean llms.txt file combined with structured FAQ content gives it a much easier target to cite from. ClaudeBot's visit frequency increases modestly rather than dramatically, consistent with periodic rather than per-query crawling. The comparison and buying-guide pages specifically, the ones that received the new schema, account for a disproportionate share of the increase relative to the rest of the site, which lines up with the well-documented finding that comparison-style content performs particularly well as AI citation source material.
That statistic is exactly why the illustrative site owner above targeted comparison and buying-guide pages first rather than spreading the schema work evenly across the whole site. Comparison content answers a specific, well-defined question in a format that is unusually easy for a model to lift and cite cleanly, which is a big part of why it shows up so disproportionately in AI citation research relative to how much of the average site's content is actually comparison-format.
The honest caveat to this kind of example is that it describes a typical pattern, not a guarantee. Some sites make identical changes and see a slower or smaller shift, particularly if the underlying content was thin to begin with or the site has deeper technical issues, like the 404 and blocking problems covered earlier, that structured data alone cannot fix. Adding an llms.txt file and schema markup removes friction for AI systems that already want to read your content well, it does not manufacture demand for content that was not worth citing in the first place. For the fuller picture of how llms.txt generation, schema, and the rest of generative engine optimization fit together, see Rankwyn's broader GEO guide, which covers the optimization side of this in more depth than this identification-and-decision-focused piece does. The AI search and GEO feature on Rankwyn is what generates the llms.txt file and structured schema referenced in this example.
It is also worth being explicit about what this example is not claiming. It is not evidence that llms.txt or schema markup directly causes a ranking boost in traditional search, and it is not a promise of a specific percentage increase in AI crawler visits within a specific number of days. What it illustrates is a directional pattern that shows up repeatedly: giving AI systems a cleaner, more structured way to understand what your site covers tends to correlate with those systems engaging with your content more broadly and more often, particularly for the content formats, like comparisons and buying guides, that AI answers already lean on heavily. Whether that translates into a citation you can point to, a referral visit you can trace, or neither, still depends on the underlying quality and specificity of the content itself.
Common mistakes site owners make with AI crawler traffic
A short list of specific, recurring mistakes is more useful here than general advice, because each of these has a concrete fix once you know to look for it.
- Security plugins blocking GPTBot as a bad bot by default. Many WordPress firewall and security plugins ship with bot-blocking rules that predate the current wave of AI crawlers, or that classify any unfamiliar automated user agent as suspicious by default. It is common for a site owner to discover, only after checking logs closely, that their security plugin has been silently rejecting GPTBot or ClaudeBot requests for months with no notification, meaning a robots.txt rule that says Allow is being overridden at a layer the owner never thought to check.
- CDN or firewall rules silently dropping AI crawler requests. Separately from your WordPress-level plugins, a CDN or a hosting-level web application firewall can have its own bot management rules, sometimes bundled in as part of a general bot-fighting or scraping-protection feature, that block AI crawlers before the request ever reaches WordPress at all. These rules often live in a dashboard the site owner rarely opens, and robots.txt has no influence over them whatsoever, since the request is being rejected before it is even parsed as HTTP traffic your server processes normally.
- Confusing real AI crawler traffic with spoofed traffic claiming to be an AI crawler. A user agent string is simply text the requesting client chooses to send, and nothing stops a scraper with no relationship to OpenAI, Anthropic, or Perplexity from sending a request with GPTBot in its user agent field to see if that gets it past a naive block list. User agent string alone is not proof of identity. For anything security-sensitive, such as deciding whether to grant a request special access or exclude it from rate limiting, verify the requester's IP address against the ranges the crawler operator officially publishes, rather than trusting the user agent string in isolation.
- Treating all AI bot traffic as one undifferentiated category. As covered earlier, GPTBot and ChatGPT-User are not the same decision, and neither are ClaudeBot and Claude-User, or Googlebot and Google-Extended. A blanket rule applied without noticing these splits usually produces an outcome the site owner did not actually intend, like accidentally blocking live user-triggered citations while believing they only blocked bulk training crawling.
- Never checking again after the first look. AI crawler behavior, naming conventions, and even which bots exist at all are changing faster than the search crawler landscape ever did. A robots.txt file or a mental model of who is crawling that was accurate a year ago may already be missing a newly introduced bot or misjudging a renamed one.
The throughline across all five of these is that most AI crawler problems are invisible until you specifically go looking, because none of them show up as an error message, a support ticket, or a drop in a metric you were already tracking. They show up only as an absence in a log you were not checking. That is the entire reason a dedicated view matters more here than it does for search bot traffic, where tools like Search Console already surface problems proactively. Nothing does that for AI crawlers by default, which is the gap that manual log checks or a feature like Rankwyn's AI traffic insights is meant to close.
If you are weighing whether this level of visibility is worth adding to your existing WordPress SEO setup, it is worth looking at what is included across Rankwyn's plans, since AI traffic insights and the llms.txt and schema tooling referenced throughout this guide are part of the same product rather than a separate add-on you would need to piece together from multiple plugins.