robots.txt is a plain text file that sits at the root of your domain and tells crawlers which parts of your site you would rather they not request. It is one of the oldest conventions on the web, it is trivial to read, and it is also one of the most commonly misunderstood files in WordPress SEO. Get it wrong and you can accidentally block Google from your entire site with a single line. Get it right and it becomes a quiet, reliable piece of infrastructure you rarely think about again.
What robots.txt actually does, and does not do
The single most important thing to understand about robots.txt is that it is a request, not an enforcement mechanism. When a crawler visits your site, it can check robots.txt first and choose to honor whatever rules it finds there. Nothing about the file itself prevents a server from returning a page to a crawler that ignores it. The file works entirely on cooperation.
Well-behaved crawlers cooperate. Googlebot, Bingbot, and the major AI crawlers documented in our AI crawler traffic guide all read robots.txt before crawling and respect Disallow rules for their own user agent. That is why the file works as well as it does in practice: the crawlers that matter for search visibility and AI citations are the ones most likely to follow the rules.
robots.txt is not a security tool
A malicious scraper, a content thief, or a bot with no reputation to protect can simply ignore robots.txt entirely, and many do. Never rely on robots.txt to keep sensitive content private or to stop bad actors from accessing a URL. If something genuinely needs to stay unreachable, protect it with authentication or server-level access rules, not a text file that only asks nicely.
The other distinction worth fixing in your head early is the difference between blocking crawling and blocking indexing. Disallowing a URL in robots.txt tells a crawler not to request that page's content. It does not remove the URL from search results if that URL is already known through other means, such as an inbound link. A blocked URL can still appear in Google's index as a bare link with no title or description, because Google knows the URL exists even though it was never allowed to read what is on it. If your goal is to keep a page out of search results entirely, the correct tool is a noindex directive in the page's own meta tags, not a Disallow rule. We cover this distinction in more detail, along with the broader set of reasons pages fail to appear in results, in why Google is not indexing your WordPress site.
robots.txt also does not directly improve your rankings. It has no scoring effect, no relevance signal, and no weight in how Google or Bing evaluates the quality of your content. What it does is manage crawl efficiency: keeping crawlers away from low-value URLs like internal search results, cart pages, or admin paths so they spend more of their limited crawl budget on the pages you actually want indexed. On a small site this rarely matters. On a large site with thousands of URLs, it can meaningfully change which pages get crawled and how often.
WordPress's default robots.txt behavior
Every WordPress installation generates a robots.txt file automatically, even if you have never created one yourself. This is a virtual file, meaning it is produced by WordPress core at request time rather than existing as an actual file in your hosting root. If you visit yoursite.com/robots.txt on a fresh WordPress install, you will see output, but there is no robots.txt sitting in your server's file directory unless you or a plugin put one there.
This default output is minimal. On a site that has not enabled search engine visibility settings, WordPress typically disallows crawling of the entire site. On a normal, publicly visible site, the default virtual file usually allows most crawling and disallows a small number of core paths, most commonly the wp-admin directory, while explicitly allowing the admin-ajax.php file within it since some front-end functionality depends on that specific request going through.
The moment you want to add your own rules, such as blocking a specific query parameter pattern or adding a Sitemap directive, WordPress needs an actual physical file to take over from the virtual one. Creating a real robots.txt file in your site's root directory, whether by uploading one directly or through an SEO plugin's editor, automatically overrides the virtual version. WordPress checks for a physical file first and only falls back to generating one on the fly if none exists.
The safest way to customize robots.txt on WordPress is through your SEO plugin's built-in editor rather than manually uploading a file over FTP. A plugin-based editor keeps the file in sync with plugin-generated sitemap paths, warns you before you save an obviously broken rule, and avoids the scenario where a manually uploaded file gets silently overwritten by a plugin update or another tool that assumes it owns that file. If you do edit the file directly, keep a copy of the previous version before you save changes, since a single mistyped rule can have site-wide consequences with no confirmation step between you and production.
A real, complete WordPress robots.txt example
The example below is a reasonable, production-ready starting point for a typical WordPress site running a caching layer and a standard sitemap setup. It is not the only correct configuration, but every line in it does something specific and defensible.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /*?replytocom=
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /*.pdf$
Sitemap: https://example.com/sitemap.xmlThe first line, User-agent: *, means every rule below it applies to all crawlers unless a later block specifically targets a different user agent. Disallow: /wp-admin/ keeps the entire admin backend out of crawling, since none of it is content anyone should find through search. The Allow: /wp-admin/admin-ajax.php line immediately after is a deliberate exception carved out of that block, because some front-end features route requests through that specific file and blocking it can quietly break functionality on the public side of the site.
Disallow: /wp-login.php keeps the login form itself out of the index, which has no SEO value and no reason to be crawled. The Disallow: /?s= rule targets WordPress's internal search results URL pattern, since search result pages are typically thin, duplicative, and not something you want competing with your real content pages for crawl attention or appearing in someone else's search results. The replytocom rule blocks the URL variants WordPress generates for comment reply links, which create large numbers of near-duplicate URLs for the same underlying page.
The trackback and feed rules block WordPress's built-in trackback URLs and RSS feed endpoints from crawling, since these are functional endpoints rather than pages meant to rank. The PDF rule is optional and site-specific: it is useful if you host downloadable PDFs that duplicate content already published as a normal page and you would rather the HTML version be the one that shows up in results. Finally, the Sitemap directive tells any crawler reading this file exactly where to find your XML sitemap, which is the single most useful line in the entire file for helping search engines discover your content efficiently. Rankwyn's sitemap and indexing tools keep this path correct automatically as your sitemap structure changes, so you never end up with a Sitemap directive pointing at a stale or renamed file.
Common mistakes that accidentally block your whole site
The single most damaging robots.txt mistake, and one that happens constantly, is a leftover Disallow: / rule carried over from a staging environment. Staging sites are commonly configured to block all crawling so an unfinished site does not get indexed. When that staging site becomes the production site, whether through a domain change, a migration, or a hosting provider's go-live process, the robots.txt file sometimes comes along for the ride unchanged. The result is a fully functional, fully indexed site that suddenly tells every crawler not to request anything at all.
Disallow: / blocks everything
A bare Disallow: / under User-agent: * blocks crawling of every single URL on your domain. There is no partial version of this mistake. If you see this rule in a live production robots.txt and it was not placed there deliberately, remove it immediately and verify in Search Console that crawling resumes.
A second common mistake is a syntax error that behaves differently than intended. robots.txt rules are matched as simple string prefixes, not full regular expressions, though wildcards are supported by most major crawlers. A rule like Disallow: /blog can unintentionally block every URL that starts with those characters, including /blog-post-title or /blogger-outreach, not just the intended /blog/ directory, because there is no trailing slash forcing an exact directory match.
A third mistake is conflicting rules from multiple sources. If a security plugin, a caching plugin, and your SEO plugin all try to write their own robots.txt rules, whichever one saves last silently overwrites the others, and there is no warning that a previous rule set has been discarded. This is one of the reasons it is worth settling on a single tool as the source of truth for this file rather than letting several plugins compete for control of it.
A fourth, quieter mistake is forgetting to remove a temporary block after a migration, redesign, or maintenance window. It is common practice to disallow all crawling while a site is mid-migration to avoid search engines indexing a half-finished state. The mistake is not adding that rule, it is forgetting to remove it once the work is done, sometimes for weeks, while organic traffic quietly drains away with no obvious error message pointing at the cause.
AI crawlers and robots.txt
robots.txt is also where you make explicit decisions about AI crawlers, and this is a newer, still-evolving part of the file for most WordPress site owners. The relevant bots include GPTBot and ChatGPT-User from OpenAI, ClaudeBot and Claude-User from Anthropic, PerplexityBot from Perplexity, Google-Extended from Google, and CCBot from Common Crawl, alongside the standard search crawlers like Googlebot and Bingbot. Each of these can be targeted individually with its own User-agent block, and the decision to allow or block one has no effect on the others, even when two bots share the same operator.
GPTBot and ChatGPT-User are a useful pair to understand this distinction through. GPTBot performs periodic bulk crawling, largely for training and index building, while ChatGPT-User performs a live fetch specifically when a person using ChatGPT asks a question that requires reading your page right now. Blocking GPTBot does not block ChatGPT-User, and the reverse is also true. The same separation exists between ClaudeBot and Claude-User, and between Googlebot and Google-Extended. Our AI crawler traffic guide walks through this framework in full, including how to read your server logs to see which of these bots are already visiting.
If you want to allow the major AI crawlers explicitly rather than relying on a wildcard rule to cover them, add individual blocks for each:
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /If your priority is being available for citation in a live AI answer while opting out of bulk training datasets, you can allow the fetch-on-demand bots while blocking the periodic training crawlers, or block CCBot specifically since it feeds a shared public training archive rather than any single assistant's live citation behavior. There is no universally correct stance here. It depends on whether you value being cited in AI-generated answers more than you value keeping your content out of model training data, and those two goals sometimes pull in different directions. Rankwyn's AI traffic insights show you which of these bots are actually visiting your site before you decide how to configure any of this, which turns the decision into something based on real traffic rather than a guess.
How to safely test a robots.txt change
Testing a robots.txt change before it goes live
Edit or configure the rule
Make the change in your SEO plugin's robots.txt editor rather than editing the live file directly, and keep a copy of the previous version.
Test with a robots.txt checking tool
Use Search Console's robots.txt testing tool, or an equivalent validator, to confirm the specific URLs you care about are allowed or blocked as intended before saving.
Verify the Sitemap directive still resolves
Open the Sitemap URL listed in the file directly and confirm it returns a valid XML sitemap rather than a 404 or an outdated file path.
Monitor crawl stats for a week
Check Search Console's crawl stats and your server logs over the following week to confirm crawl volume looks normal and no important section of the site has gone quiet.
This sequence matters most on larger or higher-traffic sites, where a mistake left unnoticed for even a few days can produce a measurable dip in crawl activity and, eventually, in rankings. On a small personal blog, the cost of a mistake is lower and recovery is faster once the error is caught. Either way, testing before saving and checking again after saving costs a few minutes and removes almost all of the risk in this part of your SEO setup.