These two files get discussed together and confused constantly, so start with the distinction that resolves most of the confusion:
Advertisement. We earn a commission if you sign up through this link, at no extra cost to you.
Robots.txt controls crawling. It tells bots which paths they may fetch. It says nothing about what appears in search results.
A sitemap suggests what to crawl. It is a list of URLs you consider worth indexing. It is a hint, not an instruction, and being in a sitemap guarantees nothing.
Neither controls indexing. That is the robots meta tag, and mixing this up is the single most expensive mistake in this area.
The mistake that costs people their traffic
Blocking a URL in robots.txt does not remove it from Google. It prevents Google from fetching it — which also prevents Google from seeing any noindex tag on it.
So a page that is blocked in robots.txt and linked from anywhere else can still appear in results, listed by URL with a note that no description is available. You have made it uglier, not invisible.
Worse, if a page is already indexed and you block it hoping it will drop out, you have guaranteed the opposite: Google can no longer fetch the page, so it can never read the noindex you added, and the URL stays in the index indefinitely.
The correct sequence when you want something out of the index:
- Allow crawling.
- Add <meta name=”robots” content=”noindex”> to the page.
- Wait for Google to recrawl and drop it.
- Only then block it in robots.txt, if you even still want to.
And do not try to use Noindex: as a robots.txt directive. Google stopped supporting it in September 2019. It was never part of the standard and it does nothing now.
A robots.txt that is actually useful
For most WordPress sites, correct is close to minimal:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlThat is genuinely most of it. WordPress serves this virtually if no physical file exists, and your SEO plugin usually manages it.
Things worth adding only if they apply to you: internal search results (Disallow: /?s=), which can generate unbounded crawlable URLs; faceted navigation parameters on a shop; and cart or checkout paths.
Things not worth adding, despite appearing in every template you will find: Disallow: /wp-includes/ and blocks on CSS or JavaScript directories. Google renders pages to evaluate them, and blocking assets means it renders a broken version of your site.
The mechanics that matter: the file must be at the domain root, one per host — example.com and shop.example.com need their own — and Google reads up to 500 KB of it. Paths are case-sensitive. A missing robots.txt returning a 404 is fine and means everything is crawlable. A robots.txt returning a 500 is not fine — Google may treat repeated server errors as a signal to stop crawling the site entirely.
AI crawlers
This is the part that has actually changed, and it is a business decision rather than a technical one.
The major AI crawlers each declare a user-agent you can address:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /Two things worth understanding before you paste that in.
Google-Extended is not Googlebot. Blocking it opts you out of training for Google’s generative products without affecting how you are crawled or ranked in Search. They are genuinely separate controls.
Blocking training crawlers can also remove you from AI answers. If a growing share of people find information through assistants rather than a results page, being absent from what those systems can cite is a visibility cost, not just a data-licensing win. For a site selling expertise, being quotable is arguably the point.
And the obvious caveat: robots.txt is a convention that well-behaved crawlers respect voluntarily. Scrapers that ignore it will keep ignoring it. If you need enforcement rather than a request, that is a firewall rule, not a text file.
Sitemaps
A sitemap is a list of the URLs you want indexed. Its real value is on sites where discovery is hard — large archives, pages with few internal links, new sites with no inbound links. On a well-linked fifty-page site, Google would find everything anyway, and the sitemap mostly earns its place through the reporting it unlocks.
WordPress has generated /wp-sitemap.xml since version 5.5. Every SEO plugin replaces it with its own, usually at /sitemap_index.xml, and disables the core one. If both are live and reachable, that is worth fixing — two competing lists of the same site is exactly the kind of contradictory signal to avoid.
What belongs in it: canonical URLs, returning 200, that you want indexed.
What does not, and what commonly ends up there anyway: URLs that redirect, URLs that are noindexed, URLs that canonicalize to something else, and paginated archives. Every one of these tells Google to index something you have separately told it not to.
On the optional tags — priority and changefreq are ignored by Google and have been for years, so leave them out. lastmod is used, but only if it is accurate; a sitemap that claims every page changed today is one Google learns to disregard.
The limits: 50,000 URLs and 50 MB uncompressed per file. Past that, split into multiple sitemaps behind an index file, which is what every plugin does automatically.
Submitting and checking
Submit in Search Console under Sitemaps, and reference it in robots.txt with the Sitemap: line. The second one matters more than people assume, because it is how crawlers other than Google find it.
Once it is submitted, the Sitemaps report gives you the number that actually matters: discovered versus indexed. A large gap is the clearest quality signal Google gives you for free. If you have submitted 74 URLs and 26 are indexed, no amount of tag optimization is your problem — Google has looked at the rest and declined, and the answer is fewer, better pages rather than more of the same.
Two tools worth knowing about:
The robots.txt report in Search Console, under Settings, shows the file as Google last fetched it and flags parse errors. It replaced the old robots.txt Tester, which was retired at the end of 2023 — guides still pointing you to the Tester are out of date.
URL Inspection answers the specific question for a specific page: is it blocked, is it indexed, and which canonical did Google choose.
The short version
Keep robots.txt short and use it to stop crawling, never to stop indexing. Never block a page you want removed from the index — noindex it and let Google fetch it. Decide deliberately about AI crawlers rather than copying someone else’s block list. Keep the sitemap to canonical, indexable URLs only, reference it from robots.txt, and watch the discovered-versus-indexed gap, because that number tells you more about your site than either file does.
Working through this list on your own site and would rather not? That is what an Expert Web Audit is for.
Advertisement. We earn a commission if you sign up through this link, at no extra cost to you.
