Robots.txt Explained: What It Actually Blocks (and What It Doesn't)

The File That Confuses More SEOs Than Any Other

I've lost count of the audits where the first red flag was a robots.txt file that somebody wrote with good intentions and broke the site with. One of my all-time favorites: a client whose entire sitemap was blocked, not because he wanted it blocked, but because he copied a "best practice" robots.txt from a forum post that disallowed /sitemap/ — while his actual sitemap lived at /sitemap.xml.

Different slash. That's all it took. Google crawled his homepage, saw the sitemap URL was disallowed, and quietly stopped discovering half his content.

The frustrating part? Nothing looked wrong. The site ranked fine for the pages Google already knew about. But new pages? They were crawling into a void. It took me three hours to figure out why a client's fresh content wasn't getting indexed, and the answer was one line in a text file that had been wrong for six months.

What robots.txt Actually Does

Robots.txt is a set of rules that tells crawlers which paths they may and may not request. It's read by Googlebot, Bingbot, and most other crawlers before they start fetching a site.

The critical thing to understand: robots.txt is not access control. It's a request. A polite sign on the door, not a lock. And crucially, blocking a URL in robots.txt does NOT remove it from the index. If a page is already indexed and you block it, Google keeps showing it in results — it just can't crawl it anymore, which means it can't see the title, description, or content updates. You get a listing with whatever snippet Google last stored.

Robots.txt is for saving crawl budget and keeping Googlebot out of the junk. The pages you genuinely don't want in search deserve a noindex tag, not a disallow.

The Structure, Demystified

A robots.txt file is boring on purpose:

User-agent: *
Disallow: /private/
Allow: /private/public/

Sitemap: https://yoursite.com/sitemap.xml

• User-agent tells the rules which crawler they apply to.
• Disallow lists paths the crawler shouldn't fetch.
• Allow carves out exceptions inside a disallowed area.
• Sitemap points crawlers at your sitemap.

Wildcards are supported by Googlebot — $ for "ends with" and * for "any characters" — but a lot of SEOs overuse them and end up with rules that block more than intended.

The Mistakes That Quietly Hurt You

1. Blocking your sitemap

Your sitemap should never be disallowed. I genuinely don't know why so many example files include it. Blocking the sitemap is like telling Googlebot "don't come look at my list of everything."

2. Disallow on pages you want indexed

Anything you want in Google's index must be crawlable. If you disallow it, Google can't discover it from your site. This sounds obvious, but I've seen sites block their entire blog directory, their categories, their products. Sometimes from an old rule nobody remembers adding.

3. Using robots.txt instead of noindex

As I said: disallow is not noindex. If you want a page out of the index, put a noindex tag on it and leave it crawlable. If you block it in robots.txt, the noindex tag becomes invisible to Google (it can't fetch the page to read the tag), and you're stuck in a worse state than you started.

4. Blocking Googlebot instead of just slow resources

Blocking an entire JS or CSS file to "save bandwidth" is a 2013-era move. Modern Googlebot renders pages, and if you block the assets that make your page render, you end up with a page Google can't understand. Block the resource files that genuinely waste crawl budget, not the ones that build your pages.

5. No sitemap line at all

The Sitemap directive isn't mandatory, but if your site is large or complex, it's free help. Crawlers will find your sitemap from your robots.txt, your Search Console submission, or your homepage links. Don't make them work for it.

How to Check Your robots.txt Properly

Here's the validation pass I run:

1. Fetch your robots.txt and read every rule line by line.

2. Confirm the sitemap path matches the real sitemap location exactly.

3. Check that no rule blocks /, /sitemap.xml, or your main directories.

4. Make sure any Disallow you have is for pages you genuinely don't want crawled.

5. Test a few real URLs against your rules with a robots tester.

6. Verify you can still fetch your pages in an incognito browser — robots.txt blocks your own browser too if you're on a disallowed path.

The robots.txt checker on AuditMe does steps 1 through 5 for you. Give it a domain, it pulls the file, evaluates the rules against your key paths, and flags the landmines — including the sitemap-blocking classic.

Should You Just Delete the File?

If your robots.txt is a mess and you're not sure which rules are safe to keep, the honest answer is often yes, delete the whole thing and start over. An empty file means "crawl everything you can find", which is the right default for 95% of sites. Then add back only the rules you can explain out loud in one sentence.

For sites that need structure — a staging area, a private admin path, an infinite pagination black hole — keep the file minimal and comment your reasoning so the next person doesn't have to reverse-engineer it.

FAQ

Does blocking a page in robots.txt remove it from Google?

No. It only stops Google from crawling it. The page can stay in the index with stale content. Use noindex if you want a page gone.

Should my sitemap be listed in robots.txt?

Yes, and the path must match exactly. It's a small convenience that saves crawlers from hunting for it.

Can robots.txt block Google from rendering JavaScript?

Yes, indirectly. If you disallow the JS or CSS files, Googlebot can't render the page properly, which can make your content invisible even though the HTML is crawlable.

Validate Before It Costs You

The most expensive bugs in SEO are the ones that look healthy. A bad robots.txt is the poster child — the site "works", traffic is fine, and nobody notices that new content is silently not being indexed. Run a robots.txt check on your domain today, read every rule, and delete anything you can't explain. Future you will thank current you.