Blocked by Robots.txt — How to Find and Fix the Issue

What robots.txt blocking means

Your site's robots.txt file tells search engine crawlers which URLs they are allowed to visit. If a URL matches a Disallow rule for Googlebot, Google cannot crawl it — and a page Google can't crawl won't be indexed.

This is different from a noindex tag: robots.txt blocks the crawl entirely, so Google may still show the URL in results with no content snippet, based on links pointing to it.

How to find your robots.txt file

Your robots.txt is always at the root of your domain:

1. Visit yourdomain.com/robots.txt

   Open your browser and go to your domain followed by /robots.txt. You'll see the plain text file.

2. Look for Disallow rules

   Find any lines starting with Disallow: under a User-agent: * or User-agent: Googlebot block. Check if your URL matches any of them.

How to test a URL against robots.txt

1. Use Google Search Console's robots.txt tester

   In Search Console, go to Settings → robots.txt. Paste the URL you want to test and click Test — it will tell you if Googlebot is blocked.

How to fix it

• Remove or narrow the Disallow rule that's blocking the URL — e.g. change Disallow: / to Disallow: /private/
• If the block is intentional (admin pages, internal tools), make sure the URL you want indexed is not in a blocked path
• After fixing, use GoIndexed's Re-inspect button or request indexing in Search Console to prompt Google to re-crawl

Common mistakes that cause accidental blocks

• Disallow: / blocks the entire site — usually left over from a staging environment
• WordPress's "Discourage search engines" setting adds Disallow: / automatically
• Overly broad wildcard patterns like Disallow: /*?* that accidentally match real URLs