Robots.txt Tester

Fetch and analyze robots.txt. See rules per user-agent, Sitemap lines, file size, and test whether a path is Allowed or Blocked for a specific bot.

Also try our Open Graph Checker and Hreflang Validator.

What Is a Robots.txt Tester?

A Robots.txt Tester is an SEO tool that fetches your site’s /robots.txt, parses User-agent sections and their Allow/Disallow rules, lists declared Sitemap URLs, and lets you test any path to see if it’s Allowed or Blocked for a specific crawler (e.g., Googlebot, Bingbot, DuckDuckBot).

Why Robots.txt Matters for SEO

Controls crawling of non-public or duplicate paths (e.g., filters, internal search, admin).
Protects crawl budget by preventing bots from wasting time on low-value URLs.
Improves discovery by declaring Sitemap: locations directly in robots.txt.
Communicates rules per bot with dedicated User-agent sections (e.g., Googlebot-Image).

How to Use This Robots.txt Tester

Enter any page on your site (we’ll resolve the correct /robots.txt).
Pick a User-agent (Googlebot by default).
(Optional) Enter a path to test (e.g., /blog/amp/ or /wp-admin/admin-ajax.php).
Click Test Robots.txt to see parsed rules, sitemaps, and path result.

Quick Tips for Clean Robots Rules

Use Allow/Disallow with wildcards (*) and end anchors ($).
Keep file size reasonable (under a few hundred KB; ideally <100KB).
Declare one or more Sitemap: lines pointing to your XML sitemaps.
Prefer absolute paths (always starting with /).
Separate sections per User-agent if you need bot-specific behavior.

Common Robots.txt Mistakes (and Fixes)

Blocking all content: Disallow: / under User-agent: * prevents crawling of everything. Fix: Remove or scope to a specific user-agent.
Confusing indexing with crawling: Robots rules don’t guarantee deindexing. Fix: Use noindex meta (on accessible pages) or remove from sitemaps; consider 410/404.
Forgetting critical assets: Blocking /wp-includes/ or /assets/ may hide CSS/JS. Fix: Allow assets needed to render pages.
Oversized file or HTML content-type: Serving robots.txt as HTML or with huge size. Fix: Serve text/plain and keep it compact.
Rules before User-agent: Some parsers ignore rules not attached to a section. Fix: Always start with User-agent: then rules.

Recommended Robots.txt Templates

Standard Website

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /search/
Sitemap: https://example.com/sitemap.xml

E-commerce (filter parameters)

User-agent: *
Disallow: /*?sort=
Disallow: /*?price=
Disallow: /*?color=
Allow: /product/
Sitemap: https://example.com/sitemap_index.xml

Media-heavy (let Googlebot-Image crawl images)

User-agent: Googlebot-Image
Allow: /images/
User-agent: *
Disallow: /tmp/
Sitemap: https://example.com/sitemap.xml

How Our “Path Test” Logic Works

We evaluate the selected User-agent section using Google-style pattern rules:

Wildcards: * matches any sequence; end anchor: $ matches end of URL.
Longest match wins; on a tie, Allow beats Disallow.
If no rule matches, the path is considered Allowed.

Robots.txt vs. Noindex vs. Canonical

Robots.txt: controls crawling (access), not indexing.
Meta noindex: controls indexing but requires the page to be crawlable to see the tag.
Canonical: consolidates duplicate signals across crawlable pages.

Robots.txt Checklist

✅ File exists at /robots.txt and returns 200 OK with text/plain.
✅ Important assets (CSS/JS) are not unintentionally blocked.
✅ Sitemap: declared and reachable (200 OK).
✅ Rules are scoped to the right User-agent and use correct patterns.
✅ No blanket Disallow: / under User-agent: * unless intentional.

FAQ: Robots.txt Tester

How do I check if a specific URL is blocked?

Enter the site URL, select a user-agent, and add your path (e.g., /blog/tag/). The tester shows Allowed or Blocked and which rule matched.

Does robots.txt remove pages from Google?

No. It only controls crawling. To remove indexed pages, allow crawling and use noindex, remove from sitemaps, or return 410/404.

Should I block staging or admin areas?

For staging, use authentication or IP restrictions (best). For public admin paths, you can disallow crawling—but never rely on robots.txt for security.

Where should I put my sitemap?

Add one or more Sitemap: lines in /robots.txt pointing to your XML sitemap(s) for faster discovery.

Try our related free tools: Open Graph Checker, Hreflang Validator, Canonical Tag Checker.