Robots.txt Rule Tester
Is your page blocked? Paste your robots.txt file below to simulate how Googlebot parses your rules and verify if a URL is allowed or disallowed.
The Bouncer of Your Website
Think of your website as a club and robots.txt as the bouncer at the door. It tells search engine bots (like Googlebot) which rooms they are allowed to enter and which are “VIP Only.”
However, one wrong line of code in this file can accidentally block Google from your entire site, de-indexing your business overnight.
This Robots.txt Rule Tester simulates Google’s parsing logic. Paste your file and test your URLs to ensure you haven’t accidentally locked yourself out of search results.
How Google Parses Rules (The “Longest Match”)
Many SEOs misunderstand how robots.txt works. It follows a specific order of operations:
- Specificity First: Google looks for a specific User-agent block (e.g.,
User-agent: Googlebot). If found, it IGNORES the wildcard block (User-agent: *). - Longest Match Wins: If you have two conflicting rules, the longer path wins.
Example:
Disallow: /admin/ (7 chars)
Allow: /admin/public/ (14 chars)
Result: A URL like /admin/public/image.jpg will be ALLOWED because the Allow rule is longer.
3 Common Robots.txt Mistakes
Never disallow your /css/ or /js/ folders. Google needs to render your page to see if it’s mobile-friendly. If you block these, your rankings will drop.
Disallow: /blog blocks EVERYTHING starting with /blog (including /blog-post).
Disallow: /blog/ blocks only the folder. Be precise.
Don’t Forget Your Sitemap
The robots.txt file is the first place bots look. Always include a link to your XML sitemap at the very bottom of the file so bots can find your content map immediately.
Sitemap: https://www.yourdomain.com/sitemap_index.xml
Technical SEO Done Right
Robots.txt is just the tip of the iceberg. We perform full technical audits to ensure your site is crawlable, indexable, and rankable.
Trusted by 500+ Businesses • Rated 4.9/5
