Smart Robots.txt Generator

Create SEO-friendly robots.txt files with expert guidance

Search Engine Bot: All Search Engines Google Bing Yahoo Custom Bot

Paths to Block:

Common Blocking Suggestions

🔒 Security Essentials:
/wp-admin/ (WordPress)
/phpmyadmin/ (Database)
/.git/ (Code repository)

🚫 SEO Recommendations:
/search?* (Internal search)
/filter=* (Duplicate content)
/print/ (Printer versions)

Paths to Allow:

Important Allowances

📈 SEO Critical:
/sitemap.xml
/contact-us/
/product-pages/

🖼️ Media Files:
/images/logo.png
/videos/intro.mp4
/downloads/

Crawl Delay (seconds):

Your Sitemap URL:

Your Generated Robots.txt

📖 How to Use This Tool

Step 1: Add Search Engine Rules

Click "Add New Bot Rules" to create different rules for:
• Googlebot (Google's crawler)
• Bingbot (Microsoft's crawler)
• Custom bots

Step 2: Specify Paths

Block Paths:
- Administration panels (/admin/)
- Sensitive files (.env, config.php)
- Internal search results

Allow Paths:
- Important pages (Contact, About)
- Image directories
- Public content

Step 3: Set Crawl Delay

• 10-30 seconds for most websites
• 30-60 seconds for large sites
• Protects server load

Step 4: Add Sitemap

Always include your sitemap URL:
https://yourdomain.com/sitemap.xml

Step 5: Generate & Download

1. Click "Generate Robots.txt"
2. Review the preview
3. Download and upload to your website's root folder

Pro Tips 🔧

• Always test with Google Search Console
• Never block CSS/JS files
• Update when changing site structure
• Keep file under 500KB

© 2023 Smart Robots.txt Generator | Free SEO Tool