Metoolzyrobots.txt Generator
User-agent: * Allow: /
Visual Builder
Build robots.txt with a form interface.
Sitemap Ref
Include sitemap URL reference.
Copy Result
One-click copy to clipboard.
Free & Fast
No signup, no limits. Works instantly.
เกี่ยวกับ robots.txt Generator
The Robots.txt Generator creates a properly formatted robots.txt file — the standard protocol that tells web crawlers like Googlebot, Bingbot, and others which pages or sections of your site to index and which to skip. Every public website should have a robots.txt file at its root. Without one, crawlers will attempt to index everything, including sensitive admin pages, duplicate content, staging URLs, and media files you may not want appearing in search results. With this visual builder you select a user-agent (or use * for all bots), add Allow and Disallow rules for specific paths, set an optional crawl delay to control crawl rate, and reference your XML sitemap. The generated output follows RFC standard formatting and can be downloaded or copied directly into your web server root.
วิธีใช้ robots.txt Generator
- Set the User-agent field — use * to target all crawlers, or enter a specific bot name like Googlebot.
- Add path rules: choose Allow or Disallow and enter the URL path (e.g. /admin/ to block an admin area).
- Add more rules with the "+ Add Rule" button for each path you want to control.
- Optionally enter a Crawl Delay (in seconds) if you want to slow down how fast bots crawl your site.
- Paste your sitemap URL so search engines can discover your XML sitemap automatically.
- Click Copy or Download to get your robots.txt file, then upload it to the root of your website.
คำถามที่พบบ่อย
Where do I place the robots.txt file on my website?
The file must be placed at the root of your domain — for example, https://example.com/robots.txt. Web servers typically expect this file at the top-level public directory (e.g. /public_html/ or /var/www/html/).
Does robots.txt prevent pages from appearing in Google search results?
Disallowing a URL in robots.txt prevents Googlebot from crawling it, but it does not guarantee the page will not appear in search results. If other sites link to a disallowed page, Google may still index the URL. To fully exclude a page from search results, use the noindex meta tag or HTTP header.
What is the difference between Disallow: / and Disallow: /admin/?
Disallow: / blocks all pages on the entire site from being crawled. Disallow: /admin/ blocks only pages within the /admin/ directory. Blocking your entire site is typically done to prevent indexing of development or staging environments.
What does the Crawl-delay directive do?
Crawl-delay tells bots to wait a specified number of seconds between requests. This is useful for servers with limited resources. Note: Google does not officially honour Crawl-delay in robots.txt — use Google Search Console to manage Googlebot's crawl rate instead.
Should I list my sitemap in robots.txt?
Yes. Including a Sitemap: directive in robots.txt is a quick, reliable way to ensure all major search engines discover your XML sitemap without requiring manual submission in each webmaster tool.
เครื่องมือที่เกี่ยวข้อง
Top 10 Free Tools Every Developer Needs
Boost your productivity instantly. Get our exclusive cheatsheet featuring the most essential web utilities you didn't know you needed, delivered straight to your inbox.
We respect your inbox. Unsubscribe at any time.