robots.txt Generator

为网站创建robots.txt。
Rules
输出
User-agent: *
Allow: /

Visual Builder

Build robots.txt with a form interface.

Sitemap Ref

Include sitemap URL reference.

Copy Result

One-click copy to clipboard.

Free & Fast

No signup, no limits. Works instantly.

关于robots.txt Generator

The Robots.txt Generator creates a properly formatted robots.txt file — the standard protocol that tells web crawlers like Googlebot, Bingbot, and others which pages or sections of your site to index and which to skip. Every public website should have a robots.txt file at its root. Without one, crawlers will attempt to index everything, including sensitive admin pages, duplicate content, staging URLs, and media files you may not want appearing in search results. With this visual builder you select a user-agent (or use * for all bots), add Allow and Disallow rules for specific paths, set an optional crawl delay to control crawl rate, and reference your XML sitemap. The generated output follows RFC standard formatting and can be downloaded or copied directly into your web server root.

如何使用robots.txt Generator

  1. Set the User-agent field — use * to target all crawlers, or enter a specific bot name like Googlebot.
  2. Add path rules: choose Allow or Disallow and enter the URL path (e.g. /admin/ to block an admin area).
  3. Add more rules with the "+ Add Rule" button for each path you want to control.
  4. Optionally enter a Crawl Delay (in seconds) if you want to slow down how fast bots crawl your site.
  5. Paste your sitemap URL so search engines can discover your XML sitemap automatically.
  6. Click Copy or Download to get your robots.txt file, then upload it to the root of your website.

常见问题

Where do I place the robots.txt file on my website?

The file must be placed at the root of your domain — for example, https://example.com/robots.txt. Web servers typically expect this file at the top-level public directory (e.g. /public_html/ or /var/www/html/).

Does robots.txt prevent pages from appearing in Google search results?

Disallowing a URL in robots.txt prevents Googlebot from crawling it, but it does not guarantee the page will not appear in search results. If other sites link to a disallowed page, Google may still index the URL. To fully exclude a page from search results, use the noindex meta tag or HTTP header.

What is the difference between Disallow: / and Disallow: /admin/?

Disallow: / blocks all pages on the entire site from being crawled. Disallow: /admin/ blocks only pages within the /admin/ directory. Blocking your entire site is typically done to prevent indexing of development or staging environments.

What does the Crawl-delay directive do?

Crawl-delay tells bots to wait a specified number of seconds between requests. This is useful for servers with limited resources. Note: Google does not officially honour Crawl-delay in robots.txt — use Google Search Console to manage Googlebot's crawl rate instead.

Should I list my sitemap in robots.txt?

Yes. Including a Sitemap: directive in robots.txt is a quick, reliable way to ensure all major search engines discover your XML sitemap without requiring manual submission in each webmaster tool.

robots.txt Generator 完全使用指南与免费在线工具

无需安装、完全免费、纯前端本地运行的高效 robots.txt Generator 工具

用于快速解决：为网站创建robots.txt。。无需安装复杂软件，打开网页即可立即使用，极速且安全。

为什么选择 Metoolzy 的 robots.txt Generator？

  • 秒级即时响应: 极致性能优化，输入即可见结果。
  • 数据绝对安全: 纯浏览器本地计算，数据绝不上传服务器。
  • 完全免费使用: 无需注册登录，无任何隐形消费。

三步快速上手

  1. 打开 Metoolzy robots.txt Generator 工具页面。
  2. 在输入区粘贴或上传您的数据。
  3. 即时查看处理结果并点击复制或下载。

功能对比表

对比项 Metoolzy robots.txt Generator 普通在线工具 本地客户端软件
使用成本 永久免费 有次数限制或收费 需购买授权
隐私保护 纯本地内存运行 数据需上传云端 本地运行

适用人群与应用场景

  • 软件开发人员日常调试与数据转换
  • UI/UX 设计师资产处理
  • 运营与办公人员效率提升

常见问题解答 (FAQ)

处理的数据会被记录或保存吗？

绝不会。所有运算均在您的浏览器本地完成，没有任何数据回传。