Web Robots

Web Robots fetches and parses robots.txt for any domain. Returns parsed allow/disallow rules grouped by user-agent, sitemap URLs declared in the file, crawl-delay directives, and the raw file content (first 5000 characters). Built for AI agents planning ethical, compliant web crawling, and for SEO audits.

Requests time out after 12 seconds and responses over 5MB are rejected. If a domain has no robots.txt (HTTP 404), the service returns exists: false rather than an error. No external API key is required.

Endpoint: GET /web/robots?domain=example.com

Web Robots（robots.txt 解析）抓取并解析任意域名的robots.txt文件，返回按user-agent分组的allow/disallow规则、文件中声明的sitemap地址、crawl-delay抓取延迟指令，以及原始文件内容（前5000字符）。适用于AI agent规划合规的网络爬取策略，以及SEO审计。

请求12秒超时，响应体超过5MB会被拒绝。若域名没有robots.txt（HTTP 404），服务返回exists: false而非报错。无需任何外部API key。

端点：GET /web/robots?domain=example.com