{"product_id":"web-robots","title":"Web Robots","description":"\u003cp\u003e\u003cstrong\u003eWeb Robots\u003c\/strong\u003e fetches and parses robots.txt for any domain. Returns parsed allow\/disallow rules grouped by user-agent, sitemap URLs declared in the file, crawl-delay directives, and the raw file content (first 5000 characters). Built for AI agents planning ethical, compliant web crawling, and for SEO audits.\u003c\/p\u003e\n\u003cp\u003eRequests time out after 12 seconds and responses over 5MB are rejected. If a domain has no robots.txt (HTTP 404), the service returns \u003ccode\u003eexists: false\u003c\/code\u003e rather than an error. No external API key is required.\u003c\/p\u003e\n\u003cp\u003e\u003cstrong\u003eEndpoint:\u003c\/strong\u003e GET \/web\/robots?domain=example.com\u003c\/p\u003e\n\u003chr\u003e\n\u003cp\u003e\u003cstrong\u003eWeb Robots（robots.txt 解析）\u003c\/strong\u003e抓取并解析任意域名的robots.txt文件，返回按user-agent分组的allow\/disallow规则、文件中声明的sitemap地址、crawl-delay抓取延迟指令，以及原始文件内容（前5000字符）。适用于AI agent规划合规的网络爬取策略，以及SEO审计。\u003c\/p\u003e\n\u003cp\u003e请求12秒超时，响应体超过5MB会被拒绝。若域名没有robots.txt（HTTP 404），服务返回\u003ccode\u003eexists: false\u003c\/code\u003e而非报错。无需任何外部API key。\u003c\/p\u003e\n\u003cp\u003e\u003cstrong\u003e端点：\u003c\/strong\u003eGET \/web\/robots?domain=example.com\u003c\/p\u003e","brand":"Nexus Agent","offers":[{"title":"Default Title","offer_id":49067217846457,"sku":null,"price":20.0,"currency_code":"USD","in_stock":true}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/0814\/3024\/9657\/files\/web_robots.png?v=1788478911","url":"https:\/\/nexusagentapi.com\/products\/web-robots","provider":"Nexus Agent LLC","version":"1.0","type":"link"}