User-agent: * Allow: / # The analytics beacon. It only ever answers POST with 204 No Content, so it is # not a page and has nothing to index. Google was still crawling it and filing # it under "Crawled - currently not indexed", which wastes crawl budget and # clutters the report. A crawler obeys only the one group that matches it, so # the named AI crawlers below repeat this line rather than inheriting it. Disallow: /api/ # AI retrieval and citation crawlers, named explicitly. This is a lead-gen local # business with no ad-revenue model to protect, so being surfaced and cited in AI # answers is upside, not a cost. Named rather than left to the wildcard so any # future change here is a deliberate decision instead of an accident. User-agent: GPTBot Allow: / Disallow: /api/ User-agent: OAI-SearchBot Allow: / Disallow: /api/ User-agent: ChatGPT-User Allow: / Disallow: /api/ User-agent: ClaudeBot Allow: / Disallow: /api/ User-agent: Claude-Web Allow: / Disallow: /api/ User-agent: PerplexityBot Allow: / Disallow: /api/ User-agent: Perplexity-User Allow: / Disallow: /api/ User-agent: Google-Extended Allow: / Disallow: /api/ User-agent: Applebot-Extended Allow: / Disallow: /api/ User-agent: CCBot Allow: / Disallow: /api/ Sitemap: https://www.strictlyjunkremoval.com/sitemap-index.xml