# Traditional search engines User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Bingbot Allow: / # AI search & answer engines (GEO: these power cited answers) User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: GPTBot # Allow AI crawlers to access publicly available content for indexing and training Allow: / User-agent: Google-Extended Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Applebot-Extended Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: MistralAI-User Allow: / User-agent: CCBot Allow: / User-agent: Diffbot Allow: / User-agent: Amazonbot Allow: / User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / User-agent: DuckAssistBot Allow: / User-agent: YouBot Allow: / User-agent: Timpibot Allow: / User-agent: AI2Bot Allow: / User-agent: ImagesiftBot Allow: / # ── Blocked crawlers (deliberate, measured — do not re-allow without evidence) ─ # ShapBot (Parallel / parallel.ai). 2026-09-22 measurement with # `wrangler tail --format json` over a 240s passive window: ShapBot accounted for # 127 of 203 Worker invocations (63%) and drove the bulk of the CPU time spent # above the Workers Free ceiling of 10ms/request. It walks the long tail # (/compare/*, /rank/{slug}/, /alternatives/*) one URL at a time — the exact # pattern a 60s edge cache cannot help with, because every request is a miss and # therefore a full ~20ms render. # # Parallel documents this directive as the supported control: "Webmasters can # manage automated access using robots.txt with the User-agent: ShapBot # directive." (https://parallel.ai/parallel-web-systems-bots) # # The trade-off was accepted knowingly: ShapBot builds Parallel's search index, # so this removes Harpd from Parallel's web APIs. Re-allow by deleting this group # once the CPU headroom is regained — i.e. once the renders are under 10ms, or # Workers is upgraded to Paid. # # Shap-User is deliberately NOT blocked. Parallel documents it as user-initiated # retrieval ("not used for crawling the web in an automatic fashion") — the same # shape as ChatGPT-User / Perplexity-User above, which this file allows. User-agent: ShapBot Disallow: / User-agent: * Allow: / Disallow: /api/ # P2.7 — the machine-readable page API is public, factual and citation-shaped. # It is the one /api/ path that should be crawlable, so it is re-allowed here. Allow: /api/seo/ # P0.4 — the citation registry API is public, factual and cacheable; AI answer # engines should be able to retrieve Harpd's claim provenance directly. Allow: /api/citations Allow: /api/citations.json # P0.6 — /api/citation/{type}/{slug} is the public per-object citation endpoint # (product / category / research / dataset / rank board). Distinct path from the # /api/citations/ registry above, so it needs its own Allow or it is swallowed # by `Disallow: /api/`. Allow: /api/citation/ # P4.1 — the /api/ai/ namespace is the AI retrieval surface over the same # public datasets; answer engines may read it directly. Allow: /api/ai/ Allow: /agent.json Allow: /.well-known/agent.json # Agent-Ready layer (2026-09-17). /.well-known/harpd-agent.json is the # capability + pricing manifest an AI agent reads before it calls anything; the # /for-agents/ and /developers/agents/ trees are the human-facing half of the # same layer. All three must be crawlable or an agent cannot discover that Harpd # is purchasable at all. Allow: /.well-known/harpd-agent.json Allow: /for-agents/ Allow: /developers/agents/ Disallow: /rank/manage/ Disallow: /rank/claim/ Disallow: /rank/credits/ # /admin/* is an internal ops surface (SEO dashboard, citation ops). It is # already gated by a shared-secret Authorization header that returns 403, but # crawlers should not request it at all. Disallow: /admin/ # Content Signals - AI content usage preferences Content-Signal: ai-train=yes, search=yes, ai-input=yes # Sitemap index plus per-page-type child sitemaps (P0.9: `programmatic` was # split into best / compare / alternatives / answers so each family carries its # own URL count and freshness signal in Search Console). Sitemap: https://harpd.com/sitemap.xml # P0.3 — mission-facing segmented sitemaps at the site root. Each is an alias # of the canonical child group in src/lib/sitemap.ts (both routes call the same # `sitemapUrlsFor`), so the two forms can never advertise different URLs. # `/sitemap-comparisons.xml` resolves to the `compare` group. Sitemap: https://harpd.com/sitemap-pages.xml Sitemap: https://harpd.com/sitemap-products.xml Sitemap: https://harpd.com/sitemap-categories.xml Sitemap: https://harpd.com/sitemap-research.xml Sitemap: https://harpd.com/sitemap-answers.xml Sitemap: https://harpd.com/sitemap-comparisons.xml Sitemap: https://harpd.com/sitemap-alternatives.xml Sitemap: https://harpd.com/sitemap-data.xml # Canonical child groups (back-compatible form, kept advertised for existing # Search Console submissions). Sitemap: https://harpd.com/sitemaps/pages.xml Sitemap: https://harpd.com/sitemaps/rank.xml Sitemap: https://harpd.com/sitemaps/categories.xml Sitemap: https://harpd.com/sitemaps/products.xml Sitemap: https://harpd.com/sitemaps/history.xml Sitemap: https://harpd.com/sitemaps/blog.xml Sitemap: https://harpd.com/sitemaps/best.xml Sitemap: https://harpd.com/sitemaps/compare.xml Sitemap: https://harpd.com/sitemaps/alternatives.xml Sitemap: https://harpd.com/sitemaps/answers.xml Sitemap: https://harpd.com/sitemaps/research.xml Sitemap: https://harpd.com/sitemaps/data.xml