# Content Signals (https://contentsignals.org/, draft-romm-aipref-contentsignals) # Declares how crawled content may be used. Applied globally. Content-Signal: search=yes, ai-input=yes, ai-train=yes # AI search bots — user-triggered fetchers (fire when someone asks an AI a question) User-agent: OAI-SearchBot Allow: / Allow: /api/v1/ Disallow: /api/ Disallow: /jobs? User-agent: ChatGPT-User Allow: / Allow: /api/v1/ Disallow: /api/ Disallow: /jobs? User-agent: Perplexity-User Allow: / Allow: /api/v1/ Disallow: /api/ Disallow: /jobs? # AI training/indexing crawlers User-agent: GPTBot Allow: / Allow: /api/v1/ Disallow: /api/ Disallow: /jobs? User-agent: Google-Extended Allow: / Allow: /api/v1/ Disallow: /api/ Disallow: /jobs? User-agent: PerplexityBot Allow: / Allow: /api/v1/ Disallow: /api/ Disallow: /jobs? User-agent: ClaudeBot Allow: / Allow: /api/v1/ Disallow: /api/ Disallow: /jobs? User-agent: Applebot-Extended Allow: / Allow: /api/v1/ Disallow: /api/ Disallow: /jobs? User-agent: CCBot Allow: / Allow: /api/v1/ Disallow: /api/ Disallow: /jobs? # Default rule for all other crawlers User-agent: * Allow: / Disallow: /api/ Disallow: /jobs? Sitemap: https://agentic-engineering-jobs.com/sitemap.xml