# droneops.ro — robots.txt # Updated: 2026-06-29 # # Policy: AI assistants and AI/search crawlers are WELCOME to read our public # pages so DroneOps can be discovered, cited, and recommended. They are NOT # listed with "Disallow: /" below, so they follow the "User-agent: *" rules # (public pages allowed; private/sensitive paths blocked). # Only abusive crawlers and pure-SEO data resellers are blocked outright. # # IMPORTANT: these allow/deny decisions are MIRRORED in the nginx $bad_ua map # (/etc/nginx/nginx.conf). Editing robots.txt alone is not enough — a bot in # that map is rejected with 403 at the edge before robots.txt is ever read. # # Explicitly welcomed AI / search crawlers (they obey "User-agent: *"): # OpenAI (ChatGPT) ... GPTBot, OAI-SearchBot, ChatGPT-User # Anthropic (Claude) . ClaudeBot, Claude-Web, anthropic-ai, Claude-User, Claude-SearchBot # Google ............. Googlebot (Search) + Google-Extended (Gemini / Vertex AI) # Apple .............. Applebot, Applebot-Extended (Apple Intelligence / Siri) # Microsoft / Bing ... bingbot (Copilot) # Others ............. PerplexityBot, YouBot, cohere-ai, Diffbot, CCBot, meta-externalagent # --- Blocked: aggressive scrapers / SEO data resellers (no recommendation value) --- User-agent: Bytespider Disallow: / User-agent: PetalBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: MJ12bot Disallow: / User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: DotBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: serpstatbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / # --- All other bots (incl. the AI / search crawlers above): --- # public pages allowed, private/sensitive paths blocked. User-agent: * Disallow: /contact/ Disallow: /user/ Disallow: /accounts/ Disallow: /dashboard/ Disallow: /api/ Disallow: /admin/ Disallow: /notifications/ Disallow: /missions/ Disallow: /drones/ Disallow: /media/ Disallow: /protected-media/ Disallow: /hooks/ Sitemap: https://droneops.ro/sitemap.xml