# robots.txt - Generated based on log analysis # Blocks 76.8% of identified bot traffic while allowing legitimate crawlers # Block aggressive crawlers and scrapers identified in logs User-agent: Amazonbot Disallow: / User-agent: facebookexternalhit Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Barkrowler Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: YandexBot Disallow: / User-agent: PetalBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: MegaIndex Disallow: / User-agent: LinkpadBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / # AI crawlers and scrapers (TK-0207). Google-Extended and Applebot-Extended only opt out of AI training; search indexing is unchanged. User-agent: ClaudeBot Disallow: / User-agent: Claude-User Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: anthropic-ai Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: YouBot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / # Allow search engines with crawl-delay for rate limiting User-agent: Googlebot Crawl-delay: 10 Disallow: /admin/ Disallow: /private/ Disallow: /api/internal/ Disallow: /static/admin/ Disallow: /django-admin/ Disallow: /__debug__/ Disallow: /accounts/ Disallow: /login/ Disallow: /logout/ Disallow: /password/ Disallow: /signup/ Disallow: /media/private/ User-agent: Bingbot Crawl-delay: 15 Disallow: /admin/ Disallow: /private/ Disallow: /api/internal/ Disallow: /static/admin/ Disallow: /django-admin/ Disallow: /__debug__/ Disallow: /accounts/ Disallow: /login/ Disallow: /logout/ Disallow: /password/ Disallow: /signup/ Disallow: /media/private/ # Allow uptime monitoring services (identified in logs) User-agent: UptimeRobot Crawl-delay: 300 Disallow: /admin/ Disallow: /private/ Disallow: /static/admin/ Disallow: /django-admin/ Disallow: /__debug__/ Disallow: /accounts/ Disallow: /api/ User-agent: Pingdom Crawl-delay: 300 Disallow: /admin/ Disallow: /private/ Disallow: /static/admin/ Disallow: /django-admin/ Disallow: /__debug__/ Disallow: /accounts/ Disallow: /api/ User-agent: StatusCake Crawl-delay: 300 Disallow: /admin/ Disallow: /private/ Disallow: /static/admin/ Disallow: /django-admin/ Disallow: /__debug__/ Disallow: /accounts/ Disallow: /api/ # Default rule for all other bots - block everything User-agent: * Disallow: / # Sitemap location (if available) Sitemap: https://www.pecapecas.com.br/sitemap.xml