User-agent: * Disallow: /tag/ Disallow: /upload/ User-agent: bingbot Disallow: /archives/ Disallow: /upload/ Disallow: /tag/ Disallow: /comment Disallow: /list_comments/ Disallow: /post_comment/ Crawl-delay: 60 User-agent: msnbot Crawl-delay: 60 User-agent: GPTBot Disallow: / User-agent: Applebot Crawl-delay: 30 User-agent: CriteoBot/0.1 Crawl-delay: 30 User-agent: Y!J-ASR/0.1 crawler Crawl-delay: 5 User-agent: SEOkicks-Robot Disallow: / User-agent: SemrushBot Disallow: / User-agent: hypefactors Disallow: / User-agent: Linespider Disallow: /archives/ Disallow: /upload/ Disallow: /comment User-agent: Googlebot Disallow: /archives/ Disallow: /upload/ Disallow: /tag/ Disallow: /comment Disallow: /list_comments/ Disallow: /post_comment/