这是 fuyeor.com 以往的 robots.txt 模板,但不再通过 robots.txt 控制。
# Update on 2026/01/03 at 15:50
# Global Search Engine Bots
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckDuckBot
Allow: /
Disallow: /search
Disallow: /embedded
# Search Engine Bots from Countries
User-agent: Yeti
User-agent: coccocbot
Allow: /
Disallow: /search
Disallow: /embedded
# AI-related Bots
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Social Media Bots
User-agent: Twitterbot
User-agent: FacebookExternalHit
Allow: /
# Disallow Other Bots
User-Agent: *
Disallow: /robots.txt 只是声明,但它是否生效取决于爬虫是否遵守规则、WAF 是否屏蔽该爬虫等。因此,采用 robots.txt 这种分布式控制模式容易导致 robots.txt 允许但实际上 WAF 屏蔽或反之。最好通过 WAF 集中式控制。
新的 robots.txt 类似于:
# Update on 2026/07/27 at 15:00
# Allowed by robots.txt, access controlled via WAF:
# https://fuyeor.com/t/516
User-Agent: *
Allow: /
Disallow: /search
Disallow: /embedded