这个网站(https://catstxt.org/logs)有日志:

但我请求后发现页面没有显示耶 👇

curl -L -A "111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" https://catstxt.org/cats.txt

我怀疑那些日志是虚假的,也可能是采样。多请求看看是不是采样:

while true; do curl -L -A "111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" https://catstxt.org/cats.txt -o /dev/null -s -w "请求时间: %{time_connect}s | 状态码: %{http_code}\n"; sleep 1; done

页面一直没有显示,我以为 UA 太奇怪,换成:

while true; do curl -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/15555555555555555555 Safari/537.36" https://catstxt.org -o /dev/null -s -w "请求时间: %{time_connect}s | 状态码: %{http_code}\n"; sleep 0.5; done

结果还是没有!!

我开始怀疑实时效果是虚假的,于是打开控制台先找 socket 和 fetch,结果都没有,再去看 JS 发现果然是伪造的:

const uas = "-\ncurl/7.88.1\nGo-http-client/1.1\nGoogleOther\nGoogleOther\nGoogleOther\nGoogleOther\nHello from Palo Alto Networks, find out more about our scans in https://docs-cortex.paloaltonetworks.com/r/1/Cortex-Xpanse/Scanning-activity\nmeta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)\nmeta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)\nMozilla/4.0 (compatible; MSIE 9.0; Windows NT 10.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729)\nMozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)\nMozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)\nMozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)\nMozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)\nMozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)\nMozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)\nMozilla/5.0 (compatible; GenomeCrawlerd/1.0; +https://www.nokia.com/genomecrawler)'\nMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\nMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\nMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\nMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\nMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\nMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\nMozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)\nMozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)\nMozilla/5.0 (compatible; MJ12bot/v2.0.4; http://mj12bot.com/)\nMozilla/5.0 (compatible; MJ12bot/v2.0.4; http://mj12bot.com/)\nMozilla/5.0 (compatible; MJ12bot/v2.0.4; http://mj12bot.com/)\nMozilla/5.0 (compatible; MJ12bot/v2.0.4; http://mj12bot.com/)\nMozilla/5.0 (compatible; ModatScanner/1.1; +https://modat.io/)\nMozilla/5.0 (compatible; Odin; https://docs.getodin.com/)\nMozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)\nMozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)\nMozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)\nMozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)\nMozilla/5.0 (compatible; wpbot/1.3; +https://forms.gle/ajBaxygz9jSR8p8G9)\nMozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)\nMozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)\nMozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)\nMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.154 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36(KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0 Safari/537.36\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36\nMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)\nMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:114.0) Gecko/20100101 Firefox/114.0\nMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36\nMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36\nMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36\nMozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)\nMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.3\nMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google-PageRenderer Google (+https://developers.google.com/+/web/snippet/)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)\nMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot\nMozilla/5.0 zgrab/0.x\nnvdorz\npython-httpx/0.28.1\npython-requests/2.32.4\nPython/3.10 aiohttp/3.12.13\nrobots-ai-permissions/2.0 (+process_splits_async.py)\n-".split("\n");
function toLogFormat(e) {
    const o = e => String(e).padStart(2, "0")
      , t = -e.getTimezoneOffset()
      , l = `${t >= 0 ? "+" : "-"}${o(Math.floor(Math.abs(t) / 60))}${o(Math.abs(t) % 60)}`;
    return `[${o(e.getDate())}/${["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"][e.getMonth()]}/${e.getFullYear()}:${o(e.getHours())}:${o(e.getMinutes())}:${o(e.getSeconds())} ${l}]`
}
function makeLogEntry(e, o, t, l, i) {
    return `[redacted] - - ${toLogFormat(e)} "GET /${o} HTTP/1.1" ${i} ${l} "-" "${t}"`
}
function aNewHope(e) {
    const o = uas[Math.floor(Math.random() * uas.length)]
      , t = Math.random() > .3 ? ".well-known/cats.txt" : Math.random() > .4 ? "cats.txt" : "llms.txt"
      , l = ".well-known/cats.txt" != t ? 301 : 200;
    return makeLogEntry(e, t, o, Math.floor(200 * Math.random() + 500), l)
}
function monitorForNewLogs() {
    const e = document.getElementById("logs")
      , o = aNewHope(new Date);
    e.textContent = o + "\n" + e.textContent;
    const t = 3e4 * Math.random() + 200;
    setTimeout(monitorForNewLogs, t)
}
function loadHistoricalLogs() {
    const e = document.getElementById("logs");
    let o = []
      , t = new Date;
    const l = new Date;
    for (l.setDate(l.getDate() - 3); t > l; ) {
        o.push(aNewHope(t));
        const e = 120 * Math.random() + 10
          , l = Math.floor(1e3 * e);
        t.setTime(t.getTime() - l)
    }
    e.textContent = o.join("\n")
}
function streamServerConnection(e) {
    const o = document.getElementById("spinner");
    let t = 0;
    setInterval(( () => {
        let e = 9 - Math.abs(t++ % 18 - 9);
        o.textContent = ".".repeat(e) + ":" + ".".repeat(9 - e)
    }
    ), 120)
}

起因是我想起以前说的 llms.txt。之前,我检查 1 亿日志后发现 0 请求,于是我认为 llms.txt 不值得写。刚刚我想看看现在情况有没有变化,结果几千万请求仍然只有几次,还分别是几条被 blocked 的机房 IP,以及一个 llms.txt 研究究机构 LLMS-Txt-Scanner/1.0 (Research)。

所以,也许目前 llms.txt 仍然不值得写。但是,搜索 Google 后发现某些文章里提到讽刺 llms.txt 的 cats.txt,就打开网站看看...