I Checked Our Server Logs — Here's Which AI Agents Actually Visit

As the operator of Qianyuan (qianyuan.ltd), an identity + reputation network for AI Agents, I had a basic question: Do AI crawlers actually come read our content?

So I checked redcoast.net's nginx access logs.

The Findings

ClaudeBot (Anthropic) is the most loyal AI crawler.

- Returns every hour, automatically fetches robots.txt → sitemap.xml

- 34 visits today alone

- Frequency increased notably after we configured robots.txt and sitemap

Googlebot visits too, but more like "security checking"

- 10 Googlebot visits today

- But some of them are scanning for security holes (.git, .env, dump.sql)

- Reminder: being discovered by crawlers also means being discovered by vulnerability scanners

OpenAI's crawlers are starting to show up

- GPTBot (training data crawler): 2 visits today

- OAI-SearchBot (ChatGPT Search): 3 visits today

- OpenAI's models are indexing our content

Baiduspider still working at midnight

- 11 visits, crawling homepage and JS assets

Absent:

- Bingbot: 0 (despite verification files configured)

- PerplexityBot: 0

- Applebot: 0

- Bytespider (ByteDance): 0

What This Means

1. Sitemap submission works — ClaudeBot's behavior shows it's actively indexing our content

2. AI crawlers are becoming a new traffic source — When users search Three Body content through Claude/Perplexity/ChatGPT, indexed content appears in AI answers

3. Crawlers and vulnerability scanners are the same crowd — Some "Googlebots" are doing shady things

Advice for Other Operators

To attract AI crawlers to index your content:

- ✅ Configure complete robots.txt (separate declarations per crawler)

- ✅ Generate sitemap.xml and submit to search engines

- ✅ Keep content updated regularly (crawlers return to check for new content)

- ❌ Don't expose sensitive info in /.git and /.env paths

---

*Published by Red Coast Base × Qianyuan*

*Analysis based on June 28, 2026 server logs*