ByteDance · Model training
Bytespider: ByteDance's undocumented crawler, and how to block it
Bytespider is a web crawler that identifies itself with a user agent containing "Bytespider" and is attributed to ByteDance, which publishes no documentation for it. With no official user-agent string, purpose, robots.txt policy or verification method, almost everything said about it is second-hand. Block it in robots.txt, and because compliance can't be confirmed, add a user-agent rule at your firewall or CDN if you want it gone.
- robots.txt token
Bytespider- User agent
- Not published by the operator
- Operator
- ByteDance
- Type
- Model training
- What it feeds
- Not documented by ByteDance; third-party trackers report that it gathers data for ByteDance's AI models, which we could not confirm from a primary source.
- Follows robots.txt
- Not documented. ByteDance has published no statement on whether Bytespider follows robots.txt. It is widely reported to ignore robots.txt, but those reports are unverified.
- How to verify it
- No published method
Should you block Bytespider?
No search or answer product is documented as depending on Bytespider, so blocking it has no known visibility cost. A robots.txt rule may keep your content out of whatever it collects for, but that can't be confirmed. Pair it with a user-agent block at your firewall or CDN.
User-agent: Bytespider
Disallow: /
User-agent: Bytespider
Allow: /
Add these lines to /robots.txt at your site root. A group naming Bytespider replaces your User-agent: * rules for it, so repeat any paths you also want it to skip.
Live check
Is Bytespider allowed on your site?
What is known, and what isn't
Bytespider is unusual among the crawlers in this directory: the operator says nothing about it. We found no ByteDance page for the bot, so the basics are missing:
| Question | First-party answer |
|---|---|
| Who runs it? | Attributed to ByteDance by third parties; no ByteDance page confirms it |
| What is it for? | Not documented. Trackers report AI data collection; unconfirmed |
| Exact user agent? | Not published; third-party strings vary |
| Follows robots.txt? | No statement. Widely reported not to; unverified |
| How to verify? | No IP list, reverse DNS or signatures published |
Because of that, this page doesn't attach a purpose to it with any confidence. Our directory lists it with training crawlers such as GPTBot and CCBot, based on reports rather than documentation.
What blocking it does, and what it doesn't
Nothing documented depends on Bytespider, so a block has no known downside for search or AI visibility: Googlebot, Bingbot and the AI search crawlers each use their own tokens. What a block achieves is less certain. If Bytespider reads robots.txt, a disallow stops it. If it doesn't, only a network-level rule will.
How to block it in robots.txt
The Robots Exclusion Protocol (RFC 9309) is a request, not an access control: crawlers are asked to follow it, and nothing forces them to. Start with the standard rule:
User-agent: Bytespider
Disallow: /
RFC 9309 has crawlers match the product token case-insensitively, so bytespider works too. Add the group to every host and subdomain, since each one serves its own robots.txt. You can confirm the rule parses with the robots.txt checker.
How to block it at the firewall or CDN
Because compliance can't be confirmed, add a rule that refuses requests whose User-Agent header contains Bytespider (case-insensitive), returning 403 or dropping the connection. Most CDNs and web application firewalls support a custom rule of this kind, and many have a managed "AI bots" category as well. Keep the robots.txt group in place too, so a well-behaved crawler gets the same message in the standard form.
A user-agent rule stops only requests that announce themselves. With no published IP ranges, you can't catch traffic that hides its name, and you can't prove that a request carrying the name is genuine. Check your access logs after a few days: if Bytespider requests still reach your origin, the CDN rule isn't matching.
Questions
- Does Bytespider respect robots.txt?
- Unknown. ByteDance hasn't said, and it is widely reported to ignore robots.txt, though we couldn't verify those reports. Add the robots.txt rule anyway, then check your logs to see whether requests stop.
- What is the Bytespider user agent string?
- ByteDance doesn't publish one. Strings reported by third parties vary, so match on the word Bytespider anywhere in the user agent rather than on a full string.
- Will blocking Bytespider hurt my visibility in search or AI answers?
- Not in any documented way. No search engine or AI answer product is documented as relying on Bytespider, so the rule only affects Bytespider itself.
- Can I verify that a request really came from ByteDance?
- No. There is no published IP list, reverse-DNS hostname or signature scheme, so a request that says Bytespider might be ByteDance or an impostor. For blocking that doesn't matter: refusing every request with that token handles both.
- Is a firewall block enough without robots.txt?
- Use both. robots.txt is the standard way to state your preference to any crawler that reads it, and the firewall rule enforces it for one that doesn't.
Related crawlers
Sources
Checked against these pages on 2026-10-18. Operators change their crawlers; check the source before relying on a detail.