Skip to content
Boberbot

Boberbot's checker

What Boberbot fetches when someone checks a website, how to recognise it, and how to stop it.

Boberbot checks websites against Cloudflare’s published agent-readiness rules. It runs only when someone asks it to check a specific site. It doesn’t crawl, follow links or keep copies of what it reads.

How to recognise it

Its requests carry this user agent, with the version of the release in place of 0.1.0:

Boberbot/0.1.0 (+https://boberbot.com/bot)

Checks run in two places. With the command-line tool, Boberbot runs on the computer of whoever uses it, so requests come from their network and we see nothing. With the checker on boberbot.com, requests come from Cloudflare’s network on our behalf; results are cached for ten minutes, and each visitor can only start a few checks a minute.

What one check fetches

About 20 to 30 requests to the site, once, within a few seconds:

  • the page that was asked for, as HTML and as Markdown (Accept: text/markdown)
  • /robots.txt, the sitemaps it names, and otherwise /sitemap.xml and its usual alternatives
  • discovery files under /.well-known/: the API catalog, MCP server card, A2A agent card, agent skills index, OAuth metadata, Web Bot Auth directory, AI catalog and the commerce profiles
  • /auth.md and /openapi.json

Plus a few DNS lookups for records under _agents. on your domain, through Cloudflare’s public DNS-over-HTTPS resolver.

How to stop it

Boberbot isn’t a crawler, so crawler rules in robots.txt don’t apply to it. It reads your robots.txt because that’s one of the checks. To refuse it, block requests whose user agent contains Boberbot at your firewall, or with a custom rule in Cloudflare’s WAF.

Questions or problems

Email info@boberbot.com.