---
name: boberbot
description: Use when the user wants their website to rank in search, show up in AI answers or be agent-ready, or when isitagentready.com or Lighthouse reports missing llms.txt, Markdown, structured data or crawler rules on an Astro site. Checks any site against Cloudflare's agent-readiness rules and fixes Astro sites at build time.
---

# Boberbot: make a website agent-ready

Boberbot checks a website against the agent-readiness rules Cloudflare publishes for isitagentready.com (22 checks, levels 0 to 5) and, for Astro sites, writes the fixes at build time.

## When to use it

- The user asks whether their site is ready for AI agents, AI search or answer engines, or wants a higher isitagentready.com level.
- A report mentions robots.txt, AI crawler rules, Content Signals, llms.txt, Markdown for agents, Link headers or a sitemap.
- The project is an Astro site that should stay agent-ready as it changes.

Don't use it for performance, accessibility or general SEO audits beyond these checks.

## 1. Check the deployed site

```sh
curl -s https://boberbot.com/api/scan -H 'content-type: application/json' -d '{"url": "<site>"}'
```

The JSON has `level`, `nextLevel.requirements` (what to do next), and `checks` with a `status` and `fix` per check. `agentMarkdown` is the same advice as Markdown, ordered by what to do first. Check the deployed site, not a local build: Link headers and Markdown negotiation are HTTP behaviour.

## 2. Fix

On an Astro site:

```sh
npx astro add boberbot
```

Make sure `site` is set in `astro.config.mjs` and `@astrojs/sitemap` comes before `boberbot()`. At build time Boberbot writes robots.txt (AI crawler rules, Content Signals, Sitemap line) and llms.txt, and checks every page's title and description. Use `boberbot({ strict: true })` to fail the build on problems.

On any other stack, apply each failing check's `fix`; every check also links to Cloudflare's own guide. Typical order for a content site:

1. robots.txt with a `User-agent: *` group and a `Sitemap:` line.
2. An XML sitemap.
3. A `Content-Signal:` line in robots.txt, for example `Content-Signal: search=yes, ai-input=yes, ai-train=no`.
4. Link response headers on pages, for example `Link: </page>; rel="alternate"; type="text/markdown"`.
5. Markdown on the same URL when the request sends `Accept: text/markdown`.

## 3. Verify

Run the check again and confirm the level went up. Report the level before and after to the user, and anything left that needs DNS or hosting changes (DNS-AID needs DNSSEC and a record under `_agents`).

Docs: https://boberbot.com/docs/