Skip to content
Boberbot

Getting started

Add Boberbot to an Astro site in one command and check what it wrote.

Install

npx astro add boberbot

This installs the boberbot package and adds it to astro.config.mjs:

import { defineConfig } from "astro/config";
import sitemap from "@astrojs/sitemap";
import boberbot from "boberbot";

export default defineConfig({
  site: "https://example.com",
  integrations: [sitemap(), boberbot()],
});

Set site: robots.txt and llms.txt need absolute URLs. Keep @astrojs/sitemap before Boberbot, so robots.txt can point to the sitemap.

What happens at build time

After astro build, Boberbot:

  • writes robots.txt with a rule for every crawler, Content Signals (by default: search yes, AI answers yes, AI training no) and a Sitemap line, unless public/robots.txt exists;
  • writes llms.txt listing every page with its title and description, unless public/llms.txt exists;
  • checks that every page has a title and a meta description, and that no two pages share a title.

Problems are warnings. Set strict: true to fail the build on them. All options.

Check the result

Deploy, then check the live site on the home page or with the web API. Robots.txt, the sitemap, AI crawler rules and Content Signals give you Level 2. Level 3 needs Markdown served on the same URLs, which depends on your host; Boberbot’s support for it is next.

Version

The integration on npm is boberbot@0.0.1. The check engine behind this site and the CLI come with 0.1.