# neermee.com — robots.txt
#
# The policy, in one line: search crawlers welcome, agents acting for a person
# welcome, training crawlers refused. See notes/Agent Surface — What a Machine
# Gets.md for the reasoning; the short version is that neermee wants to be the
# live source an agent calls, not free material for a model that then answers
# without us and without our dates.
#
# GENERATED from schema/vocabulary/crawlers.json by schema/crawlers.py.
# Do not hand-edit. Add a crawler to the JSON and re-run.
#
# Two mechanisms, deliberately. The Content-Usage rules are the standards-track
# statement of intent (draft-ietf-aipref-attach); the per-agent Disallow blocks
# are what actually works today, because that draft is not yet honored. When it
# is honored the blocks become redundant and can go. The statement of intent
# cannot, because it is the only part that covers a crawler nobody has heard of
# yet.

# ---------------------------------------------------------------------------
# Everyone. Crawl the site; do not train on it.
User-agent: *
Allow: /
Disallow: /api/
Disallow: /search?
Content-Usage: train-ai=n
Content-Usage: search=y

# ---------------------------------------------------------------------------
# Search indexes, including the ones that assemble AI answers. These send people
# back to the record, and the record is dated and sourced, which is the whole
# distribution strategy.
#   OAI-SearchBot — populates ChatGPT search; distinct from GPTBot, which trains
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckDuckBot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amazonbot
Allow: /
Content-Usage: train-ai=n
Content-Usage: search=y

# ---------------------------------------------------------------------------
# Agents fetching a page because a person asked for it right now. This is the
# traffic neermee is being built for. Allowed, including the record JSON, and
# the crawl-delay is deliberately absent: a person is waiting.
#
# Several operators state these may not consult robots.txt at all, since a human
# initiated the request. Allowing them explicitly costs nothing and documents
# the intent for the ones that do look.
#   ChatGPT-User — fires when a person in ChatGPT opens or browses the page
#   Google-CloudVertexBot — grounding fetches made on a customer's request, not training
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Google-CloudVertexBot
Allow: /
Content-Usage: train-ai=n
Content-Usage: search=y

# ---------------------------------------------------------------------------
# Training. Refused.
#
# Not a moral position and not an anti-AI one — the two groups above are the
# customers. It is that a fact of ours absorbed into a model's weights loses the
# two things that make it worth having: its date and its source. A model
# answering "G&G opens at nine" from memory in 2029 is the failure mode this
# whole company exists to fix, and it is worse if the memory came from us.
#   CCBot — not a trainer itself; the corpus most trainers use
#   Meta-ExternalAgent — both casings published; match is meant to be case-insensitive, so send both
#   Diffbot — resells extracted structured data
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Meta-ExternalAgent
User-agent: Diffbot
User-agent: omgili
User-agent: omgilibot
User-agent: ImagesiftBot
Disallow: /
Content-Usage: train-ai=n

# ---------------------------------------------------------------------------
# These fetch nothing. They are the tokens that control what another crawler's
# fetches may be used for, so disallowing them opts out of training while search
# stays allowed above. Blocking the search crawler itself instead would remove
# the site from search entirely, which is the mistake this comment exists to
# stop somebody making later.
#   Google-Extended — fetches nothing. Controls what Googlebot's crawl may be used for, so this opts out of training while search stays allowed
#   Applebot-Extended — same relationship to Applebot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Content-Usage: train-ai=n

# ---------------------------------------------------------------------------
Sitemap: https://neermee.com/sitemap.xml