# The Margin — robots.txt
# AI crawlers and agents are explicitly WELCOME on public pages: The Margin is
# an AI-native product and wants to be understood by LLMs.
# Machine-readable product docs: /llms.txt and /llms-full.txt (llmstxt.org).
# Field notes feed: /stories/feed.xml
#
# ONE GROUP ON PURPOSE. A crawler obeys only the most specific group that
# names it, so the old separate "User-agent: GPTBot / Allow: /" groups gave
# every named AI crawler an empty rule set: none of the Disallow lines below
# applied to them. Listing every agent at the head of the same group gives
# them all the same rules, which is what "same rules as everyone" meant.
# The app paths are behind sign-in anyway; these lines stop crawlers spending
# their budget on login redirects.
#
# Content-Signal (contentsignals.org) says what may be done with what a
# crawler fetches: build a search index, feed an AI answer, train a model.
# Yes to all three, decided 2026-09-27. It sits inside the one group so every
# agent above gets it, and it only reaches the pages that group allows: the
# Disallow lines still keep the app paths out entirely.
#
# Agentmap points at the ARD manifest (agenticresourcediscovery.org): the MCP
# server, the no-account agent sandbox, the A2A card, skills and the API
# catalog in one file. Like Sitemap it belongs to no group.

User-agent: *
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Claude-Web
User-agent: anthropic-ai
User-agent: Amazonbot
User-agent: Bytespider
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /api/
Disallow: /settings/
Disallow: /join/
Disallow: /invite/
Disallow: /share
Disallow: /l/
Disallow: /oauth/
Disallow: /verify
Disallow: /auth-error
Disallow: /dashboard
Disallow: /boards
Disallow: /notes
Disallow: /habits
Disallow: /expenses
Disallow: /whiteboards
Disallow: /planner
Disallow: /calendar
Disallow: /inbox
Disallow: /focus
Disallow: /family
Disallow: /team
Disallow: /templates
Disallow: /search
Disallow: /checklists
Disallow: /today
Disallow: /agents
Disallow: /game
Disallow: /admin
Disallow: /chat
Disallow: /intelligence
Disallow: /journal
Disallow: /meals
Disallow: /mind
Disallow: /organize
Disallow: /pacts
Disallow: /people
Disallow: /recipes

Sitemap: https://themarginapp.com/sitemap.xml
Agentmap: https://themarginapp.com/.well-known/ard.json