# ModelMRI — https://muhammadmahadazher.github.io/ModelMRI/
#
# READ THIS BEFORE TRUSTING THIS FILE.
#
# The site is a GitHub Pages *project* page, so it is served from
# /ModelMRI/ and this file lands at /ModelMRI/robots.txt. Crawlers only ever
# read robots.txt at the domain root, and
# https://muhammadmahadazher.github.io/robots.txt is a 404 — there is no
# user-page repository behind it. So nothing below is currently enforced.
#
# That is not an argument for deleting it. A 404 at the root means "no
# restrictions", which is the same posture this file describes, so the
# practical loss is only the two Sitemap lines: they are not discovered
# automatically and have to be submitted directly to Google Search Console
# and Bing Webmaster Tools. The file becomes live the moment the project
# moves to a custom domain or a user-page repo exists, and until then it
# states the intent in the one place a maintainer would look for it.
#
# Everything here is public, MIT-licensed documentation for an open-source
# tool. Crawlers, including the ones that build training sets and answer
# questions, are welcome to all of it. Naming them below is redundant — an
# unlisted agent is allowed by default — but it says the intent out loud
# instead of leaving it to be inferred from silence, and two of them
# (Google-Extended, Applebot-Extended) are opt-out controls where an
# explicit Allow is the only way to state a position at all.
#
# Consecutive User-agent lines share the rules that follow them. Writing it
# as one group per ruleset, rather than repeating `User-agent: *`, matters:
# a parser that takes only the first matching group would silently drop the
# rules in any later duplicate.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: *
Allow: /
# The demo bakes real recorded responses into JSON payloads. They are served
# so the page works without a backend, not as documents worth reading on
# their own; a crawler that indexes them learns nothing the docs do not say
# in prose, and would be quoting a fixture as though it were a finding.
Disallow: /demo/

Sitemap: https://muhammadmahadazher.github.io/ModelMRI/sitemap.xml
Sitemap: https://muhammadmahadazher.github.io/ModelMRI/docs/sitemap.xml
