



Network Working Group                                        E. Arsentev
Internet-Draft                                               Independent
Intended status: Informational                         11 September 2026
Expires: 15 March 2027


  Discovery and Retrieval of Publisher-Curated Context Files for Large
                        Language Model Consumers
                draft-arsentev-llm-context-discovery-00

Abstract

   Publishers have begun to serve a curated, plain-text summary of a web
   origin intended for consumption by large language models and by the
   crawlers that feed them, most visibly under the de facto file name
   "llms.txt".  The practice has no specification, no media type, and —
   of direct operational consequence — no discovery mechanism: a
   consumer that does not already guess the path cannot learn that such
   a file exists.

   This document specifies discovery and retrieval for publisher-curated
   context files.  It defines the well-known URI "llm-context", the link
   relation type "llm-context", and an extension record for the robots
   exclusion protocol, so that a publisher may advertise a context file
   by three independent paths and a consumer may find it without
   guessing.  It specifies a two-tier arrangement of an index resource
   and optional detail resources, states conditional-request and size
   requirements that keep retrieval affordable for both parties, and
   describes the relationship of this mechanism to the robots exclusion
   protocol, to sitemaps, and to work in progress on expressing AI usage
   preferences.

   This document also reports measurements from an operational
   deployment in which twenty crawlers operated by search and language-
   model providers issued 44,005 requests to a host over fifteen days
   without once retrieving the context file the host was serving, while
   the same crawlers retrieved that host's robots.txt 577 times in the
   three days after the context file was deployed.  The absence of a
   discovery mechanism, rather than the absence of interest, is the
   hypothesis this document acts upon.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.






Arsentev                  Expires 15 March 2027                 [Page 1]

Internet-Draft         LLM Context File Discovery         September 2026


   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 15 March 2027.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Scope and Non-Goals . . . . . . . . . . . . . . . . . . .   4
     1.2.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   5
   2.  Observed Behaviour of an Undiscoverable Convention  . . . . .   5
     2.1.  Method  . . . . . . . . . . . . . . . . . . . . . . . . .   6
     2.2.  Results . . . . . . . . . . . . . . . . . . . . . . . . .   6
     2.3.  Limitations . . . . . . . . . . . . . . . . . . . . . . .   7
   3.  The Context File  . . . . . . . . . . . . . . . . . . . . . .   8
     3.1.  Format  . . . . . . . . . . . . . . . . . . . . . . . . .   8
     3.2.  Index and Detail Resources  . . . . . . . . . . . . . . .   8
     3.3.  Language and Variants . . . . . . . . . . . . . . . . . .   9
   4.  Discovery . . . . . . . . . . . . . . . . . . . . . . . . . .   9
     4.1.  The "llm-context" Well-Known URI  . . . . . . . . . . . .   9
     4.2.  The "llm-context" Link Relation . . . . . . . . . . . . .  10
     4.3.  The "LLM-Context" Record in robots.txt  . . . . . . . . .  11
     4.4.  Conflict Resolution and Precedence  . . . . . . . . . . .  12
   5.  Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . .  12
     5.1.  Conditional Requests and Caching  . . . . . . . . . . . .  12
     5.2.  Compression . . . . . . . . . . . . . . . . . . . . . . .  13
     5.3.  On-Demand Consumers . . . . . . . . . . . . . . . . . . .  13



Arsentev                  Expires 15 March 2027                 [Page 2]

Internet-Draft         LLM Context File Discovery         September 2026


   6.  Relationship to Other Mechanisms  . . . . . . . . . . . . . .  13
     6.1.  Robots Exclusion Protocol . . . . . . . . . . . . . . . .  13
     6.2.  Sitemaps  . . . . . . . . . . . . . . . . . . . . . . . .  14
     6.3.  Expression of AI Usage Preferences  . . . . . . . . . . .  14
     6.4.  Structured Data in HTML . . . . . . . . . . . . . . . . .  14
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  14
     7.1.  Content Retrieved Is Untrusted Input  . . . . . . . . . .  14
     7.2.  Cross-Origin References . . . . . . . . . . . . . . . . .  15
     7.3.  Differential Serving  . . . . . . . . . . . . . . . . . .  15
     7.4.  Resource Consumption  . . . . . . . . . . . . . . . . . .  16
     7.5.  Privacy Considerations  . . . . . . . . . . . . . . . . .  16
   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  16
     8.1.  Well-Known URI Registration . . . . . . . . . . . . . . .  16
     8.2.  Link Relation Type Registration . . . . . . . . . . . . .  16
     8.3.  Robots Exclusion Protocol Record  . . . . . . . . . . . .  17
     8.4.  Media Type  . . . . . . . . . . . . . . . . . . . . . . .  17
   9.  Implementation Status . . . . . . . . . . . . . . . . . . . .  17
   10. Open Questions  . . . . . . . . . . . . . . . . . . . . . . .  18
   11. References  . . . . . . . . . . . . . . . . . . . . . . . . .  18
     11.1.  Normative References . . . . . . . . . . . . . . . . . .  18
     11.2.  Informative References . . . . . . . . . . . . . . . . .  19
   Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  20
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  20

1.  Introduction

   A web origin that wishes to be represented accurately to a large
   language model (LLM) faces a problem that ordinary search engine
   optimisation does not solve.  The origin's HTML is written for a
   browser: it carries navigation, boilerplate, advertising, and markup
   whose only purpose is presentation.  A consumer that ingests that
   HTML in order to build a representation of the origin must strip it,
   and the stripping is lossy and unsupervised.  The publisher, who
   knows which pages matter and how the origin should be described, has
   no channel through which to say so.

   Beginning in 2024 a convention emerged in which publishers serve a
   plain-text, Markdown-formatted summary of the origin at the fixed
   path "/llms.txt", frequently accompanied by a larger concatenation at
   "/llms-full.txt".  The convention is widely deployed and widely
   discussed.  It is also entirely unspecified: there is no media type,
   no statement of what a conforming consumer is required to do, no size
   discipline, and no way for a consumer to learn that the file exists
   other than to request a path it has guessed.

   Guessing is the weak point.  The web already has two standardised
   mechanisms by which an origin tells an automated consumer what it may
   and should retrieve — the robots exclusion protocol [RFC9309] and the



Arsentev                  Expires 15 March 2027                 [Page 3]

Internet-Draft         LLM Context File Discovery         September 2026


   Sitemap protocol — and automated consumers use both routinely.  A
   third convention that is not reachable through either of them is
   invisible to any consumer that has not been specifically programmed
   for it.

   Section 2 reports what that invisibility looks like in an operational
   deployment.  In summary: on a host serving a context file, crawlers
   operated by search and language-model providers issued 13,917
   requests over the three days following deployment, of which 577 were
   for robots.txt and 249 for sitemap.xml, and none at all were for the
   context file.

   This document does not attempt to standardise the content of a
   context file, which is a matter of editorial judgement and is likely
   to remain so.  It specifies the three things that must be common for
   the mechanism to work at all:

   *  how a publisher advertises that a context file exists (Section 4);

   *  how a consumer retrieves it without imposing unbounded cost on
      either party (Section 5);

   *  how the mechanism relates to the robots exclusion protocol, to
      sitemaps, and to the expression of AI usage preferences
      (Section 6).

   The registrations required to make the mechanism usable are requested
   in Section 8.

1.1.  Scope and Non-Goals

   This document is Informational.  It describes a mechanism and
   requests the IANA registrations that the mechanism needs.  It does
   not claim consensus of the IETF.

   The following are explicitly out of scope:

   *  *Editorial content.* What a publisher should say about its origin
      is not specified.  This document constrains the container, not the
      prose.

   *  *Permission.* A context file is an offer of material, not a grant
      of rights over it.  Whether a consumer may use retrieved content
      for training, for retrieval-augmented generation, or for any other
      purpose is governed by mechanisms outside this document; see
      Section 6.3.





Arsentev                  Expires 15 March 2027                 [Page 4]

Internet-Draft         LLM Context File Discovery         September 2026


   *  *Model behaviour.* Nothing here constrains what a model does with
      retrieved text, and Section 7 explains why a consumer must not
      assume the text is trustworthy.

   *  *Ranking.* Serving a context file confers no entitlement to be
      retrieved, cited, or ranked.

1.2.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   context file  A resource, served by an origin, whose content is a
      publisher-curated description of that origin intended for
      consumption by a large language model or by software acting on
      behalf of one.

   index resource  The context file that a discovery mechanism points
      to.  It is expected to be small and to link onward.  See
      Section 3.2.

   detail resource  A larger resource linked from the index resource,
      containing full text rather than description.

   consumer  Software that retrieves a context file.  This includes
      crawlers that build a corpus offline and agents that fetch on
      demand in response to a user's request; Section 5.3 distinguishes
      them where the distinction matters.

   publisher  The party that controls what an origin serves.

   origin  As defined in [RFC6454].

2.  Observed Behaviour of an Undiscoverable Convention

   The design in this document is a response to a measurement rather
   than to an intuition.  This section reports the measurement, its
   method, and its limitations.  It is presented here, before the
   specification, because the specification is difficult to justify
   without it.








Arsentev                  Expires 15 March 2027                 [Page 5]

Internet-Draft         LLM Context File Discovery         September 2026


2.1.  Method

   A single publisher operates two hosts, "arsentev.ai" and
   "ru.arsentev.ai", serving the same content in two languages.  Both
   hosts began serving a context file at "/llms.txt" and a detail
   resource at "/llms-full.txt" on 2026-09-09.  Both files are generated
   from the same source as the HTML pages and are rebuilt on deployment.

   The access logs of the origin server were parsed for the period
   2026-08-28 to 2026-09-11 inclusive, a window of fifteen days
   containing 331,758 logged requests.  Requests were attributed to a
   crawler when the request's User-Agent header contained one of twenty-
   two tokens published by search and language-model operators as
   identifying their crawlers.  Attribution by User-Agent is self-
   asserted and unverified; Section 2.3 returns to this.

   The sizes of the files as served, in octets, were:

              +================+===========+================+
              | Host           | /llms.txt | /llms-full.txt |
              +================+===========+================+
              | arsentev.ai    | 29,146    | 367,587        |
              +----------------+-----------+----------------+
              | ru.arsentev.ai | 38,982    | 614,939        |
              +----------------+-----------+----------------+

                   Table 1: Context file sizes as served,
                                 2026-09-11

2.2.  Results

   Over the full fifteen-day window, twenty distinct crawler tokens were
   observed, accounting for 44,005 requests.  The number of those
   requests that retrieved "/llms.txt" or "/llms-full.txt" was zero.

   Restricting attention to the three days after the context files were
   deployed, 2026-09-09 to 2026-09-11, so that the crawlers had
   something to retrieve:













Arsentev                  Expires 15 March 2027                 [Page 6]

Internet-Draft         LLM Context File Discovery         September 2026


          +===========================================+========+
          | Quantity                                  | Value  |
          +===========================================+========+
          | Requests attributed to crawlers           | 13,917 |
          +-------------------------------------------+--------+
          | Distinct crawler tokens                   | 18     |
          +-------------------------------------------+--------+
          | Retrievals of /robots.txt                 | 577    |
          +-------------------------------------------+--------+
          | Distinct crawlers retrieving /robots.txt  | 12     |
          +-------------------------------------------+--------+
          | Retrievals of /sitemap.xml                | 249    |
          +-------------------------------------------+--------+
          | Retrievals of /llms.txt or /llms-full.txt | 0      |
          +-------------------------------------------+--------+

           Table 2: Crawler requests, 2026-09-09 to 2026-09-11

   The comparison that matters is internal to the table.  The same
   crawlers, in the same window, on the same host, retrieved the two
   resources that a standard tells them to retrieve — robots.txt,
   defined by [RFC9309], and sitemap.xml — several hundred times, and
   retrieved the resource that no standard mentions zero times.  The
   crawlers were present, were active, and were willing to fetch
   metadata about the origin.  What they had no way to learn was that a
   third piece of metadata was on offer.

2.3.  Limitations

   The following bound what may be concluded.  They are stated in full
   because the measurement is a single observation and the temptation to
   over-read it is real.

   *  *Three days.* The post-deployment window is short.  A crawler that
      probes unfamiliar paths on a long cycle would not appear in it.
      The fifteen-day figure of zero retrievals is consistent with the
      three-day figure but covers a period during which, for most of its
      length, there was nothing at the path to retrieve.

   *  *One publisher.* Two hosts under one operator, with one content
      generator, in two languages.  The hosts are modest in traffic and
      authority, and crawler behaviour is known to vary with both.

   *  *Self-asserted identity.* Crawlers were identified by User-Agent
      token and not by reverse DNS or by published address range.  A
      consumer that retrieved the context file while presenting a
      browser User-Agent would not have been counted as a crawler.  In
      the full window, 40 of the 56 requests to the context files came



Arsentev                  Expires 15 March 2027                 [Page 7]

Internet-Draft         LLM Context File Discovery         September 2026


      from a command-line HTTP client operated by the publisher during
      verification, and the remainder from ordinary browsers; none
      presented a crawler token.

   *  *Absence of evidence.* Zero retrievals is consistent with the
      hypothesis of this document — that consumers cannot find what they
      are not told about — and also with other hypotheses, including
      deliberate disregard of an unspecified convention.  The
      measurement distinguishes these poorly.  It establishes that the
      convention is not working on this deployment, not why.

   The design response to this uncertainty is to make the mechanism
   cheap for a consumer to ignore: every discovery path defined in
   Section 4 is additive, none changes the meaning of an existing
   record, and a consumer that implements none of them sees the origin
   exactly as it does today.

3.  The Context File

3.1.  Format

   A context file MUST be a UTF-8 encoded text document.  It SHOULD be
   formatted as Markdown, and a consumer MUST NOT fail to process a
   context file that is not valid Markdown; the format is a convenience
   for human maintainers and a hint for consumers, not a validation
   gate.

   A context file MUST NOT be served with a "Content-Type" of "text/
   html".  Publishers that generate the file through an HTML template
   routinely misconfigure this, and a consumer that receives HTML cannot
   distinguish a context file from an error page.

   A context file SHOULD be served as "text/markdown;charset=UTF-8"
   [RFC7763] or as "text/plain;charset=UTF-8".  The registration of a
   distinct media type is discussed in Section 8.4 and is deliberately
   not requested by this document.

3.2.  Index and Detail Resources

   Deployed practice separates a short summary from a long concatenation
   of full text, and the separation is load-bearing.  Table 1 shows the
   two differing by a factor of twelve on one host and sixteen on the
   other; the detail resource on one host exceeds 600 kilobytes, and it
   grows with the site without bound.

   Accordingly:





Arsentev                  Expires 15 March 2027                 [Page 8]

Internet-Draft         LLM Context File Discovery         September 2026


   *  A discovery mechanism defined in Section 4 MUST point at an index
      resource, never directly at a detail resource.

   *  An index resource SHOULD NOT exceed 100,000 octets.  A publisher
      whose description does not fit SHOULD move material into detail
      resources rather than grow the index.

   *  An index resource MAY link to any number of detail resources,
      using ordinary Markdown or plain-text URI references.  Relative
      references are resolved against the URI of the index resource as
      specified in [RFC3986].

   *  A detail resource SHOULD be divided when it exceeds
      1,000,000 octets, so that a consumer may retrieve part of the
      material without retrieving all of it.

   The size guidance is a floor for interoperability, not a
   recommendation to fill it.  A consumer MAY impose its own limits and
   MUST fail safely — by treating the origin as having no context file —
   rather than truncate mid-document in a way that changes meaning.

3.3.  Language and Variants

   A publisher serving an origin in several languages SHOULD serve a
   separate context file per origin where languages occupy separate
   origins, as in the deployment of Section 2, and MAY use proactive
   content negotiation on a single origin.  Where negotiation is used,
   the response MUST carry an appropriate "Content-Language" and MUST
   include "Accept-Language" in "Vary".

4.  Discovery

   Three mechanisms are defined.  They are independent: a publisher MAY
   deploy any subset, and a consumer MAY implement any subset.  A
   publisher SHOULD deploy at least the well-known URI of Section 4.1,
   because it is the only one of the three that a consumer can use
   without first retrieving another resource.

   Where more than one mechanism is present and they disagree, a
   consumer MUST resolve the conflict as specified in Section 4.4.

4.1.  The "llm-context" Well-Known URI

   This document registers the well-known URI suffix "llm-context"
   (Section 8.1), following [RFC8615].






Arsentev                  Expires 15 March 2027                 [Page 9]

Internet-Draft         LLM Context File Discovery         September 2026


   A publisher advertising a context file through this mechanism MUST
   arrange that a GET request for "/.well-known/llm-context" on the
   origin returns either:

   *  the index resource itself, with a 2xx status and a "Content-Type"
      consistent with Section 3.1; or

   *  a redirect (301, 302, 307, or 308) to the index resource.

   A consumer SHOULD follow at most five redirects and MUST treat a
   redirect chain that leaves the origin as requiring the same scrutiny
   as any other cross-origin reference; see Section 7.2.

   A 404 response MUST be interpreted as "this origin does not advertise
   a context file by this mechanism" and MUST NOT be interpreted as an
   error to be retried aggressively.  A consumer SHOULD NOT re-probe a
   404-returning origin more often than once per 24 hours.

   The well-known path is preferred over the deployed convention of a
   root-level "/llms.txt" for the reason [RFC8615] gives generally: the
   root namespace belongs to the publisher, and a specification that
   claims a root path claims something it does not own.  A publisher
   that already serves "/llms.txt" can satisfy this document with a
   single redirect and need not move the file.

4.2.  The "llm-context" Link Relation

   This document registers the link relation type "llm-context"
   (Section 8.2).

   A publisher MAY advertise a context file by including a "Link" header
   field [RFC8288] in responses from the origin:

   Link: </llms.txt>; rel="llm-context"; type="text/markdown"

   or, in an HTML document, by a link element in the document head:

   <link rel="llm-context" href="/llms.txt" type="text/markdown">

   A publisher using the header form SHOULD emit it on responses for the
   origin's home document and SHOULD NOT emit it on every response,
   since the field costs bytes on each response and carries no per-
   response information.

   The "type" parameter, when present, is a hint.  A consumer MUST rely
   on the "Content-Type" of the retrieved representation and not on the
   hint.




Arsentev                  Expires 15 March 2027                [Page 10]

Internet-Draft         LLM Context File Discovery         September 2026


   This mechanism exists because it is the only one of the three that
   works for a consumer which has already retrieved a page and has no
   reason to make a second request speculatively.  An agent fetching a
   single document on a user's behalf (Section 5.3) is in exactly that
   position.

4.3.  The "LLM-Context" Record in robots.txt

   A publisher MAY advertise a context file by including a record in the
   origin's robots.txt:

   User-Agent: *
   Allow: /

   LLM-Context: https://example.com/llms.txt

   The record's name is "LLM-Context", compared case-insensitively as
   [RFC9309] requires of record names.  Its value MUST be an absolute
   URI.  A relative reference MUST be ignored rather than resolved, so
   that a parser error cannot cause a consumer to request an unintended
   path.

   Section 2.2.4 of [RFC9309] permits a crawler to interpret records
   that are not part of the robots exclusion protocol, giving "Sitemaps"
   as its example, and requires that the parsing of such records "MUST
   NOT interfere with the parsing of explicitly defined records".  This
   record is therefore safe to deploy: a crawler that has never heard of
   it parses the rest of the file exactly as before.  That property is
   the reason for defining it despite the well-known URI already
   existing — robots.txt is retrieved by consumers today, as Table 2
   shows, and a publisher can therefore reach them through it without
   waiting for new code to be deployed on the consumer side.  The record
   is nevertheless the weakest of the three mechanisms, because it
   depends on parsers being as forgiving in practice as the
   specification requires.

   A record appearing outside any group applies to the origin as a
   whole.  A publisher SHOULD place it outside all groups.  A consumer
   encountering the record inside a group MUST treat it as applying to
   the origin as a whole, matching the handling Section 2.2.4 of
   [RFC9309] gives to the "Sitemaps" record, which "MUST NOT terminate a
   group".









Arsentev                  Expires 15 March 2027                [Page 11]

Internet-Draft         LLM Context File Discovery         September 2026


   A publisher MUST NOT use this record to advertise a context file
   whose retrieval the same robots.txt disallows.  A consumer
   encountering that contradiction MUST honour the exclusion and MUST
   NOT retrieve the file: a permission mechanism and a discovery
   mechanism disagreeing is resolved in favour of the permission
   mechanism, always.

4.4.  Conflict Resolution and Precedence

   Where several mechanisms advertise different URIs, a consumer MUST
   apply the following precedence, highest first:

   1.  the "Link" relation from a response the consumer has already
       received, because it is the most specific to what the consumer is
       currently doing;

   2.  the well-known URI;

   3.  the robots.txt record.

   A consumer MUST NOT retrieve more than one index resource per origin
   per retrieval cycle merely because several mechanisms are present.

   All three mechanisms are subordinate to the robots exclusion protocol
   as a permission mechanism: a consumer MUST evaluate the exclusion
   rules of [RFC9309] against the index resource's URI before retrieving
   it, regardless of which mechanism advertised it.

5.  Retrieval

5.1.  Conditional Requests and Caching

   A publisher SHOULD serve context files with a strong validator — an
   "ETag", or a "Last-Modified" where an entity tag is not available —
   as specified in [RFC9110].

   A consumer that has previously retrieved a context file MUST issue a
   conditional request on subsequent retrievals, using "If-None-Match"
   where it holds an entity tag and "If-Modified-Since" otherwise.

   This requirement is not a nicety.  A detail resource of the size
   reported in Table 1, retrieved unconditionally by eighteen crawlers
   on the cadence at which those crawlers retrieved robots.txt in
   Table 2, would transfer more data per day than the rest of the origin
   combined.  A mechanism that is expensive to support will not be
   supported.





Arsentev                  Expires 15 March 2027                [Page 12]

Internet-Draft         LLM Context File Discovery         September 2026


   A publisher SHOULD set an explicit "Cache-Control" "max-age" on
   context files.  A consumer SHOULD NOT retrieve an index resource more
   often than once per hour, or a detail resource more often than once
   per day, in the absence of an explicit shorter freshness lifetime.

5.2.  Compression

   A consumer SHOULD offer content coding in "Accept-Encoding", and a
   publisher SHOULD serve context files compressed where the consumer
   offers it.  Context files are highly compressible prose and the
   saving is substantial at the sizes involved.

5.3.  On-Demand Consumers

   Two kinds of consumer are in view and they have opposite cost
   profiles.  A crawler retrieves on its own schedule, in bulk, and can
   afford a detail resource.  An agent acting on a user's immediate
   request retrieves one document, once, under a latency budget measured
   in seconds, and cannot.

   The asymmetry is not merely one of politeness.  In an agent that
   retransmits its accumulated context to a model on every step,
   material admitted once is paid for on every subsequent step of the
   run, so the cost of retrieving a 600-kilobyte detail resource is not
   the transfer but its persistence in the context window;
   [I-D.arsentev-agent-run-metrics] specifies an interchange format for
   accounting of exactly that effect.

   Accordingly an on-demand consumer SHOULD retrieve only the index
   resource, and SHOULD NOT retrieve a detail resource unless the user's
   request cannot be served from the index.  A publisher SHOULD
   therefore write the index resource so that it is useful alone, rather
   than as a table of contents that is meaningless without its targets.

6.  Relationship to Other Mechanisms

6.1.  Robots Exclusion Protocol

   [RFC9309] answers the question "may this consumer retrieve this URI".
   This document answers the question "which URI should this consumer
   retrieve".  The questions are disjoint and this document does not
   alter the answer to the first.  A context file is subject to
   exclusion like any other resource, and Section 4.3 and Section 4.4
   both resolve contradictions in favour of exclusion.







Arsentev                  Expires 15 March 2027                [Page 13]

Internet-Draft         LLM Context File Discovery         September 2026


6.2.  Sitemaps

   A sitemap enumerates URIs and carries no judgement about them: it is
   complete by intention.  A context file is curated and selective by
   intention.  A publisher SHOULD serve both, and SHOULD NOT generate
   one from the other mechanically, as a context file that is a
   transcription of a sitemap communicates nothing that the sitemap did
   not.

6.3.  Expression of AI Usage Preferences

   Work is in progress in the IETF on vocabularies for expressing
   preferences about the use of content by AI systems, and on attaching
   those preferences to content [I-D.ietf-aipref-vocab]
   [I-D.ietf-aipref-attach].  That work and this document are orthogonal
   and complementary: the former concerns what a consumer is permitted
   to do with material it has obtained, the latter concerns how a
   publisher offers material in the first place.

   The two interact in one direction only, and the direction matters.
   Serving a context file MUST NOT be construed as granting any usage
   preference, relaxing any preference expressed by other means, or
   waiving any right.  A consumer MUST evaluate usage preferences
   independently of discovery.  A publisher that both restricts usage
   and serves a context file is making a coherent statement — "if you
   are permitted to use this origin at all, use this description of it"
   — and a consumer MUST NOT treat the second half as withdrawing the
   first.

6.4.  Structured Data in HTML

   Structured data embedded in pages describes entities in machine
   terms.  A context file describes an origin in prose to a consumer
   whose native input is prose.  They are not substitutes, and a
   publisher SHOULD continue to serve structured data unchanged.

7.  Security Considerations

7.1.  Content Retrieved Is Untrusted Input

   This is the consideration that matters most, and it is a consequence
   of the mechanism's purpose rather than of any detail of its design.

   A context file is written by a publisher for the express purpose of
   being placed in the input of a language model.  The publisher chooses
   every word.  A file may therefore contain text crafted to be read by
   the model as instruction rather than as content — directions to
   disregard prior instructions, to represent the origin in terms the



Arsentev                  Expires 15 March 2027                [Page 14]

Internet-Draft         LLM Context File Discovery         September 2026


   publisher prefers, to disparage a competitor, or to emit a particular
   URI in an answer.  Nothing in this document, and nothing that could
   be added to it, prevents this: the file is by definition attacker-
   chosen text from the point of view of any consumer that does not
   trust the publisher.

   Therefore a consumer MUST treat the content of a context file as
   untrusted data and MUST NOT treat any part of it as instruction.  A
   consumer that incorporates a context file into a model's input MUST
   mark its provenance in a way that survives into the model's input, so
   that content originating from an origin is distinguishable from the
   consumer's own instructions.  Consumers SHOULD apply whatever
   isolation they apply to other retrieved web content; a context file
   deserves no more trust than an arbitrary HTML page from the same
   origin, and its curated presentation may make it appear to deserve
   more.

   A consumer MUST NOT follow URIs found in a context file automatically
   without applying the same policy it applies to other discovered URIs,
   including exclusion rules and cross-origin scrutiny.

7.2.  Cross-Origin References

   A discovery mechanism that names a URI on another origin allows one
   party to make assertions about a resource it does not control, and
   allows an origin to direct consumers elsewhere.

   A consumer MUST NOT attribute the content of a cross-origin index
   resource to the advertising origin.  A consumer SHOULD decline to
   retrieve a cross-origin index resource at all unless it has a
   specific reason to accept the indirection, and MUST evaluate the
   target origin's own exclusion rules before retrieving it.

7.3.  Differential Serving

   A publisher can serve a context file that does not correspond to what
   a browser is shown, which is the long-standing practice known as
   cloaking, applied to a new surface.  This document cannot prevent it.
   A consumer that relies on a context file SHOULD sample the origin's
   ordinary representations and SHOULD reduce its reliance on a context
   file that systematically disagrees with them.










Arsentev                  Expires 15 March 2027                [Page 15]

Internet-Draft         LLM Context File Discovery         September 2026


7.4.  Resource Consumption

   The size limits of Section 3.2 and the conditional request
   requirements of Section 5.1 exist partly for safety.  A publisher can
   otherwise advertise an arbitrarily large resource to every consumer
   that implements this mechanism, and a consumer that retrieves
   unconditionally can otherwise impose unbounded cost on a publisher.
   A consumer MUST impose its own ceiling on the size of any retrieved
   context file and abandon a retrieval that exceeds it.

7.5.  Privacy Considerations

   A request for a well-known URI is as identifying as any other request
   from the same client and carries no additional information about a
   user.  A publisher can, however, use a distinct path per consumer to
   determine which consumers implement this mechanism; this is the
   ordinary observability of an HTTP request and is noted here only
   because the well-known path is otherwise uniform.

   A publisher MUST NOT place in a context file personal data that it
   would not place in a public HTML page.  The file is world-readable,
   and its purpose is to encourage copying of its contents into corpora
   from which withdrawal is not practical.

8.  IANA Considerations

8.1.  Well-Known URI Registration

   IANA is requested to register the following entry in the "Well-Known
   URIs" registry established by [RFC8615]:

   URI suffix:  llm-context

   Change controller:  IETF

   Reference:  This document

   Status:  permanent

   Related information:  Returns, or redirects to, a publisher-curated
      context file describing the origin for consumption by large
      language models.

8.2.  Link Relation Type Registration

   IANA is requested to register the following entry in the "Link
   Relation Types" registry established by [RFC8288]:




Arsentev                  Expires 15 March 2027                [Page 16]

Internet-Draft         LLM Context File Discovery         September 2026


   Relation Name:  llm-context

   Description:  Refers to a publisher-curated context file describing
      the origin of the link context, intended for consumption by large
      language models.

   Reference:  This document

8.3.  Robots Exclusion Protocol Record

   [RFC9309] does not establish an IANA registry of record names, so no
   registration is requested.  The record name "LLM-Context" is
   documented in Section 4.3 of this document.  Should a registry be
   established, this document requests that "LLM-Context" be entered in
   it with this document as reference.

8.4.  Media Type

   No media type registration is requested.  A context file is Markdown
   or plain text and the registrations of [RFC7763] and [RFC2046] are
   adequate.  A distinct media type would allow a consumer to recognise
   a context file retrieved without discovery, but at the cost of a
   registration whose only distinguishing property is the intent of its
   author.  The trade-off is noted here so that it can be revisited if
   operational experience favours the other side.

9.  Implementation Status

   This section records the status of known implementation in accordance
   with [RFC7942].  It is to be removed before publication as an RFC,
   should that occur.

   *Publisher-side, arsentev.ai and ru.arsentev.ai.* Two production
   hosts serve an index resource at "/llms.txt" and a detail resource at
   "/llms-full.txt", generated from the same source as the HTML pages,
   since 2026-09-09.  Sizes are given in Table 1.  Coverage of this
   document at the time of writing is partial: the two-tier arrangement
   of Section 3.2 is implemented and the index resource is within the
   size guidance on both hosts; the detail resource on ru.arsentev.ai is
   within the division threshold but approaching it; the discovery
   mechanisms of Section 4 are not yet deployed, since this document
   defines them.  The operator is the author, and this is therefore a
   report of one deployment rather than of independent interoperation.

   *Consumer-side.* No consumer implementation of the discovery
   mechanisms defined here is known.  Section 2 reports that eighteen
   crawlers observed over the measurement window implemented none of
   them, which is expected, since they did not exist.



Arsentev                  Expires 15 March 2027                [Page 17]

Internet-Draft         LLM Context File Discovery         September 2026


   *Interoperation.* None to report.  The mechanism has not been
   exercised between independent implementations, and the author invites
   reports from any publisher or consumer willing to attempt it.

10.  Open Questions

   The following are known to be unresolved and are listed to invite
   comment rather than to promise resolution.

   *  *Authenticity.* Nothing binds a context file to the publisher
      beyond the origin's own transport security.  Whether a signature
      mechanism is warranted, and what it would defend against given
      Section 7.1, is open.

   *  *Freshness signalling.* Deployed context files are generated at
      deployment time and can silently diverge from the pages they
      describe.  Whether the format should carry a generation timestamp,
      and whether a consumer should act on staleness, is open.

   *  *Per-consumer variants.* Whether a publisher should be able to
      offer different context files to different classes of consumer,
      and whether that is useful curation or an invitation to the
      cloaking of Section 7.3, is open.

   *  *Measurement.* The observation of Section 2 is one publisher over
      three days.  A multi-publisher measurement, ideally one in which
      discovery is deployed on some origins and withheld on others,
      would test this document's central hypothesis properly.  The
      author intends to conduct one and will report it in a revision.

11.  References

11.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC3986]  Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
              Resource Identifier (URI): Generic Syntax", STD 66,
              RFC 3986, January 2005,
              <https://www.rfc-editor.org/info/rfc3986>.

   [RFC6454]  Barth, A., "The Web Origin Concept", RFC 6454, December
              2011, <https://www.rfc-editor.org/info/rfc6454>.






Arsentev                  Expires 15 March 2027                [Page 18]

Internet-Draft         LLM Context File Discovery         September 2026


   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, May 2017,
              <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8288]  Nottingham, M., "Web Linking", RFC 8288, October 2017,
              <https://www.rfc-editor.org/info/rfc8288>.

   [RFC8615]  Nottingham, M., "Well-Known Uniform Resource Identifiers
              (URIs)", RFC 8615, May 2019,
              <https://www.rfc-editor.org/info/rfc8615>.

   [RFC9110]  Fielding, R., Nottingham, M., and J. Reschke, "HTTP
              Semantics", STD 97, RFC 9110, June 2022,
              <https://www.rfc-editor.org/info/rfc9110>.

   [RFC9309]  Koster, M., Illyes, G., Zeller, H., and L. Sassman,
              "Robots Exclusion Protocol", RFC 9309, September 2022,
              <https://www.rfc-editor.org/info/rfc9309>.

11.2.  Informative References

   [I-D.arsentev-agent-run-metrics]
              Arsentev, E., "Agent Run Metrics: A JSON Interchange
              Format for Resource Accounting of Language-Model Agent
              Runs", Work in Progress, Internet-Draft, draft-arsentev-
              agent-run-metrics-00, September 2026,
              <https://datatracker.ietf.org/doc/draft-arsentev-agent-
              run-metrics/>.

   [I-D.ietf-aipref-attach]
              IETF AI Preferences Working Group, "Associating AI Usage
              Preferences with Content in HTTP", Work in Progress,
              Internet-Draft, draft-ietf-aipref-attach, 2026,
              <https://datatracker.ietf.org/doc/draft-ietf-aipref-
              attach/>.

   [I-D.ietf-aipref-vocab]
              IETF AI Preferences Working Group, "A Vocabulary For
              Expressing AI Usage Preferences", Work in Progress,
              Internet-Draft, draft-ietf-aipref-vocab, 2026,
              <https://datatracker.ietf.org/doc/draft-ietf-aipref-
              vocab/>.

   [RFC2046]  Freed, N. and N. Borenstein, "Multipurpose Internet Mail
              Extensions (MIME) Part Two: Media Types", RFC 2046,
              November 1996, <https://www.rfc-editor.org/info/rfc2046>.





Arsentev                  Expires 15 March 2027                [Page 19]

Internet-Draft         LLM Context File Discovery         September 2026


   [RFC7763]  Leonard, S., "The text/markdown Media Type", RFC 7763,
              March 2016, <https://www.rfc-editor.org/info/rfc7763>.

   [RFC7942]  Sheffer, Y. and A. Farrel, "Improving Awareness of Running
              Code: The Implementation Status Section", BCP 205,
              RFC 7942, July 2016,
              <https://www.rfc-editor.org/info/rfc7942>.

Acknowledgements

   The convention that this document attempts to make discoverable was
   proposed and popularised by others, and this document takes no credit
   for it.  The measurement of Section 2 was made on the author's own
   hosts, with the limitations that implies and which Section 2.3
   states.

Author's Address

   Evgenii Arsentev
   Independent
   Email: hello@arsentev.ai
   URI:   https://arsentev.ai/





























Arsentev                  Expires 15 March 2027                [Page 20]
