Metadata-Version: 2.4
Name: fromwhere
Version: 0.0.1a1
Summary: Know where it came from: artifact provenance tracking.
Author-email: Pete Bachant <petebachant@gmail.com>
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: c2pa-python>=0.37.2
Requires-Dist: cryptography>=49.0.0
Description-Content-Type: text/markdown

# `fromwhere`

`fromwhere` provides artifact provenance tracking, including support for
composite artifacts like figure PNGs, publication PDFs,
static notebook and website HTML, machine learning models, and more,
which can be traced all
the way back to their primary inputs, e.g., scripts or images generated
with AI tools.
This helps authors and readers know exactly how an output was created,
so they can assess if it's trustworthy.

`fromwhere` outputs are signed and contain Content Credentials metadata.
All of a project's artifact provenance information lives inside a
`provenance.toml` file, which includes a comment at the top to prevent AI agents
from tampering with the provenance information, which may indicate
falsified evidence in scientific projects.

## Installation

```sh
uv tool install fromwhere
```

The distribution, the import name, and the command it installs are all
`fromwhere`.

## Usage

If you have a script that generates a figure, run it through the `fromwhere` CLI:

```sh
fromwhere run -i data/raw.csv -o figures/plot.png -- uv run plot.py
```

The command (what comes after `--`) must be run with an environment manager
that uses lock files so those can be tracked as part of the provenance,
since they are important input information.
Practically this means that the command must start with one of the following:

- `uv run` (as long as a `uv.lock` file will be created)
- `pixi run`
- `bun run`
- `cargo run`
- `nix develop --command` (as long as a `flake.nix` file is present)
- `calkit xenv` (use for Docker, Conda, Julia, renv environments)
- `calkit nb exec`
- `calkit latex build`

### Declaring primary artifacts

Every chain ends somewhere. Follow a figure back far enough and you reach a
file `fromwhere` did not make: measurements typed into a CSV, a dataset someone
downloaded, a plotting script. Left alone those sit at the bottom of the chain
unexplained, and `fromwhere status` warns about them, because every automated
check above them passes — which is exactly what makes the gap easy to miss.

Declare them with:

```sh
fromwhere declare <path> --mine
```

This records your "attestation". AI agents should similarly declare
when they've created primary artifacts so readers can assess if that
is an acceptable use of generative AI — but an agent cannot be the one
answering for the file, so the tool is disclosed _beside_ a person rather
than in place of one:

```sh
fromwhere declare <path> --mine --with-ai 'Claude Opus 5'
fromwhere declare <path> --author 'A Researcher' --with-ai 'Claude Opus 5'
```

Using `--with-ai` without naming a person is refused.

Both flags repeat, because an artifact rarely has exactly one author and code
almost never does — a script grows through several hands and, increasingly,
several models:

```sh
fromwhere declare <path> --author 'A Researcher' --author 'A Colleague' \
    --with-ai 'Claude Opus 5' --with-ai 'GitHub Copilot'
```

Git already knows most of this, so it can be read out instead of retyped.
Every commit touching the file names an author, and `Co-authored-by:` trailers
name everyone else — which is exactly where an agent's own signature lands:

```sh
fromwhere declare <path> --from-git-history
```

Whether an author is a person or a tool is a guess when it comes from git, so
`fromwhere declare` prints what it decided and `--author`/`--with-ai` override it.

Hand-authored artifacts have authors too. A `.drawio` canvas is assembled from
recorded panels but arranged by people and agents, so it carries a derivation
chain _and_ an author list; declaring it adds the authors without disturbing
its inputs.

An attestation is the weakest thing in the record: nothing verifies it, and
fromwhere says so wherever it is displayed. When the file came from somewhere a
reader could actually fetch, say that instead:

```sh
fromwhere declare <path> --doi 10.5281/zenodo.1234567
fromwhere declare <path> --git github.com/myuser/myproject/path/to/data.csv@{git_rev}
fromwhere declare <path> --calkit calkit.io/myuser/myproject/path/to/data.csv@{git_rev}
```

The revision is required. `github.com/myuser/myproject/data.csv` names whatever
is at that path today — a mutable claim wearing the costume of a citation.
The `--calkit` form exists because Calkit tracks large files with DVC, so a
path in a Calkit project can name data that is not in the git tree at all.

Scripts count too. A figure resting on a script nobody will claim is as
unplaceable as one resting on data nobody collected, and a script a model wrote
is exactly what a reader needs told.

A declaration is about _people_, not bytes, so editing a declared file does not
invalidate it and never needs redoing. `fromwhere declare` records a hash too,
but only as a note of what was seen — nothing checks it. `fromwhere run` keeps it
current whenever it uses the file, so the record does not drift, and says so
rather than doing it quietly:

```
recorded figures/cp_curve.png (signed)
   in   data/performance.csv
   in   scripts/plot_cp.py
   refreshed hash for scripts/plot_cp.py (declared; authorship unchanged)
```

Only declarations are eligible. A hash fromwhere wrote itself is evidence, and
rewriting that is the tampering the record exists to catch — so an artifact
with a command keeps reporting `modified` no matter how many later runs
consume it.

### Downloads

Some data just lives at a URL, and `--url` is the one origin fromwhere **checks
while making it**:

```sh
fromwhere declare data/raw.csv --url https://example.org/datasets/raw.csv
```

fromwhere fetches the address. If the file is not there yet it is downloaded; if
it is, fromwhere downloads it anyway and compares — and refuses if the bytes
differ, naming both hashes, because either the file was edited after it was
downloaded or the address has moved on and both are things you want told:

```
https://example.org/datasets/raw.csv does not serve what is in raw.csv.
   on disk:    sha256:5179d9e0…
   downloaded: sha256:4fd12f8c…
```

What lands in the record is the address **and the moment**:

```toml
origin_kind = "url"
origin = "https://example.org/datasets/raw.csv"
origin_fetched = "2026-08-06T02:49:24+00:00"
```

That pair is the claim: _this address served exactly these bytes, at this
time_. It sits between the other two kinds and is honest about being there — a
DOI resolves to a deposit that will not change, an attestation cannot be checked
by anyone, and this was genuinely fetched and genuinely verified but names
something mutable. So the panel shows the date beside the link every time, and
says the URL can serve something else later. Being told a file came from a URL
is an attestation; going and looking is evidence, and the timestamp is what
keeps that distinction meaningful a year on.

Downloading uses nothing but the standard library. A provenance tool that pulled
in an HTTP stack to fetch a CSV would have made itself harder to trust than the
claim it records.

### Where AI disclosure lives

A file that carries Content Credentials already says whether it is
machine-generated, signed by whoever made it — the example's AI-generated
figure really does carry Google's own "Created by Google Generative AI". There
is no need to restate that in `provenance.toml`, and fromwhere reads it from the
file instead.

But a manifest is fragile: any tool that re-encodes the bytes silently discards
it, which is the whole reason `fromwhere drawio import` exists. And most formats
cannot carry one at all — a `.py`, a `.csv`, a PDF, or a notebook's HTML have
nowhere to put it. So the record holds the disclosure whenever the file cannot,
and where both exist fromwhere compares them and **reports the disagreement**
rather than quietly picking a winner. A disclosure that evaporates the first
time somebody opens the image in an editor is exactly the failure worth seeing.

### What can be signed

Reading credentials and writing them are different questions, and the second
set is smaller. Verified against c2pa-python 0.37.4 (c2pa-rs 0.90.4), fromwhere
can **embed** a manifest in PNG, JPEG, GIF, SVG, TIFF, WebP and WAV.

**PDF is the case worth knowing about.** The C2PA specification covers PDF, and
other tools do sign them — but c2pa-rs cannot yet write one, though it reads
them perfectly well. So a PDF signed elsewhere will have its credentials read
and displayed by fromwhere; a PDF fromwhere produces is recorded but unsigned. That
is a limitation of the library, not of the format, and it should lift on its
own.

HTML is not applicable: c2pa does not recognize the type at all, in either
direction. There is nowhere in an HTML page for a manifest to live.

This matters for whole-document outputs — a built paper is exactly the artifact
you would want to hand someone with its provenance attached. So for a MyST PDF,
for `calkit latex build`, and for the HTML a notebook renders to, fromwhere
records the artifact in `provenance.toml`, skips signing rather than failing the
build, and says which it did:

```
recorded paper.pdf (not signed: .pdf cannot carry Content Credentials;
                    the record in provenance.toml is its only provenance)
```

For those outputs the record is the only provenance there is, which is why the
warning at the top of `provenance.toml` is not decoration.

### Freshness checking

To see if a given output's inputs (including environment lock files) have
changed, rendering it stale, use the `status` command.

```sh
fromwhere status <path>
```

The states are kept apart, because they have different fixes:

- **stale** — an input changed; regenerate it.
- **modified** — a _produced_ output changed without going through fromwhere, so
  the record no longer describes the file it names. This is how tampering with
  an output after generation is caught.
- **upstream** — sound in every direct link, but resting on one that is not.

A **declared** artifact is never checked against its own hash, and this is the
rule worth stating plainly, because everything else follows from it:

> A declaration says who is answerable for a file, not what it contained on
> some particular afternoon. What matters about an edit is whether it reached
> an output — and that is already recorded, because every output carries the
> hash of each input as it was when the output was made, recomputed whenever
> `fromwhere run` regenerates it.

So editing a script, a dataset, or a document source is not a finding and never
requires re-declaring anything. The consequence shows up where it can be acted
on: on the outputs built from the old bytes. The honest cost is that a declared
file nothing consumes — no output to go stale — is not watched at all; if it is
swapped, nothing notices.

- **upstream** — this artifact is sound in every direct link but rests on one
  that is not. Staleness does not stop at the first link: a composite whose
  `.drawio` is untouched passes every direct check even when the data three
  steps back was edited, because nothing regenerated the panel in between.

None of them is repaired by editing `provenance.toml`, which is the one thing a
reader in a hurry might try.

### Rebuilding

Every artifact records the command that made it and the inputs it was made
from, so the repair sequence is a property of the record rather than something
you have to reconstruct:

```sh
fromwhere rebuild             # everything that is out of date
fromwhere rebuild <path>      # that artifact, and everything behind it
fromwhere rebuild --dry-run   # say what would happen, in order
```

Dependencies first, because rebuilding a document before the figure it embeds
accomplishes nothing and you would end up running everything twice. Each
command is repeated in-process rather than through a shell, so nothing depends
on quoting the record was written to avoid.

Nothing is rebuilt that does not need it — an artifact whose inputs still hash
to what the record says keeps its bytes and its signature. Anything that cannot
be rebuilt is named rather than passed over: nobody can regenerate raw data,
and a hand-arranged diagram is refreshed by the export that consumes it.

### Signing certificates

The `fromwhere run` command includes the `--cert` option to provide a signing
certificate for the Content Credentials metadata.

### draw.io

`fromwhere` enables composite figure provenance while retaining interactive
editing with draw.io.
To import a PNG into a draw.io diagram, run:

```sh
fromwhere drawio import my-figure.png my-composite-figure.drawio
```

It's important to use the `fromwhere` CLI since this will embed metadata into
the `.drawio` file.

`import` is an _authoring_ step: it puts a new panel on the canvas. You should
not need to run it again when a figure is redrawn, and you don't — each shape
records the `src` it came from and the hash it had, so the export below
re-embeds any panel that has moved on, keeping the position and size you gave
it, and says which ones it touched. Re-running `import` on a panel already
present is still safe: it replaces rather than adding a second copy.

To export an SVG containing the provenance information, run:

```sh
fromwhere drawio export my-diagram.drawio my-diagram.svg
```

Export is the build step. It refreshes stale panels first — draw.io renders the
_copy_ inside the diagram, so refreshing afterwards would publish the old
pictures — then renders, signs the result with every panel as an ingredient,
and re-records the diagram itself.

#### A panel that went in through the GUI

Sometimes a figure is already on the canvas: dragged in, pasted, or inserted
with draw.io's own **Insert → Image** before anybody had heard of fromwhere.
fromwhere cannot tell where such a shape came from, so the diagram is recorded
without it and the composite rests on a picture nothing accounts for.

You can say where it came from by hand. In draw.io, select the shape and use
**Edit → Edit Style…** — or right-click → **Edit Style** — and check that the
shape is wrapped in an `<object>`; if the style dialog shows a bare `mxCell`,
add a label first (**F2**, then anything), which is what makes draw.io wrap it.
Then use **Extras → Edit Diagram…** and add one attribute to that `<object>`:

```xml
<object label="" src="figures/cp_curve.png" id="7">
```

That is the whole of it. `src` is a path relative to the project root, and it
is the only thing fromwhere needs:

- **The hash is optional.** fromwhere computes one from the bytes already
  embedded in the shape, so a person who can type a path but cannot work out a
  SHA256 is not stuck. Requiring both would silently drop the panel from the
  record — the diagram would look complete while resting on a figure nothing
  accounted for, which is the exact failure the record exists to make visible.
- **The next export fills it in.** `fromwhere drawio export` writes the hash onto
  the shape once it has verified it, so the gap closes itself and the diagram
  becomes checkable by anything that reads it without also reading
  `provenance.toml`.

One caveat that is the reason `fromwhere drawio import` exists at all: draw.io
re-encodes anything over 1200 px through a canvas when _it_ embeds an image,
which destroys any Content Credentials the file carried, including an
AI-generation disclosure. Adding `src` afterwards restores the _link_, not the
manifest. For a panel that carries credentials worth keeping, re-import it.

### GIMP

To export a PNG from GIMP with provenance tracking, run:

```sh
fromwhere gimp export my-input.xcf my-output.png
```

### MyST

`fromwhere` includes a MyST plugin for inspecting and checking embedded figure
provenance and freshness.
It also enables inspecting document-level provenance, since that's a
composite artifact itself.

There is **one directive**, because there was only ever one question: what is
this, what was it made from, and is it still true. The scope is the argument.
Name a file and the panel is about that file:

```
:::{fromwhere} figures/composite.svg
:name: fig-performance
The composite figure.
:::
```

The directive is not only for figures. Point it at a `.csv` or `.tsv` and it
renders the numbers as a table — numbered and cross-referenceable like any
other, with the same provenance panel underneath:

```
:::{fromwhere} data/performance.csv
:name: tbl-performance
:rows: 25
The measurements underlying [](#fig-performance).
:::
```

That is where the record earns the most, because a CSV cannot carry Content
Credentials at all: the line in `provenance.toml` is its only provenance. Long
tables are truncated at `:rows:` (25 by default) with a note saying so — a
table is for reading, and a thousand rows of it is a scroll bar.

Point it at an `.html` file — an interactive Plotly or Altair chart, a rendered
notebook — and it is embedded in a frame, numbered and cross-referenced like
any other figure:

```
:::{fromwhere} figures/cp_curve_interactive.html
:name: fig-cp
:height: 400px
Hover for the numbers; drag to zoom.
:::
```

That is the case where the record earns the most of all. **HTML cannot carry
Content Credentials in either direction** — c2pa does not recognize the type —
so for an interactive figure the line in `provenance.toml` is the only provenance
there is. It also has to be self-contained: an artifact that fetches half of
itself from a CDN at read time is not an artifact anyone can hash, because what
a reader sees depends on what that URL served them.

The panel has a state between green and red, and it is the one worth knowing
about: **incomplete provenance**. Every hash matches, nothing is stale, nothing
was tampered with — and something in the chain is a file nobody has claimed:

```
⚠️ Figure has incomplete provenance — nothing accounts for scripts/plot_cp.py
   To fix: fromwhere declare scripts/plot_cp.py --mine [--with-ai <tool>]
```

A warning rather than an error, because nothing is broken and no output needs
regenerating; what is missing is a person's statement, and only a person can
supply it. It gets its own state because every automated check passes, which is
exactly what makes the gap easy to miss — and a panel that printed a green tick
with the finding folded away inside it would be the thing doing the hiding. The
document-level panel says the same, so the two cannot appear to disagree.

Anything that is none of those renders as its filename with the panel attached,
and the panel names what it is looking at: _Figure_, _Table_, or _Artifact_.
The extension decides which, and `:kind:` overrides it for the `.dat` that is
really delimited, or the `.svg` that is a diagram of the method and has no
business being numbered as a figure.

Name **no** file and the panel is about the document you are reading — a
composite artifact in its own right, and the same question one scope wider.
`:artifact:` names the document's own output(s):

```
:::{fromwhere}
:artifact: _build/html/index.html, _build/exports/paper.pdf
:table: true
:graph: true
:::
```

(`:kind: document` says it outright, for a block that has a path and wants the
document anyway.)

Naming them does two things. It shows the command that rebuilds the document,
read from the record so it cannot drift from what actually produced the file.
And it leaves those outputs out of the panel's own freshness tally — a document
cannot honestly report on itself from the inside, because while the page is
being written its recorded hash still describes the previous build. Without
this the panel reads "out of date" on every single build and stops meaning
anything. `fromwhere status` checks them from outside, where the answer is
settled.

The table lists **outputs** — anything the project made, including
intermediates like a `.drawio` — and for each one names the input responsible
when it is behind:

| Output                  | Built from | State                                |
| ----------------------- | ---------- | ------------------------------------ |
| `figures/cp_curve.png`  | 3          | ⚠️ `scripts/plot_cp.py` changed      |
| `figures/composite.svg` | 1          | ⚠️ waiting on `figures/cp_curve.png` |

Sources are what those answers point at, not rows of their own. Asking whether
a plotting script is "up to date" has no answer — nothing produces it — and
listing it green above the figure it just broke is the confusion this avoids.

One caveat about live preview. `myst start` re-renders when one of _its own_
sources changes: markdown, `myst.yml`, a linked image. Editing a script or a
dataset is invisible to it, so the panels keep showing the previous render and
a stale figure looks current for as long as the tab is open. The example's
`make serve` works around this by watching everything fromwhere records as an
input and touching the document when any of it moves.

### Quarto

The same panels, in Quarto. Install the extension into a project and wire it
into `_quarto.yml`:

```sh
fromwhere quarto install
```

```yaml
filters:
  - at: pre-ast
    path: fromwhere
```

It comes from the package rather than from `quarto add` because the Lua filter
and the `fromwhere-quarto` executable it spawns speak a protocol private to
fromwhere, and installing them together is what keeps the two the same version.

`at: pre-ast` is not decoration. Quarto builds callouts, figure numbers, and
cross-references in its own filters, so a panel emitted after them is a grey
box sitting under an unnumbered picture — it renders, and everything about it
is subtly wrong.

A fenced div does what MyST's directive does, because it is the one Quarto
construct that takes a path, options, and a caption that is _parsed_ — so
citations and cross-references inside a caption keep working:

```
::: {.fromwhere src="figures/composite.svg" #fig-performance width="95%"}
Power coefficient against tip speed ratio.
:::
```

One Pandoc habit to know about, because scientific captions walk into it
constantly: a caption beginning `(a)` is parsed as an **ordered list**, not as a
panel label. Pandoc's `fancy_lists` treats `(a)`, `(1)` and `a.` at the start of
a block as list markers, and the result is a caption turned into an `<ol>` with
Quarto's own "Figure 1:" prefix chopped into list items. Nothing warns you. Bold
the label — `**(a)** Power coefficient…` — or escape it as `\(a\)`. This is
plain Quarto behavior rather than anything fromwhere does, and it applies to every
caption in the document.

Point it at a `.csv` or `.tsv` and it renders the numbers as a table, numbered
and cross-referenceable like any other, with the same provenance panel
underneath. Anything that is neither a picture nor a table renders as its
filename with the panel attached:

```
::: {.fromwhere src="data/performance.csv" #tbl-performance rows="25"}
The measurements underlying [@fig-performance].
:::
```

Name no `src` and the panel is about the document, exactly as in MyST. It takes
the same `artifact` option, naming the document's own output(s) so it can show
the rebuild command and leave them out of its own freshness tally:

```
::: {.fromwhere artifact="_site/index.html" table="true" graph="true"}
:::
```

A `.fromwhere-provenance` div is no longer a thing, and rather than rendering as
an anonymous grey box it says what replaced it — an unknown class is silence,
and silence is the failure this panel exists to prevent.

Everything that decides _what to say_ — the chain, the AI disclosure, the
headline, the truncation note — is the same Python that answers `fromwhere
status` and renders the MyST panels. The Lua half only turns those nodes into
Pandoc's AST, so a document built either way says the same thing.

Two things are Quarto-specific and worth knowing. The derivation graph is drawn
with Quarto's own bundled Mermaid, asked for as an HTML dependency: Quarto
renders Mermaid in its _engine_, before pandoc runs, so a filter cannot produce
a diagram the ordinary way. That works in HTML; in a format with no Mermaid
runtime — PDF, docx — the graph falls back to its source rather than
disappearing.

The second is live preview, and it has a sharp edge worth knowing before you
hit it. `quarto preview`, like `myst start`, re-renders only when one of _its
own_ sources changes: the `.qmd`, `_quarto.yml`, a linked image. Editing a
plotting script is invisible to it, so the panels keep showing the previous
render and a stale figure looks current for as long as the tab is open. The
example's `make preview` closes that by watching everything fromwhere records as
an input and touching the document when any of it moves.

**That workaround does not work under VS Code's Quarto extension**, which
launches `quarto preview --no-watch-inputs` and re-renders only when you save
the document it is previewing. With input watching off, nothing on the
filesystem can trigger a render — not the script, and not a touched `.qmd`. Use
`make preview` in a terminal, or save the `.qmd` after editing a script.

## Examples

Five worked projects, each runnable, in [`examples/`](examples):

| Example                           | What it shows                                                                                                                                           |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`myst`](examples/myst)           | The MyST plugin, a draw.io composite, and an AI-generated panel whose disclosure travels in its own Content Credentials                                 |
| [`quarto`](examples/quarto)       | The Quarto extension: Plotly panels, an interactive figure, a composite, a CSV as a cross-referenceable table — under pixi, so the lock pins Quarto too |
| [`calkit`](examples/calkit)       | fromwhere wrapping a Calkit stage command, with `provenance.toml` beside a `dvc.lock` — and why signing belongs at the boundary                         |
| [`snakemake`](examples/snakemake) | The same arrangement under Snakemake, and what a build cache is for versus what a record is for                                                         |
| [`astra`](examples/astra)         | ASTRA declaring the decision space while fromwhere records which options actually produced the bytes                                                    |

The last three share a shape worth naming. fromwhere wraps the **stage command**,
never the workflow manager: wrapping the manager would make fromwhere the
entrypoint, record one enormous artifact, and lose the per-figure chain that is
the whole point. And in each of them the repair for a stale artifact is the
manager's own command, not `fromwhere rebuild` — two components that each claim to
know how to rebuild a project, from different graphs, will drift.

### What the lock file does not cover

fromwhere records the lock of the environment the **command** ran in. Anything
that wraps that command from the outside is not in it — fromwhere itself, the
workflow manager, and any tool they shell out to. That is a real limit on what
the record means, and it is worth knowing where it bites:

- **MyST needs Node.** `mystmd` on PyPI is a wrapper around a JavaScript CLI.
  The JS ships in the wheel so the MyST version is pinned, but the runtime is
  taken from `PATH` — and with no Node installed it stops and _asks
  interactively_ whether to install one, which hangs a CI build rather than
  failing it. `MYSTMD_ALLOW_NODEENV=1` answers that up front.
- **draw.io is a desktop app.** No conda or PyPI package exists, so nothing can
  pin it, and an exported composite rests on a version the record cannot name.
- **Quarto can be pinned, and should be.** The Quarto example uses pixi for
  exactly this reason: rendering the document is a recorded step, so a lock that
  did not include the renderer would be asserting an environment it does not
  control.

The general remedy is to choose an environment manager that can hold the whole
toolchain — pixi reaches conda-forge, which has Quarto and Node — rather than
one that can only hold the Python part. Each example's README says which of
these applies to it.
