Metadata-Version: 2.4
Name: fileroute
Version: 0.1.2
Summary: Convenient connectors to make interoperating and collaborating across multiple content management systems easier
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: pydantic>=2.10.6
Requires-Dist: pydantic-settings>=2.7.0
Requires-Dist: ruamel.yaml<0.20,>=0.19.1
Requires-Dist: requests>=2.32.3
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: typer>=0.24.1
Requires-Dist: boto3>=1.42.55
Requires-Dist: google-auth-oauthlib>=1.2.1
Requires-Dist: google-auth>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: msal>=1.33.0
Provides-Extra: docs
Requires-Dist: mkdocs>=1.6.1; extra == "docs"
Requires-Dist: mkdocs-material>=9.7.1; extra == "docs"

# fileroute

Fileroute describes where project artifacts come from, where they live locally,
and where they should be published. A small YAML or JSON catalog connects local
files with SharePoint, Google Drive, and S3 URLs, so a document or data export
can keep its provenance and multiple destinations in one place. The CLI can
preview changes without credentials, visualize descriptor workflows as SVG, HTML,
or Markdown, pull remote inputs, and push supported outputs; the same descriptor, graph, and
transfer APIs are available from Python. Use it inside a project for repeatable,
versioned workflows, or run it as a standalone tool to inspect and retrieve
individual files. Python 3.11 or newer is required.

## Choose how to run it

**Within a project:** add Fileroute to that project's dependencies, commit its
`uv.lock` and descriptor, and run commands from the project root. `uv run` uses
the project's environment and resolves its declared dependencies before running
the command. This is the better fit for automation and Python API imports.

```bash
uv add git+https://github.com/mbkranz/fileroute.git@dev
uv run fileroute diagram config/fileroute.yaml
uv run fileroute pull config/fileroute.yaml --dry-run
uv run fileroute push config/fileroute.yaml --dry-run
```

Use a commit SHA instead of `dev` in the Git dependency for a fixed revision,
or `uv add ./path/to/fileroute` when developing against a local checkout. See
uv's [project command guide](https://docs.astral.sh/uv/concepts/projects/run/)
and [dependency guide](https://docs.astral.sh/uv/concepts/projects/dependencies/).

**As a standalone tool:** `uvx` (an alias for `uv tool run`) runs Fileroute in
its own cached, disposable environment, separate from any project environment.
It reads descriptor paths and files from your current working directory, but it
does not add Fileroute to the project's dependencies or make it importable by
that project's Python code. Use this for ad hoc CLI operations:

```bash
uvx --from git+https://github.com/mbkranz/fileroute.git@dev fileroute list config/fileroute.yaml
uvx --from git+https://github.com/mbkranz/fileroute.git@dev fileroute diagram config/fileroute.yaml
uvx --from git+https://github.com/mbkranz/fileroute.git@dev fileroute pull config/fileroute.yaml --dry-run
```

Once the intended Fileroute distribution is available from your package index,
the shorter forms include `uvx fileroute diagram config/fileroute.yaml` and
`uvx fileroute list config/fileroute.yaml` (or `uvx fileroute --help`). For a
fixed tool version, pin the package version or Git commit. See uv's
[tool guide](https://docs.astral.sh/uv/concepts/tools/).

To develop this repository itself:

```bash
uv sync
uv run fileroute --help
```

Configure credentials using `.env-sample`. SharePoint uses the `AZURE_*` and
`SHAREPOINT_*` settings; Google supports ADC, service accounts, and user OAuth;
S3 uses the standard AWS credential chain. Descriptor loading, diagramming, and
dry runs do not authenticate.

```bash
fileroute auth login gdrive
fileroute auth login microsoft --auth-mode delegated
fileroute auth login sharepoint --auth-mode delegated
```

See [descriptor diagrams](docs/diagram.md),
[Google authentication](docs/google-auth.md), and the generated
[CLI](docs/cli.md) and [Python API](docs/api.md) references.

## Descriptor model

Four models define the public descriptor API: `Catalog`, `Resource`, `Location`,
and `CatalogReference`. A catalog groups resources and nested catalogs. A
resource describes one artifact. Sources and targets are lists of locations.

| Field | Meaning |
| --- | --- |
| `path` | Artifact location; a local file/directory for transfers |
| `sources` | Upstream inputs or provenance, including local authoring files |
| `targets` | Downstream publication destinations |
| `serviceType` | Optional location provider: `GoogleDrive`, `SharePoint`, or `S3` |
| `serviceId` | Optional provider-native identifier on a location |
| `$ref` | Another local catalog document, resolved beside its containing document |

For example, a rendered Word document can identify its Quarto source without
making that source a publication destination:

```yaml
$schema: fileroute-catalog
catalogs:
  - name: documentation
    path: docs/_output
    targets:
      - path: https://contoso.sharepoint.com/sites/dev/Shared%20Documents/Docs
        serviceType: SharePoint
    resources:
      - name: guide
        path: docs/_output/guide.docx
        sources:
          - path: docs/guide.qmd
      - name: internal
        path: docs/_output/internal.docx
        targets: []
```

A retrieval descriptor explicitly identifies its remote source:

```yaml
resources:
  - name: source-export
    path: downloads/source.csv
    sources:
      - path: s3://my-bucket/exports/source.csv
```

All artifact paths are relative to the **working directory**, even when the
descriptor is in `config/` or references another descriptor. The Python planners
also accept an explicit `root`. Paths are preserved when loading and saving.
Transfers reject paths outside that root and symbolic links.

Reference paths are the exception: `$ref` is relative to the containing
**descriptor's directory**. References stay references on normal load/save. Transfer planning explicitly
expands them once; they cannot escape the containing directory. Cycles raise an error.

```yaml
catalogs:
  - name: research
    $ref: catalogs/research.yaml
```

Unrecognized metadata fields survive model round trips. Existing YAML descriptors
are round-trip edited where practical, retaining comments and authored styles.
This is a Fileroute format inspired by Data Package
and DCAT, not a full implementation of either standard. `$schema` is an optional
profile label; loading does not fetch a schema from the network.

## Common use cases

Each resource has one local artifact `path`. Its `sources` record where the
artifact came from; its `targets` list every intended publication destination.
One resource can have multiple targets. `pull` downloads **one remote source**
to `path`, and `push` uploads the file at `path` to **all targets**. Run them as
separate steps; Fileroute does not stream directly between cloud providers or
convert source formats. These examples can be saved as `config/fileroute.yaml`.

### One SharePoint source and two SharePoint targets

Retrieve a report to the project, then publish copies to two SharePoint sites:

```yaml
resources:
  - name: monthly-report
    path: artifacts/monthly-report.csv
    sources:
      - path: https://contoso.sharepoint.com/sites/data/Shared%20Documents/monthly-report.csv
    targets:
      - path: https://contoso.sharepoint.com/sites/reports/Shared%20Documents/monthly-report.csv
      - path: https://contoso.sharepoint.com/sites/archive/Shared%20Documents/monthly-report.csv
```

For this supported combination, run `fileroute pull config/fileroute.yaml`
and then `fileroute push config/fileroute.yaml`, with `--dry-run` on either
command to inspect its plan first. Each SharePoint site needs working access.

### SharePoint source and Google Drive target

Describe retrieval from SharePoint and intended publication to Google Drive:

```yaml
resources:
  - name: partner-report
    path: artifacts/partner-report.csv
    sources:
      - path: https://contoso.sharepoint.com/sites/data/Shared%20Documents/partner-report.csv
    targets:
      - path: https://drive.google.com/file/d/GOOGLE_FILE_ID/view
```

### SharePoint source, SharePoint and Google Drive targets

Use the same local copy for a supported SharePoint publication and a planned
Google Drive publication:

```yaml
resources:
  - name: partner-report
    path: artifacts/partner-report.csv
    sources:
      - path: https://contoso.sharepoint.com/sites/data/Shared%20Documents/partner-report.csv
    targets:
      - path: https://contoso.sharepoint.com/sites/reports/Shared%20Documents/partner-report.csv
      - path: https://drive.google.com/file/d/GOOGLE_FILE_ID/view
```

### Local authoring source with SharePoint, Google Drive, and S3 targets

Keep the input document as provenance while publishing its rendered output:

```yaml
resources:
  - name: guide
    path: docs/_output/guide.docx
    sources:
      - path: docs/guide.qmd
    targets:
      - path: https://contoso.sharepoint.com/sites/docs/Shared%20Documents/guide.docx
      - path: https://drive.google.com/file/d/GOOGLE_FILE_ID/view
      - path: s3://example-docs/guide.docx
```

Render `docs/guide.qmd` to `docs/_output/guide.docx` with Quarto before
publishing. A local source is provenance; `pull` does not render or copy it.

**Current transfer support:** SharePoint, Google Drive, and S3 remote sources
can be pulled; only SharePoint targets can be pushed. A descriptor with a Google
Drive or S3 target is valid metadata, but `push` **and `push --dry-run` fail at
planning** until upload support is implemented. If you need the SharePoint
destination now, put it in a separate descriptor (or remove the unsupported
targets) for that run. `resolve` and `diagram` can still resolve those providers
and inspect the descriptor without authentication. Source and target URLs above
are examples; replace them with your own accessible files and sites.

## Visualize file workflows

`diagram` renders descriptor intent rather than executing a transfer. It expands
local `$ref` catalogs, resolves known provider URLs in memory, applies catalog
target inheritance, and draws sources → artifacts → targets as a standalone SVG.
It does not require Graphviz or another rendering dependency.

```bash
fileroute diagram config/fileroute.yaml
fileroute diagram config/fileroute.yaml --output docs/fileroute-workflow.svg
fileroute diagram config/fileroute.yaml --output docs/fileroute-workflow.html
fileroute diagram config/fileroute.yaml --output docs/fileroute-workflow.md --detail full
```

HTML provides a searchable offline file dictionary linked to diagram nodes; Markdown
writes an anchored dictionary and companion SVG. `--detail summary` exports curated
metadata; `--detail full` includes custom fields and service IDs. Inherited targets
identify their declaring catalog. Reports link to HTTP(S) locations but do not
preview file contents or authenticate. See the diagram guide for anchor stability
and duplicate-identity behavior.

The descriptor argument follows the same saved/default selection behavior as
`pull`, `push`, and `resolve`, so after `fileroute activate` you can simply run
`fileroute diagram`. The default output is `fileroute-diagram.svg` in the
current working directory. See [Descriptor diagrams](docs/diagram.md) for the
Python `DescriptorGraph` API and repository-integration guidance.

## Commands

```bash
fileroute add export --path downloads/source.csv --source s3://my-bucket/source.csv
fileroute add documentation --catalog --path docs/_output --target https://contoso.sharepoint.com/sites/dev/Docs
fileroute list config/fileroute.yaml --format json
fileroute activate config/fileroute.yaml
fileroute update --name documentation --title "Published documentation"
fileroute resolve config/fileroute.yaml
fileroute resolve config/fileroute.yaml --write
fileroute diagram config/fileroute.yaml
fileroute migrate old.yaml new.yaml --direction push
fileroute pull config/fileroute.yaml --dry-run
fileroute push config/fileroute.yaml --dry-run
fileroute push config/fileroute.yaml
```

`activate` saves the active descriptor in `.fileroute/descriptor`.
The optional descriptor argument also accepts an explicit override. `update`
selects an exact name or dot-path and reports ambiguous names; `clone descriptor`
copies a single authored document. For a resource with one source,
`update --name export --service-type S3` edits that source's provider.
Edit `sources`/`targets` in YAML for more involved changes, or supply a JSON array
using `--sources`/`--targets`.

### Resolve URLs

`resolve` previews canonical JSON without authenticating or contacting a remote
service. `resolve --write` saves the result to the selected YAML/JSON descriptor.
It infers `serviceType` from `s3://` URLs, `drive.google.com` / `docs.google.com`,
and SharePoint hosts, while preserving the original URL exactly as a clickable
link. For example:

```yaml
path: https://contoso.sharepoint.com/sites/dev/Docs/guide.docx
serviceType: SharePoint
```

A conflicting explicit provider is an error. Unrecognized target URLs require
an explicit `serviceType`; local files and general web citations remain valid
provider-less sources. Pull requires a supported remote source. Resolution
does not follow redirects, fetch remote IDs, or check remote permissions.

Write-back updates only the selected document and preserves `$ref` entries;
resolve referenced descriptors separately to persist their inferred metadata.
Repeated resolution is idempotent. Existing YAML comments and styles are retained
where practical; byte-for-byte whitespace preservation is not guaranteed.
Transfers expand references and resolve the relevant sources or targets in
memory, so write-back is optional. `push --dry-run` and `pull --dry-run` provide
concrete transfer plans; there is no separate `plan` command or stored lockfile.

### Push

`push` publishes `path` to `targets`.
Sources are never treated as targets. Only SharePoint uploads are currently
implemented. Other providers fail during planning, before authentication.

- Catalog targets denote existing remote folders. Children inherit those
  targets using paths relative to the declaring catalog's `path` (or the working
  root when no catalog path is given).
- A child's explicit `targets` replace inherited targets. `targets: []` disables
  publication for that child and its descendants unless a descendant declares
  its own targets.
- A catalog with children publishes only its declared children. A leaf catalog
  with a path publishes all files in that directory tree.
- Explicit resource targets are exact file URLs, allowing renaming. Set a
  target's `entityType: Directory` to append the local filename to a folder URL.
- Multiple targets publish multiple copies. Conflicting files aimed at the same
  destination fail before any transfer.

The planner checks all inputs before authentication. SharePoint transfers
create missing child folders and create/replace files; they never delete remote
files. Files above Microsoft Graph's 250 MB single-request limit fail planning.
Dry runs show the proposed destinations without contacting the remote service;
they cannot verify remote permissions or folder existence.

### Pull

`pull` materializes one remote `sources` entry into each artifact's `path`.
A leaf catalog can retrieve a whole remote directory. A catalog with children
retrieves only those children. Targets never affect retrieval.

Multiple sources may represent a transformation, so pull refuses to choose one.
Local provenance such as `.qmd` inputs is not a download operation: build those
artifacts with their authoring tool, then push them. Use separate catalogs for
retrieval and publication when their source semantics differ. Dry runs plan
remote directories as units; execution enumerates and checks remote file paths
before writing any files. Pull replaces existing local files at planned paths.

## Python API and architecture

```python
from pathlib import Path
from fileroute import Catalog, Resource, Location
from fileroute.descriptor import save
from fileroute.diagram import load_graph, render_svg
from fileroute.transfer import plan_push, push

catalog = Catalog(resources=[Resource(
    name="guide", path="docs/guide.docx",
    sources=[Location(path="docs/guide.qmd")],
    targets=[Location(path="https://contoso.sharepoint.com/sites/dev/Docs/guide.docx")],
)])
save(catalog, "config/fileroute.yaml")
graph = load_graph("config/fileroute.yaml")
render_svg(graph, "fileroute-diagram.svg")
plan = plan_push(Path("config/fileroute.yaml"), root=Path.cwd())
# Inspect plan before transfer.
push(plan)
```

The core has one module per responsibility:

- `models.py`: declarative `Catalog`, `Resource`, `Location`, `CatalogReference`
  and `ServiceType` validation.
- `descriptor.py`: `load`, `save`, `walk`, `find`, and offline `resolve`.
- `diagram.py`: semantic descriptor graphs and dependency-free SVG rendering.
- `diagram_reports.py`: offline HTML inspector and Markdown file dictionary.
- `migration.py`: explicit one-way conversion of legacy descriptors.
- `transfer.py`: `plan_pull` / `plan_push`, then `pull` / `push` execution.
- `item.py`: runtime `ServiceItem` hierarchy.
- `clients/sharepoint.py`, `clients/googledrive.py`, `clients/s3.py`: provider APIs.
- `auth/` and `commands/`: credential handling and CLI workflows.

After resolution, `location.service_type` is the authoritative `ServiceType`
enum used directly for provider dispatch. Pydantic accepts aliases such as
`gdrive` or `google_drive`; saved values are `GoogleDrive`, `SharePoint`, or `S3`.
Python attributes use `service_type`, `service_id`, and `entity_type`; descriptors
use `serviceType`, `serviceId`, and `entityType`. No provider-string conversion
layer or dynamic registry is needed.

`ServiceItem` and provider clients retain their runtime roles. Clients own
provider-specific authentication and HTTP behavior; runtime items expose
`refresh()`, `children`, `get_path()`, `iter_items()`, `iter_files()`, and
`download()`. `item.to_catalog()` emits canonical artifact paths and sources.
Runtime paths remain relative to the provider container. `get_path()` is
relative to the current item. Parent relationships and recursive traversal use
a snapshot until refresh or mutation invalidates it.

## Breaking API changes

Use the four models above and the functions in `descriptor`, `diagram`, and
`transfer`. Model I/O and traversal methods, `upload.py`, `download.py`,
`helpers.py`, and the provider registry have been removed. The CLI has no
`upload` alias or `set` command. Run `activate` again to select a descriptor
using the new single-path selection file; obsolete saved workflow defaults are
no longer read.

Legacy `Drive*` classes, packages, `_cache`, and artifact-level provider fields
are unsupported. Update authored descriptors to `path`, `sources`, `targets`,
`resources`, and `catalogs`; provider metadata belongs on a location. Normal loading has no compatibility shims or automatic legacy conversion. Use
`fileroute migrate OLD NEW --direction pull|push` for an explicit, one-way
conversion that preserves the input file.

## Development

```bash
poe check
poe docs-check
poe docs-update
poe docs-build
poe docs-serve
```

Install the task runner with `uv tool install poethepoet==0.48.0`, or use
`uvx --from poethepoet==0.48.0 poe <task>` for a one-off run. `poe docs-build`
installs the documentation extra through `uv run`.

Provider tests use mocks; they do not prove live tenant permissions or transfers.

Field naming references: [Data Resource](https://datapackage.org/standard/data-resource/),
[DCAT](https://www.w3.org/TR/vocab-dcat-3/), and
[OpenMetadata Drive Service](https://docs.open-metadata.org/latest/main-concepts/metadata-standard/schemas/entity/services/driveservice).

## Package releases

Install the repository task runner with `uv tool install poethepoet==0.48.0`,
then run `poe release-check` for validation and `poe build` for a local wheel
and source distribution. One-off usage is
`uvx --from poethepoet==0.48.0 poe release-check`.

The `publish-to-pypi.yaml` workflow has three jobs: **prepare**, **publish**, and
**release**. Pushes to `dev` start or increment a patch development version
(`1.0.0 → 1.0.1.dev1 → 1.0.1.dev2`). Pushes to `main` promote a prerelease to
stable, or increment the patch when the source already has a stable version.
After a stable tag exists, `dev` starts the next patch series. Normal releases
should not edit the version manually. Merge updated `main` back into `dev`
when needed to keep the branches' version baselines aligned.

`poe release --branch dev --source <full-commit-sha>` runs `scripts/release.py`
in a clean checkout of that source. It uses `uv version --no-sync` to update
`pyproject.toml` and `uv.lock`, invokes `poe build`, and creates a local release
commit and annotated tag. This is the CI preparation command: it changes the
local checkout but does not push or publish. The workflow first saves the built
distributions, then atomically pushes the release commit and tag. Both branches
share one release concurrency group; a stale run fails rather than overwriting
newer work. Rapid pushes may supersede pending runs; the latest source should
be released. Workflow pushes use `GITHUB_TOKEN` and do not recursively trigger
another push workflow.

PyPI publication uses Trusted Publishing bound to this repository and
**`publish-to-pypi.yaml`**. No long-lived PyPI token is needed, and the publish
job does not require a GitHub environment unless the PyPI publisher is configured
to expect one. Only stable `main` versions get a GitHub Release, after PyPI succeeds.
The branch rules must permit the workflow's version commit; rejected pushes
leave both remote refs unchanged.

For a failed publish, use **Re-run failed jobs** on the original Actions run.
The publish job downloads the saved wheel and source distribution without
rebuilding. Identical PyPI uploads can be retried, including a partially
completed upload. A full rerun recognizes the tagged source/branch before
calculating a version and finds its unexpired artifact in the original workflow
run by source commit. Existing GitHub Releases
are left intact. If preparation failed before the atomic push, a fresh attempt
can rebuild and replace that run's unpublished artifact. If the original
artifact has expired or been deleted after the push, stop and recover those
exact files; the workflow deliberately does not rebuild a published version.

`poe release-test` exercises the release helper with temporary local Git remotes
and real uv version changes; distribution builds are mocked. No test publishes
packages or contacts cloud providers.
