Metadata-Version: 2.4
Name: agent-folder-workspace
Version: 0.1.0
Summary: Offline, lazy, agent-oriented exploration of heterogeneous folders
Author-email: Dark Light <darklight@noreply.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,documents,mcp,office,offline,sqlite
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: <3.15,>=3.11
Requires-Dist: defusedxml<1,>=0.7
Requires-Dist: lxml<7,>=5
Requires-Dist: mcp<2,>=1.10
Requires-Dist: odfpy<2,>=1.4
Requires-Dist: openpyxl<4,>=3.1
Requires-Dist: pdfplumber<0.12,>=0.11
Requires-Dist: platformdirs<5,>=4
Requires-Dist: pydantic<3,>=2.11
Requires-Dist: pypdf<7,>=5.7
Requires-Dist: python-docx<2,>=1.2
Requires-Dist: python-pptx<2,>=1
Requires-Dist: pywin32>=310; sys_platform == 'win32'
Requires-Dist: pyyaml<7,>=6
Requires-Dist: typer<1,>=0.16
Description-Content-Type: text/markdown

# agent-folder-workspace

`agent-folder-workspace` presents one local directory as a stable, paginated
node hierarchy for agents. It discovers file metadata first and opens document
bodies only when a caller expands or reads a node. The default parser path is
local Python code: Microsoft Office and LibreOffice are not required.

> **Project status:** `0.1.0` is an alpha contract. Read-only exploration and
> byte access are implemented; editing, rendering, OCR, and complete semantic
> coverage of every supported container are not.

Python 3.11 through 3.14 is supported. This repository does not assume that a
release has already been published to a package index. Install a checkout with:

```console
python -m pip install .
folderws --help
```

Runtime installation uses only Python packages from `pip`; the Windows-only
`pywin32` dependency is selected by an environment marker. Java, .NET,
Microsoft Office, and LibreOffice are never installed or required by this
package. Already-installed office suites are optional probe/conversion
backends.

## Quick start

```python
from pathlib import Path

from agent_folder_workspace import FolderWorkspace

with FolderWorkspace.open(Path("./documents")) as workspace:
    if not workspace.wait_until_ready(timeout=10):
        raise TimeoutError("metadata scan is still running")

    info = workspace.workspace_info()
    first_page = workspace.list_children(info.root_node.id, limit=50)
    for node in first_page.nodes:
        print(node.id, node.kind, node.relative_path, node.content_status)
```

Call `list_children(file_node.id)` to expose a format-specific hierarchy, then
use `read_content()` on a semantic node or `read_binary()` for a bounded byte
range. Results are Pydantic models and can be serialized with
`result.model_dump(mode="json")`.

## Architecture

The package separates transport, workspace policy, node discovery, format
adapters, and caching:

1. `FolderWorkspace` validates and owns one root directory and one generation.
2. `NodeRegistry` builds an in-memory metadata tree. The initial scan uses
   directory entries and file stat data; it does not read file payloads.
3. `AdapterManager` keeps text, CSV, and bounded binary access direct. It sends
   semantic `children`/`read` calls for structured text, SQLite, OOXML, ODF,
   PDF, and legacy Office to a bounded pool of spawn-based parser workers.
4. Parser workers block Python socket/DNS entry points before importing risky
   adapters and enforce one deadline across startup, queueing, IPC, and parsing.
5. `SizedLru` retains bounded decoded pages in memory. `ContentIndex` stores an
   optional per-workspace SQLite search index in the application cache.
6. The Python API, Typer CLI, and FastMCP stdio server share the same workspace
   methods and versioned response contracts.

Node IDs are deterministic hashes of relative and logical paths; they do not
contain the absolute workspace path. Pagination cursors are opaque,
integrity-protected, workspace-instance tokens. `refresh_workspace()` advances
the generation, rebuilds metadata, clears decoded pages, synchronizes an open
index, and makes older cursors stale.

See [the architecture notes](docs/architecture.md) for lifecycle, data-flow,
cache, and error details.

## Python API

The supported top-level entry points are:

| API | Purpose |
| --- | --- |
| `FolderWorkspace.open(root, config=None)` | Open one existing local directory. |
| `workspace_info()` | Report the root node, generation, scan state, and index coverage counts. |
| `list_children(node_id, cursor=None, limit=100)` | Page through filesystem or virtual children. |
| `get_node(node_id)` | Fetch a node by stable ID. |
| `read_content(node_id, cursor=None, limit=100, representation="semantic")` | Read a semantic/source-record page; `raw` delegates to bounded binary access. |
| `read_binary(node_id, offset=0, length=..., cursor=None)` | Return at most 1 MiB as base64 and an optional continuation cursor. |
| `search_content(query, cursor=None, limit=20)` | Search filenames plus progressively indexed content. |
| `index_content()` | Walk supported semantic leaves into the persistent content index. |
| `refresh_workspace()` | Re-scan and invalidate old generation-bound cursors. |
| `close()` | Stop the scan worker/parser pool and close an open index. |

`FolderWorkspace` is a context manager. Known file nodes and shallow requested
directories remain usable while the deep metadata scan continues; a lookup for
an as-yet undiscovered node can wait for that scan. Full indexing and exact
coverage require the inventory to finish.

`WorkspaceConfig` controls page, binary, cache, embedded-file, parser-worker,
archive, XML, and structural limits. It is frozen and rejects unknown fields.
The package exports a stable
error hierarchy (`FolderWorkspaceError`, `InputValidationError`,
`NodeNotFoundError`, `StaleCursorError`, and `ResourceLimitError`) and six
versioned wire models:

- `CapabilityReportV1`
- `ContentPageV1`
- `DiagnosticV1`
- `NodePageV1`
- `NodeV1`
- `SearchPageV1`

Committed JSON Schemas live in
`src/agent_folder_workspace/schemas/`. Regenerate them with
`python scripts/generate_json_schemas.py`; CI checks that they still equal
Pydantic's `model_json_schema()` output.

## MCP server

Run one stdio server for one root:

```console
folderws mcp --root ./documents
```

The server exposes exactly these tools:

- `workspace_info`
- `list_children`
- `get_node`
- `read_content`
- `read_binary`
- `search_content`
- `index_content`
- `refresh_workspace`
- `probe_backends`

Results use the same versioned models as the Python API. A method that is absent
from an injected workspace implementation returns a structured
`capability_unavailable` response instead of failing server construction.

The transport is stdio. There is no HTTP listener, authentication layer, or
multi-root broker. The process and its MCP client share the operating-system
permissions of the account that launched `folderws`; choose the root narrowly.

## CLI

```console
# Serve a folder to an MCP client over stdio
folderws mcp --root DIR

# Detect optional backends without launching them (default)
folderws doctor

# Opt in to isolated Office/LibreOffice round-trip probes
folderws doctor --probe-office --json --timeout-seconds 10

# Build the complete semantic content index
folderws index --root DIR

# Remove only the selected application cache tree
folderws clear-cache [--cache-dir PATH]

# Materialize the bundled, managed OpenCode skill and plugin
folderws install-opencode --target DIR
```

Legacy positional roots/targets and `doctor --active` remain compatibility
aliases; the explicit forms above are the primary interface.

`install-opencode` does not edit `opencode.json` or `opencode.jsonc`. It writes
two ownership-marked resources atomically and always refuses foreign files,
symbolic links, and Windows reparse points. The compatibility `--force` flag
never bypasses ownership checks.

`clear-cache` removes only a non-link directory containing the package's cache
ownership marker; an arbitrary directory supplied through `--cache-dir` is
refused.

## Lazy loading and cache

Lazy means that the initial recursive scan records names, types, sizes, and
timestamps, but not document bodies. Expanding a file asks its adapter for
virtual children. Reading a semantic leaf parses only the requested format path
where the adapter permits it; some container adapters must still parse a whole
XML part, table, or package directory to answer that request.

Text and CSV semantics, metadata scanning, and bounded memory-mapped
original-file byte reads run in
the workspace process. Structured text, SQLite, OOXML, ODF, PDF, and legacy
Office semantic `children`/`read` calls, plus byte extraction from their
container parts/streams/records, use a lazily started pool of
`parser_max_workers` long-lived spawn processes (two by default). A
`parser_timeout_seconds` deadline covers pool startup, queue wait, request and
response IPC, and execution. A timed-out worker is terminated and replaced;
the caller receives a stable `parser_timeout` diagnostic.

Supported embedded package parts and PDF attachments are mounted only when
their node is expanded. The bytes are copied atomically, in bounded chunks, to
a hashed per-workspace `embedded` cache file, then exposed below a `contents`
node through the normal adapter/worker path. Depth, count, individual file size,
and total disk-cache limits apply. This feature creates derived document copies
on disk; `folderws clear-cache` removes them with the index.

Decoded `ContentPageV1` values use a byte-sized in-memory LRU capped by
`max_content_cache_bytes` (256 MiB by default). Reads are also indexed on a
best-effort basis. `index_content()` performs a full supported semantic walk
without retaining all decoded bodies in RAM.

The persistent index is an application-owned SQLite database below the
platform cache directory, separated by a hash of the workspace root. It uses
FTS5 when available and a plain-table fallback otherwise. `max_disk_cache_bytes`
is checked after read-through chunks and completed files; least-recently-used
indexed files are evicted and SQLite WAL/database pages are compacted. Embedded
and alternative views enforce the same application-cache budget at their own
materialization boundaries.
Read-through index failures do not make an otherwise successful content read
fail. Embedded mounting, by contrast, requires a writable bounded cache. Use
`folderws clear-cache` to remove an explicitly selected cache tree safely.

Normal completion removes scratch immediately. On a later workspace/cache open,
package-owned partial files and conversion/probe sessions older than five
minutes are removed without traversing linked paths; recent items are retained
to avoid interfering with another process.

Search `coverage` is the fraction of discovered files whose full indexing pass
completed. Filename matches are available before content coverage reaches 1.
If any directory could not be inventoried, the true denominator is unknown;
`coverage` is then conservatively `0` and `complete=false`, even when every
successfully discovered file has been indexed. The affected directory node is
`partial` and carries a `directory_scan_failed` diagnostic.

### Optional legacy conversion views

`WorkspaceConfig.backend_policy` accepts four closed values:

- `python_first` (default) uses the built-in DOC/XLS/PPT parser and attempts an
  actively usable Microsoft Office backend, then LibreOffice, only if that
  parser fails;
- `python_only` never starts an office suite;
- `microsoft_office` and `libreoffice` keep the Python view and additionally
  create a converted, provenance-marked OOXML child when a legacy file is first
  expanded.

Optional conversion is performed by a timeout-bounded child Python process.
The result is verified as OOXML and mounted through the normal OOXML adapter.
Its cache key includes the backend, adapter version, and SHA-256 of the source,
and both conversion scratch and the retained result stay below the
`platformdirs` per-workspace application cache rather than `%TEMP%` or `/tmp`.
Microsoft automation disables macros, UI, events, external-link updates, and
recent-file writes where the application API exposes those controls.
LibreOffice uses a disposable cache-local profile, headless/safe-mode flags,
and never touches the user's normal profile. The external applications are
optional and are used only after an active round-trip capability check.

## Security model

The normal Python parser path is offline and read-only. It does not upload
documents and does not execute document macros. Important controls include:

- roots that are symbolic links or reparse points are rejected;
- child links are not followed by default, and resolved sources must remain
  under the selected root;
- SQLite files are opened with `mode=ro` and `PRAGMA query_only=ON`;
- YAML uses `safe_load`; XML uses defused parsers; HTML parsing disables network
  access;
- OOXML and ODF apply entry-count, expanded-size, XML-size, and compression-ratio
  checks; ODF rejects encrypted package entries;
- PDFs reject encryption and enforce configured content/attachment limits;
- binary reads, page sizes, and search inputs are bounded;
- risky semantic adapters run in bounded spawn workers; Python socket/DNS APIs
  and proxy variables are blocked before those adapters import, and hard call
  deadlines retire a hung worker;
- supported embedded files are materialized atomically only inside a checked,
  per-workspace cache path with depth/count/file/disk limits;
- pagination cursors are signed and generation-bound.

These controls reduce accidental traversal and common parser abuse; they are
not a sandbox. A caller with `read_binary` access can intentionally obtain bytes
from any regular file below the chosen root. Spawn workers inherit the account's
filesystem permissions and are not an OS sandbox; their network block covers
Python socket paths, not arbitrary native code. Text/CSV parsing, original-file
reads, and embedded-byte materialization remain in the workspace process;
container-part range extraction runs in parser workers.
Size and timeout limits do not prove that every malformed file is harmless. Run
the server under a minimally privileged account for untrusted collections.

`folderws doctor --probe-office --json` is different from default parsing: it
explicitly launches isolated Microsoft Office COM applications or a headless
LibreOffice process, copies bundled minimal legacy DOC/XLS/PPT fixtures into
application-cache scratch, and verifies real OOXML round trips with timeouts and
no-UI/no-macro/no-link settings. It never uses documents from the workspace,
but it can still be affected by local application policy.

See [SECURITY.md](SECURITY.md) before exposing sensitive folders or reporting a
vulnerability.

## Supported formats

| Family | Extensions | Current semantic surface |
| --- | --- | --- |
| Plain/delimited text | `.txt`, `.md`, `.csv`, `.tsv` | Line or row blocks; header-derived CSV records. |
| Structured text | `.json`, `.jsonl`, `.yaml`, `.yml`, `.xml`, `.html`, `.htm` | Hierarchical values/elements, attributes, and text. |
| SQLite | `.sqlite`, `.sqlite3`, `.db` | Schema objects and paginated table/view rows, read-only. |
| OOXML | `.docx`, `.docm`, `.xlsx`, `.xlsm`, `.pptx`, `.pptm` | Core document/sheet/slide semantics plus raw package parts. |
| OpenDocument | `.odt`, `.ods`, `.odp` | Core text/table/sheet/slide semantics plus package parts. |
| PDF | `.pdf` | Metadata, pages, text blocks, extracted tables, image descriptors, attachments, structural objects, and original bytes. |
| Legacy Office | `.doc`, `.xls`, `.ppt` | Partial semantic extraction plus CFB streams and lossless known/unknown record bytes. |
| Other files | any other extension | Metadata and bounded original-byte reads only. |

The detailed coverage and per-format caveats are in
[docs/formats.md](docs/formats.md). Windows-specific backend behavior is in
[docs/windows.md](docs/windows.md).

## Limitations

- This is an explorer, not an editor, renderer, office-suite replacement, or
  general-purpose file-conversion service; optional legacy conversion exists
  only to mount an alternative semantic view.
- Semantic extraction is deliberately partial. Raw package/stream access makes
  omitted structures inspectable but does not turn them into high-level data.
- Formulas are reported where supported but never calculated. Cached values can
  be absent or stale.
- Scanned PDFs and images have no OCR path. Visual layout, charts, animations,
  tracked changes, styles, signatures, and embedded-object semantics are not
  comprehensively reconstructed.
- Encrypted/password-protected documents are unsupported. Macros and external
  links are exposed only as inert package data where available and are never
  executed or refreshed.
- Recognized embedded files are mounted recursively only from supported
  package/attachment nodes and only within configured depth/count/byte limits.
  Unsupported extensions remain raw; mounted bytes persist in the application
  cache until eviction or explicit clearing.
- Legacy `.doc`, `.xls`, and `.ppt` parsing recognizes a useful subset of binary
  records; valid files can contain semantics that appear only in raw streams.
- The filesystem is not watched. Call `refresh_workspace()` after changes. A
  refresh invalidates all prior cursors.
- Full content search is opt-in/progressive and limited to semantic leaves the
  indexer considers searchable. Check `coverage` instead of assuming complete
  search results.
- Resource limits are defense in depth, not hard OS isolation.
  `parser_timeout_seconds` is a hard total-call deadline for isolated semantic
  adapters, but not for metadata scanning, text/CSV, or binary reads.
- Optional suite conversion is a semantic alternative view, not a pixel-exact
  rendering. A hard parent timeout cannot guarantee cleanup of a vendor process
  that is itself stuck below COM or LibreOffice process control.
- The stated 100,000-node and warm p95 performance figures are design targets,
  not certified guarantees in `0.1.0`; CI covers correctness across Windows,
  Linux, and macOS but does not emulate a specific Windows-x64 SSD workload.

## Development

See [CONTRIBUTING.md](CONTRIBUTING.md) and the
[0.1.0 alpha acceptance record](docs/acceptance.md). The complete local gate is:

```console
python scripts/generate_json_schemas.py --check
ruff check .
mypy
pytest
python -m build --no-isolation
python scripts/check_wheel.py dist --install-smoke
python scripts/check_wheel.py --reproducible
```

The wheel check can also build in a temporary directory when no path is given:

```console
python scripts/check_wheel.py
```

Use `--reproducible` to build twice with a fixed ZIP epoch and require identical
SHA-256 digests.

## License

Licensed under the [Apache License 2.0](LICENSE).
