Metadata-Version: 2.4
Name: pymoku-parts
Version: 0.14.1
Summary: Another build tool
Author-email: Ben Coughlan <ben@liquidinstruments.com>
Maintainer-email: Ben Coughlan <ben@liquidinstruments.com>
License-Expression: MIT
Keywords: build,dependencies,tasks
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: psutil
Requires-Dist: tomli; python_version < "3.11"
Requires-Dist: protobuf>=7.35.1
Requires-Dist: googleapis-common-protos<1.75.4
Requires-Dist: grpcio
Provides-Extra: cloud
Requires-Dist: httpx; extra == "cloud"
Dynamic: license-file

# Installation

Using Pip

    pip install parts

Or using the self-contained executable

    ./parts

# Introduction

Parts is a build dependency and execution tool heavily inspired by SCons,
bitbake, GitLab-CI among others.

The motivation for parts was the scale of a particular project being managed
which had components larger and more independent than a typical SCons or
Makefile targets; but not requireing the external package and configuration
management provided by bitbake.  These existing solution either felt awkward or
overkill.

Parts intends to provide a simple, lightweight dependency tree API and runtime
to build these medium sized projects.  The command line interface and interation
will feel very familiar.

## Example

Assuming a folder structure

    ./parts.def
    ./module/parts.def


```python
# parts.def
mod = include('module')

default(mod['library'])  # use the instance of Module created in module/parts.def
```

```python
# module/parts.def
import asyncio
from parts import BuildStep


class Module(BuildStep):
    async def do_step(self):
        self.update_progress(total=100, phase='running')
        self.log.info(f'Running Module: {self.name}')
        for i in range(100):
            self.update_progress(advance=1, phase=f'loop {i:d}')
            await asyncio.sleep(0.1)  # do some work
        self.log.info(f'Completed Module: {self.name}')


Module('library')  # automatically registers target with name 'library'
```

Running

    parts --no-cache

will result in

```
[11:29:56] INFO     Building ['default']...                                parts
           INFO     Running Module: library                              library
[11:30:06] INFO     Completed Module: library                            library
╭──────────────────────────────────────────────────────╮
│ 0:00:10 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 │
╰──────────────────────────────────────────────────────╯

           INFO     Complete                                               parts
```


## Design Goals

* parts.def use Python syntax with some shortcuts
* File paths are relative to the parts.def that references them
* BuildStep is the unit of work used by parts
* (sub)projects define all their BuildStep objects in a parts.def
* Build objects are useable by include()'ing another parts.def
* Build objects can be reproduced out of context if their input/output artefacts
match (to enable distribution over multiple hosts)
* One parts.def can reference object instances created from an included parts.def
* BuildSteps can be used to communicate configuration options to depenent steps

# Running parts

From within the toplevel of a project directory, next to a `parts.def`

    parts [targets]

or by specifying the `parts.def` using `--path`

    parts --path path/to/parts.def

Multiple targets can be specified simultaneously.  If no target is provided then
'default' will be run, assuming it was provided by a call the `default()`.

## Listing available targets

    parts --list

## Graphing project dependencies

If graphviz is installed, parts can produce a graph for specific build targets
to help visualize project dependencies.

    parts --graph target

# parts.def Reference

## include()

    # TODO

## source()

    # TODO

## alias()

    # TODO

## copy()

    # TODO

## link()

    # TODO

## publish()

This will create a BuildStep that copies the given dependencies to the global
ouput folder.

    publish('a', 'b', BuildStep('c'))

will copy the outputs of task 'a', 'b' and the newly created 'c' to ./output.

    alias('all', publish('a', 'b', 'c')

Will create a build target 'all' that publishes the outputs of 'a', 'b' and 'c'.

    publish('a', dest='module_a')

Will publish ouputs of 'a' to the `./output/module_a` folder.

## arg()

    # TODO

## default()

    # TODO

# BuildStep

A unit of work in parts is represented by the abstract BuildStep which provides
dependency linking with other BuildSteps as well as informative scheduling and
execution of each task.

    # TODO

## Dependencies

    # TODO

## ProcessStep

    # TODO

# Elaborate vs Execution

Parts will elaborate all included config files to resolve their dependecies
before beginning any execution.

This implies the inputs and outputs for each step must be defined during
elaboration and will not change during execution.

While the execution of each step is parallelised (co-routines, though steps will
likely produce their own processes), elaboration is serialized and not
thread-safe.

Elaboration is fail-fast: the first config statement that raises aborts the
whole run with the offending `file:line`, as do a syntax error and an
`include()` of a path that doesn't exist. This stops the rest of a config
running against a half-populated namespace and silently elaborating a different
graph. `--lenient` restores the old skip-and-warn behaviour, which is useful for
`--list`ing a tree that is known to be broken.

# Logging and Progress Bars

    # TODO

## ParseNodes

    # TODO

# State tracking

    # TODO

## phony targets

    # TODO

# parts.toml

Project-level configuration lives in `parts.toml`. All matching files are
merged, with later locations overriding earlier ones section by section:

1. `~/.config/parts/runners.toml` — legacy name, still accepted
2. `~/.config/parts/parts.toml` — user-global
3. `./parts.runners.toml` — legacy name, still accepted
4. `./parts.toml` — project-local, takes precedence

So a user-global `[runners.X]` and a project-local `[runners.Y]` coexist,
while a project-local `[runners.X]` overrides the user-global one.

The file currently has three sections:

- `[parts]` — overrides for the `--build` and `--output` CLI defaults.
  CLI flags still win when supplied.

  ```toml
  [parts]
  build  = "~/PyMoku-shared/build"
  output = "~/PyMoku-shared/output"
  ```

- `[runners.<name>]` — remote-runner rules (see below).

- `[store]` — the artifact store (see below).

# Artifact store

A `ProcessStep`'s cache has two tiers. The local one is the SQLite
`productions` table next to the build (one row per output: rooted path,
content hash, size, executable bit). The second is an **artifact
store** speaking the Bazel Remote Execution API v2 — any conformant
`ActionCache` + `ContentAddressableStorage` server (bazel-remote,
NativeLink, BuildBarn), or a plain directory.

```toml
[store]
type = "disk"            # or "grpc"
path = "~/.cache/parts-store"
push = true              # false makes the store read-only for this checkout
```

```toml
[store]
type = "grpc"
url = "grpc://build.example.com:9092"   # grpcs:// for TLS
instance_name = "ben"
push = true
concurrency = 16          # transfers in flight while materialising

[store.headers]
authorization = "Bearer ${PARTS_STORE_TOKEN}"
```

With no `[store]` section nothing changes: the local tier is the whole
cache.

## What it stores

A step is keyed on its action digest (see *Cache Invalidation*). Against
that key the store holds an `ActionResult` — every declared output as an
`output_files` entry (rooted path, digest, executable bit), a
`creates('<pattern>')` glob
as an `output_directories` entry naming a `Tree`, and the run's
`stdout.log` as `stdout_digest` — plus the output blobs themselves in the
CAS.

## Flows

- **Read.** On a local miss — or when the local rows describe outputs that
  are no longer on disk — parts asks the store for the `ActionResult`, writes
  it back as local rows, and treats the outputs as deliverable while
  `FindMissingBlobs` says the store still holds them. The step reports
  `cached` without running.
- **Write.** After a step produces its outputs, parts uploads the blobs the
  store is missing and updates the `ActionResult` — only when every declared
  output was hashed, and only when `push` is on.
- **Deliver.** Nothing is materialised until a reader that needs local
  bytes asks for it. Then the blob is downloaded, hash- and size-verified,
  into place, and its executable bit restored — the CAS stores bytes, the
  `ActionResult` stores the mode.

So deleting `build/` and rebuilding re-materialises outputs from the store
instead of re-running, and a second checkout with the same layout hits
without running anything.

## Who materialises what

| Reader | Needs local bytes |
|---|---|
| plain `BuildStep` (`Timing`, `BarFile`, `PythonZip`…) | yes |
| `ProcessStep` on a local, Docker or ssh runner | yes |
| `ProcessStep` on a runner that takes inputs by digest (cloud) | no |
| `copy()`/`link()` | only if a reader of one of its outputs does |
| a requested target's deliverables | yes |

A runner declares which it is with `needs_local_inputs` (True by default).
A `copy()`/`link()` nobody local reads skips the copy entirely: its outputs
already share their source's content hash, so they are recorded as usual
and a later reader still gets them through the store — or, for a source
the store never saw, by staging on demand.

## Backends

- `disk` — `ac/<hh>/<hash>` holds a serialised `ActionResult`, `cas/<hh>/<hash>`
  holds blobs, written temp-and-rename. Uploads hardlink into the store when
  the filesystem allows it. A blob whose size doesn't match its digest reads
  as missing (so the step re-runs); one whose bytes don't match is refused on
  download. A shared filesystem makes this a usable team cache; it is not a
  wire format anyone else reads.
- `grpc` — a real REAPI client: `ActionCache`, `ContentAddressableStorage`
  and `ByteStream`. `GetCapabilities` supplies the batch size limit; blobs
  below it ride `BatchUpdateBlobs`/`BatchReadBlobs`, larger ones stream over
  `ByteStream`. `instance_name` scopes the keys; `headers` are sent as call
  metadata. Materialising many outputs reads small blobs 16 to a
  `BatchReadBlobs` and keeps `concurrency` requests in flight, shared
  across every step asking.

Custom types register through the `parts.stores` entry point group or
`register_store_type()`, exactly like runners.

# Remote Runners

By default, all ProcessStep commands run locally. Runners allow steps to
be redirected to remote execution backends — SSH hosts, Docker containers,
or cloud VMs — without changing parts.def files.

Each runner has a `match` pattern that selects which BuildStep classes it
handles, and a `type` that determines the execution backend.

```toml
[runners.my-runner]
match = "VivadoStep|SynthStep"   # fnmatch patterns, pipe-separated
type = "ssh"                      # local, ssh, pool, docker, reapi, pymoku-cloud
```

Match patterns are tested against each class in the step's MRO (so
matching `VivadoStep` also catches its subclasses). First matching rule
wins. Unmatched steps run locally.

## Runner Types

### local

Runs commands via local subprocess. This is the default when no runner
matches.

### ssh

Executes on a remote host via SSH. Input files are rsynced to the remote
host, the command runs there, and declared output files are rsynced back.
Absolute paths in the command are rewritten to match the remote mirror.

```toml
[runners.build-server]
match = "VivadoStep"
type = "ssh"
host = "build.example.com"
user = "builder"
key = "~/.ssh/id_ed25519"
workdir = "~/parts-builds"
setup = "source /opt/tools/settings.sh"  # run before each command
jobs = 8                                  # max concurrent jobs

[runners.build-server.env]
LICENSE_FILE = "/opt/licenses/license.lic"
```

### pool

Distributes jobs across multiple SSH hosts. Each host has a concurrency
limit; jobs are dispatched to the first host with a free slot.

```toml
[runners.farm]
match = "VivadoStep"
type = "pool"
setup = "source /opt/tools/settings.sh"

[[runners.farm.hosts]]
host = "build1.example.com"
user = "builder"
key = "~/.ssh/id_ed25519"
workdir = "~/parts-builds"
jobs = 4

[[runners.farm.hosts]]
host = "build2.example.com"
user = "builder"
key = "~/.ssh/id_ed25519"
workdir = "~/parts-builds"
jobs = 8
```

### docker

Runs commands inside a local Docker container.

```toml
[runners.sim]
match = "CocoTB*"
type = "docker"
image = "cocotb:latest"
```

### reapi

Executes the step's REAPI `Action` on a remote-execution scheduler — the
same objects the artifact store is keyed on, so nothing is packaged,
tarred or polled.

```toml
[store]
type = "grpc"
url = "grpc://build.example.com:9092"

[runners.route]
match = "VivadoStep"
type = "reapi"
url = "grpc://build.example.com:9092"   # must be the store's server
instance_name = "ben"                   # defaults to the store's
priority = 10                           # ExecutionPolicy.priority
max_inflight = 200                      # concurrent Execute streams
exec_properties = { pool = "vivado-large" }

[runners.route.headers]
authorization = "Bearer ${PARTS_STORE_TOKEN}"
```

`timeout`, `transfer_timeout` and `retries` are accepted as for `[store]`.

A `[store] type = "grpc"` on the same server is required: the scheduler
reads the `Action`, the `Command` and every input by digest out of that
CAS and publishes its outputs back into it, so the store is this runner's
only artifact channel. A missing or differently-addressed store is a
config error on the first step routed here.

The rule's `exec_properties` are routing properties — they are merged into
an *execution copy* of the Action, never into the cache key, so which rule
matched never affects cache hits. The step's own `exec_properties()` (what
it needs of the platform: `vivado = "2024.2"`, a container image) are in
the key and reach the scheduler unchanged.

The runner takes its inputs by digest (`needs_local_inputs = False`), so a
`copy()`/`link()` upstream of it never stages anything and an input a
previous remote step produced is never downloaded. Only a source file that
nothing has pushed is read off disk and uploaded before `Execute`.

Progress follows the `Operation` stream — `ExecuteOperationMetadata.stage`
drives the phase and `stdout_stream_name` is read over `ByteStream` into
the parse tree and `<workdir>/stdout.log`. The stream is reconnected with
`WaitExecution` whenever it ends before the job does (a scheduler behind
Cloud Run caps one request at 60 min; a route runs for hours).

### pymoku-cloud

Submits each step as a job to an HTTP job API. Needs httpx
(`pip install pymoku-parts[cloud]`) and a `[store]` on the same CAS the
workers publish to.

```toml
[store]
type = "grpc"
url = "grpcs://store.example.com"

[runners.cloud]
match = "SynthStep|RouteStep"
type = "pymoku-cloud"
url = "https://jobs.example.com"
api_key = "${PARTS_API_KEY}"
worker_type = "large"                   # required: the worker pool
timeout = 28800                         # per-job wall clock, s (0: server default)
rule_priority = 10                      # claim order within a pool
max_inflight = 200                      # concurrent jobs per worker_type
max_jobs = 30                           # concurrent create/start requests
transient_patterns = ["licence checkout failed"]
```

Inputs go by digest: whatever the store lacks is uploaded before the job
is created, and the worker fetches each one by hash. The worker publishes
every output to the same CAS and the job's completion manifest names each
one, so the outputs are recorded here and fetched only when something local
reads them. A successful job whose manifest doesn't cover a declared
output, or whose outputs the store doesn't hold, fails the step.

A failed job whose `stdout.log` contains one of `transient_patterns` is
retried once on a fresh worker.

### Custom runner types

External packages can register their own runner types via the
`parts.runners` entry point group, or by calling `register_runner_type()`
at import time.

**Entry point (pyproject.toml):**

```toml
[project.entry-points."parts.runners"]
myrunner = "mypackage.runner:create"
```

Where `create(name, config)` is a factory that returns a `Runner` instance.

**Programmatic registration:**

```python
from pymoku.parts.runners import register_runner_type

def my_factory(name, config):
    return MyRunner(name, config)

register_runner_type('myrunner', my_factory)
```

Then use it in `parts.toml`:

```toml
[runners.example]
match = "SomeStep"
type = "myrunner"
custom_option = "value"
```

## Cache Invalidation

A step's cache key (`in_state`) is `SHA256(config_state + sorted input
states)`. A `ProcessStep` instead keys on its **REAPI action digest** —
`SHA256(Action.SerializeToString(deterministic=True))` over
`{command_digest, input_root_digest, platform}` — which covers the same
things:

- `Command.arguments` = `['/bin/sh', '-c', cmd]` with every configured
  root rewritten relative to the workdir, which is also the command the
  runner executes (cwd = the workdir, local or mirrored)
- `Command.environment_variables` — the declared environment (runner
  `[env]`, then `set_env()`, then `extra_env`), sorted by name. Not
  `os.environ`: the local runner leaks that to the process, which is a
  hermeticity gap, not a cache input.
- `Command.output_paths` and `working_directory`, rooted
- `Platform.properties` from `exec_properties()` — what the step
  declares of its execution platform (a tool release, a container image
  id). Default `{}`; `DockerStep` reports `container-image`.
- the input root `Directory` tree: every input file at its rooted path
  (`build/…`, `project/…`, `output/…`; a path outside every root lands
  under `abs/`), with its content digest, size and executable bit.

Runner *identity* is not part of the key: a runner that provides a
different tool than the step declares is a configuration error, not a
different action. So switching runners does not invalidate anything, while
editing a runner's `[env]` does.

Every run writes `<workdir>/.action.json` with the key and its components
(`in_state`, `action_digest`, `command_digest`, `input_root_digest`,
command, env, exec properties, input paths/hashes/sizes, output
paths). Diff two of them to explain a cache miss.

## Environment Variables

Environment variables are merged from three sources (later wins):

1. Runner config `[runners.name.env]` table
2. `set_env()` calls in parts.def (`config_env`)
3. ProcessStep `extra_env` kwarg

## DockerStep with Remote Runners

When a `DockerStep` is matched by an SSH or cloud runner, the step's
`image` attribute is forwarded. The remote runner wraps the command in
`docker run` on the remote host, or passes it to the cloud API as a
container job.
