Metadata-Version: 2.4
Name: privacyfence-sync
Version: 6.1.0
Summary: Run a PrivacyFence data-sync recipe as a plain MCP client, with no AI model involved.
License: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: mcp<3.0.0,>=2.2
Requires-Dist: httpx2>=2.5
Requires-Dist: pyyaml>=6
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-asyncio; extra == "test"
Requires-Dist: pytest-cov; extra == "test"

# privacyfence-sync

`privacyfence-sync` runs a data-sync recipe against PrivacyFence with no AI model involved. A recipe
reads data from your connected services (Salesforce reports, Jira searches, Grist tables, files in
Google Drive), runs your own scripts on the files, writes the result back to Grist, and can publish
an HTML dashboard on Peek.

The intended split of work:

- **Designing** a sync is a job for Claude: the Grist data model, which sources to read, how to
  diff and transform them, what the dashboard shows. Claude writes the recipe, the scripts and the
  dashboard template, and you review them like any code change.
- **Running** the sync on a schedule is a job for `privacyfence-sync`. It makes the same tool calls
  Claude would make, in a fixed order, and runs your scripts between them. It costs no model tokens.

It is an ordinary MCP client of PrivacyFence, like Claude Code. Every call goes through
PrivacyFence's gate, rules and audit log, exactly as Claude's calls do. `privacyfence-sync` never
holds your Salesforce, Jira, Grist, Drive or Peek credentials, cannot approve anything, and cannot
change your rules. A scheduled run declares itself unattended, so any step that no auto-accept rule
covers fails at once instead of waiting for an approval nobody will give.

## Contents

- [Install](#install)
- [Connecting to PrivacyFence](#connecting-to-privacyfence)
- [The rules a recipe needs](#the-rules-a-recipe-needs)
- [Recipe format](#recipe-format)
- [Files and references](#files-and-references)
- [Step types](#step-types)
- [Script steps](#script-steps)
- [Dashboard templates](#dashboard-templates)
- [Examples](#examples)
- [Commands](#commands)
- [Scheduling](#scheduling)
- [Limits](#limits)
- [Security notes](#security-notes)
- [Not covered](#not-covered)

## Install

Python 3.11 or newer. Install it in its own virtual environment, together with anything your
scripts import (for example `openpyxl` to read Excel files):

```sh
python3 -m venv ~/.venvs/privacyfence-sync
~/.venvs/privacyfence-sync/bin/pip install privacyfence-sync openpyxl
```

Its version matches the PrivacyFence release it ships with. It needs no local PrivacyFence install
of its own: in organization mode it talks to your organization's server over HTTPS.

## Connecting to PrivacyFence

### Local mode (PrivacyFence on your own computer)

1. Get your AI client token: choose **Copy AI Client Token** in the PrivacyFence companion, or run
   `privacyfence-app --print-mcp-token`. It is the same token Claude Code uses.
2. Put it in the environment variable `PRIVACYFENCE_TOKEN` of the process that runs
   `privacyfence-sync`. A token is never accepted as a command-line argument, because other users
   on the machine can read the process list.
3. The server address defaults to `http://127.0.0.1:8765/mcp`. If you changed PrivacyFence's port,
   pass `--server` or set it in the recipe.

### Organization mode (a PrivacyFence server)

1. Sign in once:

   ```sh
   privacyfence-sync login --server https://privacyfence.example.com/mcp
   ```

   A browser opens on your organization's sign-in page. After you sign in, the tab says "Signed in
   to PrivacyFence" and the terminal prints `Signed in to https://privacyfence.example.com/mcp.`
2. Runs then use the stored sign-in automatically, as long as `PRIVACYFENCE_TOKEN` is not set.
3. A sign-in lasts 30 days. After that, a run fails with exit code 4 and the message
   `Not signed in to <server>, or the sign-in expired. Run: privacyfence-sync login --server <server>`.
   Sign in again once a month.

The sign-in is stored in `~/.config/privacyfence-sync/<hash of the server URL>.json` (or under
`$XDG_CONFIG_HOME`), readable only by you. `login` listens on `http://127.0.0.1:33418/callback` for
the browser's return; pass `--callback-port` if that port is taken. The port is registered with
the server at the first sign-in, so keep using the same one.

The server must hand out agent download links. That is the default; if your administrator turned
off `download_delivery.agent_links`, steps that receive a file fail with a message saying so.

### Unattended sessions must be allowed

Both modes need `unattended_sessions.enabled` in PrivacyFence's organization config bundle (built
with `build_org_bundle.py --enable-unattended-sessions`). Without it, a run stops before the first
step with:

```
privacyfence-sync: This PrivacyFence does not allow unattended sessions (unattended_sessions.enabled in org_config.json). Ask your administrator, or run with --attended.
```

## The rules a recipe needs

A scheduled run gets no approval cards, so every step needs an auto-accept rule, unless the tool is
one PrivacyFence never asks about (listing tools such as `drive_list_files`). Create rules in
PrivacyFence's **Settings → Auto-accept → Add a rule**. Each one asks for your passkey.

| Recipe step | Scope in Settings | Verbs to tick |
|---|---|---|
| `salesforce_export_report`, `salesforce_run_report` | **Salesforce — report**, the report id | read |
| `jira_export_issues` | **Jira — project**, the project key(s) | search |
| `drive_download_file` | **Drive — folder**, the folder id the file is in | download |
| `drive_list_files` | none needed | |
| `grist_export_records`, `grist_export_sql`, `grist_get_records` | **Grist — document**, the document id | read |
| `grist_update_csv`, `grist_update_records` | **Grist — document** | update |
| `grist_import_csv`, `grist_add_records` | **Grist — document** | create |
| `peek_upload_page` with a `slug` | **Peek — page**, the page's slug | update (new versions) |

Notes:

- **Personal data.** PrivacyFence scans what a read returns for personal data (names, email
  addresses, phone numbers and similar). When it finds some, it asks a person, and an unattended
  run is denied, even when a rule matches. For the tools whose data goes to a file or a page
  rather than to an AI model, the rule can say otherwise. When you add it, tick:
  - **Exports may include personal data** on a Salesforce report, Jira project or Grist document
    rule (it covers `salesforce_export_report`, `jira_export_issues`, `grist_export_records` and
    `grist_export_sql`);
  - **Downloads may include personal data** on a Drive folder rule (it covers
    `drive_download_file` when the file is delivered as a file; on an organization server a small
    file, up to about 75 KB by default, comes back inside the tool result instead, and is still
    scanned);
  - **New versions may include personal data** on a Peek page rule.

  Only you can set these boxes, in Settings, behind your passkey; an AI client cannot propose
  them. `salesforce_run_report`, `grist_get_records` and the other reads that return data in the
  tool result are always scanned.
- **A Jira project rule** covers an export only when every exported issue belongs to an approved
  project. A JQL query that reaches into another project needs that project in the rule too.
- **A Drive folder rule** covers files whose parent folder is the one named. A file in a subfolder
  needs the subfolder's id.
- **A Peek page rule** covers new versions of pages that already exist. Publish the page once by
  hand (or through Claude) to get its slug. Creating a page, changing who can see it, renaming or
  deleting it always needs your approval.
- To find out which step is missing a rule, run the recipe once with `--attended`: each uncovered
  step shows its approval card, with a link, and the card offers the matching "Always allow"
  button where one exists.

## Recipe format

A recipe is one YAML file. Paths in it are relative to the recipe file.

```yaml
version: 1                          # required, always 1
name: sales-mirror                  # required: lowercase letters, digits and dashes, up to 63
server: http://127.0.0.1:8765/mcp   # optional, see "Commands" for the order of precedence
work_dir: work                      # optional, default "work"
steps:                              # required, 1 to 50 steps, run in order
  - id: opps
    tool: salesforce_export_report
    args:
      report_id: 00O000000000001
      cell_format: both
      file_name: opps.csv
```

Every step has an `id` (lowercase letters, digits, `-` and `_`, up to 40 characters, unique in
the recipe) and exactly one of `tool`, `run` or `template`. A step may only refer to steps above
it. Any unknown key is an error, so a typo fails `check` instead of being ignored.

**The work folder** (`work_dir`) is where every file of a run lands. It is created if needed.
Files from the previous run are overwritten, not deleted, and stay there between runs. They hold
the data you read, so keep the folder somewhere only you can read. Your scripts may keep their own
state anywhere else you choose (for example a "last committed load" folder); `privacyfence-sync`
does not touch it.

**Tool arguments** are the tool's own arguments, as listed in PrivacyFence's
[tools reference](https://github.com/privacyfence/privacyfence/blob/main/docs/tools-reference.md).
Several tools take JSON as text (`filters`, `filter`, `args`, `fields`). In a recipe you can write
those as a YAML list or mapping; `privacyfence-sync` sends any list or mapping value as JSON text.
Strings, numbers and `true`/`false` are sent as they are. `reason` is optional: by default each call
says `privacyfence-sync <recipe name>: step <id>`, which is what the audit log and any approval
card show.

## Files and references

Steps pass files to each other by name. These steps produce files:

| Step | Its files | Referred to as |
|---|---|---|
| An export tool | `args.file_name` | `<step id>` |
| `drive_download_file` | `save_as` | `<step id>` |
| Any other tool step with `save_result` | the tool's result, as JSON | `<step id>` |
| A script step | each entry of `outputs`, and `stdout` if given | `<step id>.<output name>` |
| A template step | `output` | `<step id>` |

Two kinds of reference use them:

- **A file reference** (`file:` on an upload step, `data:` on a template step, and `{file:...}` in
  a script step's arguments) names a step, or a script step's output: `file: plan.upsert`.
- **A value reference** reads one value out of an earlier JSON file and puts it into a tool
  argument or a script argument: `${pick.stdout:pick.drive_id}` is the value at the path
  `pick` → `drive_id` in the `stdout` output of step `pick`. Path parts are object keys or list
  indexes (`files.0.id`). When a value reference is the whole string, the value keeps its JSON
  type (a number stays a number); inside a longer string it is inserted as text. A path that does
  not exist fails the step before anything is called.

## Step types

### Export steps

`salesforce_export_report`, `jira_export_issues`, `grist_export_records`, `grist_export_sql`.

```yaml
- id: issues
  tool: jira_export_issues
  args:
    jql: project = OPS AND updated >= -30d
    fields: ["Story Points", "duedate"]
    file_name: issues.csv
```

- `file_name` is required. It must be a plain file name ending in `.csv` or `.json`, which picks
  the format. The file lands in the work folder under that name.
- Don't give `destination_dir`: `privacyfence-sync` sets it.
- How the file arrives depends on the install, and the recipe doesn't change: a PrivacyFence that
  can write to your folders writes it there; a privilege-separated local install or an
  organization server hands out a one-time download link, which `privacyfence-sync` fetches at
  once (the link expires after 5 minutes). Either way the file's SHA-256 is checked against what
  PrivacyFence reported, and the step fails if they differ.
- A `jira_export_issues` query that matches no issue writes no file on PrivacyFence's side;
  `privacyfence-sync` writes a CSV with just the header row (or `[]` for `.json`), so later steps
  still find their input.

What each export writes:

| Tool | Columns | Notes |
|---|---|---|
| `salesforce_export_report` | the report's detail columns, or `columns` | `cell_format: value` (default) writes raw values: dates as YYYY-MM-DD, amounts as numbers, record ids for lookups. `label` writes what Salesforce shows. `both` writes, for each column `C`, the raw value as `C` and the shown text as `C.label`, plus `C.currency` for money columns. A report over 2,000 rows or a grouped report needs `page_by`, a column unique per row. |
| `jira_export_issues` | key, summary, status, issue_type, priority, assignee, reporter, labels, created, updated, url, then `fields` | People are display names. Lists (labels, multi-select fields) are JSON in a CSV cell. |
| `grist_export_records` | every column of the table, including formula columns | `filter` narrows rows by column values. |
| `grist_export_sql` | what the `SELECT` returns | Cells are as Grist stores them: dates as seconds since 1970, true/false as 1/0. Only people who can copy the whole document may run SQL; others get a message pointing to `grist_export_records`. |

The report totals (the aggregates at the bottom of a Salesforce report) are not in the export. To
check your load against them, add a `salesforce_run_report` step with `summary_only: true` and
`save_result`; it returns the groupings and totals, no rows.

### Download steps

`drive_download_file`, for a file such as an Excel workbook in Google Drive.

```yaml
- id: workbook
  tool: drive_download_file
  args:
    file_id: "${pick.stdout:pick.drive_id}"
  save_as: workbook.xlsx
```

- `save_as` is required: the plain file name the download gets in the work folder, whatever the
  file is called in Drive.
- Don't give `destination_dir`: `privacyfence-sync` sets it.
- The file arrives like an export's (written in place, or by one-time link), or, on an
  organization server, inside the tool result for a small file; `privacyfence-sync` handles all
  three.

### Tool steps that save their result

Any tool step except an export or download step (whose file is already its result) can keep the
tool's result as a JSON file for later steps. The name must end in `.json`:

```yaml
- id: listing
  tool: drive_list_files
  args:
    query: "'1AbCdEfGhIjKlMnOpQrStUvWxYz' in parents and trashed = false"
    max_results: 200
  save_result: listing.json
```

Use it for a listing a script picks from, for report totals, or to check a write's counts
afterwards. Remember that a tool result is what an AI model would see: tools that return rows
(`grist_get_records`, `salesforce_run_report` without `summary_only`) are always PII-scanned and
capped in size. For bulk data, use an export.

### Upload steps

`grist_update_csv`, `grist_import_csv`, `peek_upload_page`.

```yaml
- id: load
  tool: grist_update_csv
  file: plan.upsert
  skip_if_empty: true
  args:
    doc_id: 8CAN8gKdxY7z
    table_id: Deals
    key_column: Deal_Id
    create_missing: true
    clear_empty: true
  save_result: load_result.json
```

- `file` names an earlier file; `privacyfence-sync` uploads it through a one-time upload slot and
  passes its `upload_id`. Don't give `local_path`, `upload_id` or `html`.
- `skip_if_empty: true` skips the step when the file has no data rows (a CSV with no row after
  the header that has a non-empty cell, or a JSON `[]`). Without it, `grist_update_csv` and `grist_import_csv` refuse such a file and the
  run fails. A skipped step with `save_result` writes `{"skipped": true}` there, so later steps can
  tell.
- `grist_update_csv` matches each CSV row to the record whose `key_column` has the same value,
  changes only the cells that differ, and with `create_missing: true` adds rows that match nothing.
  The key column must be in the CSV header and unique in the table. An empty cell leaves the value
  unchanged, unless `clear_empty: true`, in which case it clears it. Columns missing from the CSV
  are never touched.
- `grist_import_csv` adds every row as a new record.
- `peek_upload_page` with `slug` uploads a new version of that page; its visibility stays as it is.

### Other tool steps

Any other PrivacyFence tool can be a step, called with its `args`, for example
`grist_add_records`. It produces no file unless it has `save_result`. It needs a rule like any
other step.

### Script steps

```yaml
- id: plan
  run: scripts/datalake.py
  args: [plan, deals, --home, /Users/me/datalake, --source, "{file:opps}", --out, "{work_dir}/plan"]
  outputs:
    upsert: plan/upsert.csv
    flag_out: plan/flag_out.csv
    summary: plan/summary.json
  timeout_seconds: 600        # optional, 1 to 3600, default 600
```

Runs your Python script. See [Script steps](#script-steps) below.

### Template steps

```yaml
- id: page
  template: templates/dashboard.html
  data: dashboard_data        # a file ending in .json
  output: dashboard.html
```

Puts a JSON file into an HTML template. See [Dashboard templates](#dashboard-templates).

## Script steps

A script step runs an ordinary Python script with the same Python as `privacyfence-sync`:

```
python <run> <args...>
```

- **Arguments.** `args` is a list of strings. In each one, `{file:<ref>}` becomes that file's full
  path, `{work_dir}` the work folder, `{recipe_dir}` the recipe's folder, `{output:<name>}` the
  path of one of this step's own `outputs`, and a value reference `${<ref>:<path>}` the value
  (as text). Any other `{name}` or `{name:...}` is an error, caught by `check`, because it is
  almost always a typo; other text, JSON included, is passed as written. Arguments go to the script
  directly, never through a shell.
- **Outputs.** `outputs` maps names (lowercase letters, digits, `-` and `_`) to paths inside the
  work folder (a subfolder is fine, `..` is not). After the script exits with 0, every declared output must exist. `stdout: <file name>`
  also saves what the script prints, as output `stdout`; the printed text is shown in the run's
  log either way.
- **Exit codes.**
  - `0`: the step succeeded; the run continues.
  - `10`: there is nothing to do this time (for example no new source file). The run stops here
    and `privacyfence-sync` exits with 0, printing `[<id>] nothing to do, run stopped`.
  - Anything else: the step failed and the run stops with exit code 2. The last 2,000 characters
    of the script's error output are shown.
- **Guards.** A script is also how a recipe stops before writing when something looks wrong. Put a
  small check after your plan step: it reads the plan's summary and exits non-zero on warnings, a
  row count that moved more than you expect, or an empty parse. Nothing after it runs, and a
  person looks at it.
- **Environment.** The script runs in the work folder and gets your environment **without** any
  variable whose name starts with `PRIVACYFENCE_`, so it never sees the token. It has no
  PrivacyFence access: everything it needs comes in as files, and everything it produces goes out
  as files. It may read and write your own files elsewhere (state, run history).
- The script's own folder is on its import path, so it can import modules that sit next to it.
  The work folder is not, so a file that lands there is never imported by accident.
- `run` must be inside the recipe's folder. Keep the recipe, its scripts and templates in one
  folder (or one git repository).

### Rows deleted at the source

PrivacyFence's Grist connector never deletes a record. A row that disappears from the source
therefore stays in Grist. Keep a Bool flag column such as `In_Report` in the table, diff the source
against your last committed load (or against an export of the Grist table), and write the flag as
`false` for every key the source no longer has. Dashboards filter on the flag, and history
survives.

## Dashboard templates

A template is an HTML file with this placeholder exactly once:

```html
<script>
  const DATA = /*PRIVACYFENCE_SYNC_DATA*/null;
  // ... draw the dashboard from DATA ...
</script>
```

The template step replaces the placeholder with the contents of the `data` JSON file. The data is
escaped so it can never close the `<script>` element, whatever text it holds. The JSON can come
from a `grist_export_sql` export with a `.json` file name (a list of objects, one per row), or from
a script step that shapes it exactly as the template expects.

Claude writes the template once, when the dashboard is designed; each run only refreshes `DATA`.
`grist_export_sql` writes dates as seconds since 1970 and true/false as 1/0, so convert them in the
template's script or in a script step.

The page should be self-contained (inline CSS and JavaScript, or libraries from a CDN your Peek
viewers can reach). Peek takes pages up to 10 MB.

## Examples

Two recipes in the shape of a real datalake: one engine script (`scripts/datalake.py`) that parses
a source, diffs it against the last committed load, writes the CSVs for Grist, and commits the load
once Grist has accepted it.

### A Salesforce report mirrored into Grist

```yaml
version: 1
name: sf-opportunities
steps:
  - id: report
    tool: salesforce_export_report
    args: {report_id: 00O000000000001, cell_format: both, file_name: opps.csv}

  - id: totals
    tool: salesforce_run_report
    args: {report_id: 00O000000000001, summary_only: true}
    save_result: totals.json

  - id: plan
    run: scripts/datalake.py
    args: [plan, sf-opportunities, --home, /Users/me/datalake, --source, "{file:report}",
           --totals, "{file:totals}", --out, "{work_dir}/plan"]
    outputs: {upsert: plan/upsert.csv, flag_out: plan/flag_out.csv, summary: plan/summary.json,
              sync_log: plan/sync_log.csv}

  - id: guard
    run: scripts/guard.py
    args: ["{file:plan.summary}", --max-change, "0.15", --max-flagged-out, "0.10"]

  - id: upsert
    tool: grist_update_csv
    file: plan.upsert
    skip_if_empty: true
    args: {doc_id: 8CAN8gKdxY7z, table_id: Opportunities, key_column: Opportunity_Id,
           create_missing: true, clear_empty: true}
    save_result: upsert_result.json

  - id: flag_out
    tool: grist_update_csv
    file: plan.flag_out
    skip_if_empty: true
    args: {doc_id: 8CAN8gKdxY7z, table_id: Opportunities, key_column: Opportunity_Id}
    save_result: flag_out_result.json

  - id: verify
    run: scripts/datalake.py
    args: [verify, "{file:plan.summary}", "{file:upsert}", "{file:flag_out}"]

  - id: commit
    run: scripts/datalake.py
    args: [commit, sf-opportunities, --home, /Users/me/datalake, --run, "{work_dir}/plan"]

  - id: sync_log
    tool: grist_import_csv
    file: plan.sync_log
    args: {doc_id: 8CAN8gKdxY7z, table_id: Sync_Log}
```

### The newest Excel workbook in a Drive folder

```yaml
version: 1
name: cashflow
steps:
  - id: listing
    tool: drive_list_files
    args: {query: "'1AbCdEfGhIjKlMnOpQrStUvWxYz' in parents and trashed = false", max_results: 200}
    save_result: listing.json

  - id: pick
    run: scripts/datalake.py
    args: [pick, cashflow, --home, /Users/me/datalake, --listing, "{file:listing}", --exit-10-if-current]
    stdout: pick.json

  - id: workbook
    tool: drive_download_file
    args: {file_id: "${pick.stdout:pick.drive_id}"}
    save_as: cashflow.xlsx

  - id: plan
    run: scripts/datalake.py
    args: [plan, cashflow, --home, /Users/me/datalake, --source, "{file:workbook}",
           --source-info, "${pick.stdout:source_info_arg}", --out, "{work_dir}/plan"]
    outputs: {upsert: plan/upsert.csv, flag_out: plan/flag_out.csv, summary: plan/summary.json}

  - id: guard
    run: scripts/guard.py
    args: ["{file:plan.summary}"]

  - id: upsert
    tool: grist_update_csv
    file: plan.upsert
    skip_if_empty: true
    args: {doc_id: 9DbO9hLeyZ8a, table_id: Cashflow_Revenue, key_column: Opportunity_Id,
           create_missing: true, clear_empty: true}

  - id: flag_out
    tool: grist_update_csv
    file: plan.flag_out
    skip_if_empty: true
    args: {doc_id: 9DbO9hLeyZ8a, table_id: Cashflow_Revenue, key_column: Opportunity_Id}

  - id: commit
    run: scripts/datalake.py
    args: [commit, cashflow, --home, /Users/me/datalake, --run, "{work_dir}/plan"]
```

Here `pick` prints its JSON and exits with 10 when the newest workbook is the one already loaded, so
a run with nothing new stops after two cheap steps.

Rules for these: Salesforce report `00O000000000001` (read), Drive folder `1AbCd…` (download), and
each Grist document (update, create), with the personal-data boxes where the data holds names.

### Publishing a dashboard

```yaml
version: 1
name: pipeline-dashboard
steps:
  - id: data
    tool: grist_export_sql
    args:
      doc_id: 8CAN8gKdxY7z
      sql: >-
        SELECT Stage, COUNT(*) AS deals, SUM(Amount_USD) AS amount
        FROM Opportunities WHERE In_Report = 1 GROUP BY Stage
      file_name: data.json

  - id: page
    template: templates/dashboard.html
    data: data
    output: dashboard.html

  - id: publish
    tool: peek_upload_page
    file: page
    args: {slug: pipeline-dashboard}
```

Before the first scheduled run of any recipe:

```sh
privacyfence-sync check recipe.yaml
privacyfence-sync run recipe.yaml --attended   # approve, and add any missing rule
privacyfence-sync run recipe.yaml              # unattended: no card should appear
```

## Commands

```
privacyfence-sync run RECIPE [--server URL] [--attended] [--token-env NAME]
privacyfence-sync login --server URL [--callback-port N]
privacyfence-sync check RECIPE
```

- **`run`** runs every step in order and stops at the first failure. Each step prints a line
  starting with `[<step id>]`.
  - `--attended` skips the unattended declaration. A step that needs approval prints
    `[<id>] waiting for approval: <link>`, waits for your decision (up to 16 minutes), and repeats
    the call once you approve. Use it for a first run and to find missing rules.
  - `--token-env NAME` reads the token from the variable `NAME` instead of `PRIVACYFENCE_TOKEN`.
  - Without a token in the environment, `run` uses the stored sign-in from `login`.
- **`login`** signs in to an organization server, then checks the connection.
- **`check`** validates the recipe, its references and its paths, and prints
  `<name>: <n> step(s), OK`. It makes no network call and runs no script.

The server address is, in order: `--server`, the recipe's `server`, the variable
`PRIVACYFENCE_MCP_URL`, then `http://127.0.0.1:8765/mcp`.

Exit codes:

| Code | Meaning |
|---|---|
| 0 | Every step succeeded, or a script step said there was nothing to do (exit 10) |
| 1 | The recipe, a command-line argument, a reference or a template is wrong |
| 2 | A tool call, a file transfer or a script failed, or the server refused the connection |
| 3 | PrivacyFence denied a step (in an unattended run: no rule covers it) |
| 4 | Not signed in, the sign-in expired, or no token |

Every error prints one line starting with `privacyfence-sync:` to the error output; when a script
failed, the script's own error output is shown just above it. No message ever contains a token.

## Scheduling

Run recipes as your own user, the same user PrivacyFence's token belongs to. Keep one recipe per
source, so a failure in one source doesn't hold up the others, and give each recipe its own folder
with its scripts, templates and work folder. `run` and `template` paths must stay inside the
recipe's folder, and the default `work_dir: work` then lands next to the recipe. Group the folders
by schedule and number them for the order they run in, with the dashboard last:

```
~/datalake/                     a git repository
  .gitignore                    work/  state/  logs/
  daily/
    10-sf-opportunities/
      recipe.yaml
      scripts/plan.py
      work/                     this recipe's run files (not in git)
    20-drive-cashflow/
      recipe.yaml
      scripts/...
    90-dashboard/
      recipe.yaml
      templates/dashboard.html
  hourly/
    ...
  state/                        your scripts' own state, e.g. state/sf-opportunities/
  logs/
```

One wrapper runs every recipe of a schedule, so adding a recipe means adding a folder, not another
scheduler entry. Recipes hand data to each other through Grist, not through each other's work
folders. Code several recipes share can't be linked in, since a symbolic link that points outside
the recipe's folder is refused; make it a small package and install it into the same virtual
environment (`pip install -e ~/datalake/lib`), and keep the scripts in each folder thin.
Keep `~/datalake` out of `~/Documents`, `~/Desktop` and `~/Downloads` on macOS: a background job
is not allowed to read them.

### macOS: launchd and the Keychain

Use a launchd agent, not cron. A launchd calendar job that was due while the Mac slept runs once
when it wakes; cron skips it. (Neither runs a job that was due while the Mac was shut down.)

Keep each recipe's token in your login Keychain, under the recipe's folder name. `-U` replaces an
existing item, so the same command renews an expired token; `-w` with no value asks for the token,
so it stays out of your shell history:

```sh
security add-generic-password -U -s privacyfence-sync -a 10-sf-opportunities -T /usr/bin/security -w
```

`-T /usr/bin/security` lets the wrapper read the item without a prompt. One token for all recipes
works too; store it under each name, or change the lookup below.

The wrapper, `~/bin/pf-sync.sh` (`chmod 700`), runs every recipe of a schedule with its own token,
carries on after a failure, and shows a notification naming the recipes that failed:

```zsh
#!/bin/zsh
set -u
group="${1:-daily}"
S="$HOME/.venvs/privacyfence-sync/bin/privacyfence-sync"
cd "$HOME/datalake" || exit 1

# Just after wake, give PrivacyFence up to two minutes to come up.
for i in {1..24}; do /usr/bin/curl -s -o /dev/null http://127.0.0.1:8765/mcp && break; sleep 5; done

failed=()
for recipe in $group/*/recipe.yaml(N); do
  name="${recipe:h:t}"
  token="$(/usr/bin/security find-generic-password -s privacyfence-sync -a "$name" -w 2>/dev/null)" \
    || { echo "$(date) $name: no token in the Keychain"; failed+="$name(no token)"; continue; }
  echo "$(date) start $name"
  PRIVACYFENCE_TOKEN="$token" "$S" run "$recipe"
  rc=$?
  echo "$(date) end $name rc=$rc"
  [ $rc -ne 0 ] && failed+="$name($rc)"
done

if (( ${#failed} )); then
  /usr/bin/osascript -e "display notification \"${failed[*]}\" with title \"privacyfence-sync: $group\""
  exit 2
fi
```

The agent, `~/Library/LaunchAgents/local.pf-sync.daily.plist`, runs it every day at 06:10. launchd
does not expand `~` or `$HOME`, so write your home folder out, and create the log folder first:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key><string>local.pf-sync.daily</string>
  <key>ProgramArguments</key>
  <array><string>/Users/you/bin/pf-sync.sh</string><string>daily</string></array>
  <key>StartCalendarInterval</key>
  <dict><key>Hour</key><integer>6</integer><key>Minute</key><integer>10</integer></dict>
  <key>StandardOutPath</key><string>/Users/you/datalake/logs/daily.log</string>
  <key>StandardErrorPath</key><string>/Users/you/datalake/logs/daily.log</string>
</dict>
</plist>
```

For weekdays only, make `StartCalendarInterval` an array of five such dicts, each with a `Weekday`
key (1 to 5). Load it, and run it once now to check it:

```sh
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/local.pf-sync.daily.plist
launchctl kickstart -p gui/$(id -u)/local.pf-sync.daily
```

`launchctl bootout gui/$(id -u)/local.pf-sync.daily` turns it off. The job runs only while you are
logged in (a locked screen is fine), which is also when the Keychain and a local PrivacyFence are
available.

### Linux: cron

cron does not catch up on a run that was due while the machine was off or asleep. Keep the token
out of the crontab line, for example in a wrapper script only you can read:

```sh
#!/bin/sh
# ~/bin/datalake-sync.sh, chmod 700
export PRIVACYFENCE_TOKEN='...'
S=~/.venvs/privacyfence-sync/bin/privacyfence-sync
status=0
for r in ~/datalake/daily/*/recipe.yaml; do
  "$S" run "$r" || status=1
done
exit $status
```

```
# crontab -e: every weekday at 06:10
10 6 * * 1-5 ~/bin/datalake-sync.sh >> ~/datalake/last-run.log 2>&1
```

The exit code tells your scheduler or monitoring what happened (see [Commands](#commands)). A run
that exits 3 means a rule is missing, or a step's data changed so that a rule no longer covers it,
for example a new project appearing in a Jira export or a file moved to another Drive folder.

## Limits

These come from PrivacyFence's tools, not from `privacyfence-sync`:

| What | Limit |
|---|---|
| `grist_export_records`, `grist_export_sql` | 50,000 rows per export |
| `salesforce_export_report` | 2,000 rows without `page_by`; with it, up to `salesforce.report_max_pages` runs (50 by default) |
| `jira_export_issues` | 10,000 issues per export; narrow the JQL and export in parts above that |
| Any delivered file | 200 MB (`file_bridge.max_download_bytes`) |
| `grist_update_csv`, `grist_import_csv` | 20,000 rows, 100 columns, 10,000 characters per cell, 50 MB per file; one key column; formula columns can't be written |
| Upload slot | 50 MB, valid 10 minutes |
| `peek_upload_page` | 10 MB per page |
| Download link | one use, 5 minutes |
| Grist writes | add and update only, never delete (see [Rows deleted at the source](#rows-deleted-at-the-source)) |

Grist writes are not atomic. If one fails part-way, the earlier chunks are in the table; running
`grist_update_csv` again with the same file finishes the rest, which is why a commit step belongs
after the writes, not before.

## Security notes

- `privacyfence-sync` imports nothing from PrivacyFence and has no access to its credentials,
  rules or approvals. It can only make the calls your rules allow, like any AI client.
- The token (or the stored sign-in) is the key to everything your rules allow without a card.
  Treat it like a password: environment variables or a file only you can read, never a recipe,
  a crontab line or a shared folder.
- Files are written to the work folder in plain text. They hold whatever the source holds,
  personal data included when your rule allows it.
- Scripts and templates run with your user's rights. Review them like any code; a recipe folder in
  git makes every change a diff you can read.
- Every call is in PrivacyFence's audit log with the recipe and step name in its reason. A call that
  auto-accepted with personal data in it is logged as auto accepted, with the rule's id and the
  personal-data categories found.

## Not covered

- **No branching beyond stopping.** A script can stop a run successfully (exit 10) or fail it, and
  an upload can be skipped when its file is empty. There are no conditions or loops in a recipe.
- **No retries.** A failed step stops the run; the next scheduled run starts again from the top.
  A script that commits state only after the writes makes that safe.
- **No parallel steps.** Steps run one after another.
- **No new Peek pages and no new Grist tables.** A recipe publishes new versions of an existing page
  and writes to existing tables. Creating them is a one-time design step, done with Claude.
- **No per-row tool calls.** A recipe can't call a tool once per row of a file. Bulk writes go
  through `grist_update_csv` and `grist_import_csv`; for clearing values, use `clear_empty`.
