Metadata-Version: 2.5
Name: dactopus
Version: 0.2.0
Summary: Runner of Dactopus: loads sources into ClickHouse, builds the canonical tables, runs checks, serves the models over MCP
Project-URL: Homepage, https://github.com/Dactopus/dactopus
Project-URL: Issues, https://github.com/Dactopus/dactopus/issues
Project-URL: Changelog, https://github.com/Dactopus/dactopus/blob/main/CHANGELOG.md
Author: Alex Kosachev
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: clickhouse,ecommerce,ga4,mcp,shopify
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Requires-Python: >=3.11
Requires-Dist: certifi>=2024.2
Requires-Dist: clickhouse-connect[arrow]>=0.8
Requires-Dist: dactopus-ossie-clickhouse[mcp]~=0.13.0
Requires-Dist: dbt-clickhouse~=1.10.0
Requires-Dist: dbt-core~=1.11.0
Requires-Dist: google-auth>=2.30
Requires-Dist: google-cloud-bigquery-storage[pyarrow]>=2.27
Requires-Dist: pyyaml>=6
Requires-Dist: tzdata>=2026.5
Description-Content-Type: text/markdown

# dactopus

[![CI](https://github.com/Dactopus/dactopus/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/Dactopus/dactopus/actions/workflows/ci.yml)
[![License](https://img.shields.io/github/license/Dactopus/dactopus)](https://github.com/Dactopus/dactopus/blob/main/LICENSE)

The runner of Dactopus: the process a client installs next to their
[ClickHouse](https://clickhouse.com/docs). It loads sources into that
ClickHouse, builds the canonical e-commerce tables, checks them, and
serves their [Apache Ossie](https://github.com/apache/ossie) models to AI
agents over [MCP](https://modelcontextprotocol.io).

A prototype; 0.1.0 is the first release. While the major version is 0, a
minor release may change the command line and the configuration file.

## How the parts fit

| Part | Role |
| --- | --- |
| dactopus (this repository) | Loads GA4 and Shopify, runs the packages, checks, serves |
| [dactopus-data-models](https://github.com/Dactopus/dactopus-data-models) | Canonical entities and source packages (dbt projects) |
| [dactopus-ossie-clickhouse](https://github.com/Dactopus/dactopus-ossie-clickhouse) | Executes Ossie models on ClickHouse; the MCP server |

The runner owns delivery of data to a package's input tables and running
the package. The library owns the input contract and the transformations.
The executor knows nothing of Dactopus.

## Install

Python 3.11 or later and [uv](https://docs.astral.sh/uv/).

```bash
uv tool install dactopus
```

The `dactopus` command gets its own environment with dbt, the dbt
ClickHouse adapter and the executor. The machine needs no ClickHouse
binaries; the runner connects to the client's ClickHouse server over HTTP.

Then, in the folder of the installation:

```bash
dactopus init     # writes dactopus.yaml; edit it
dactopus run      # loads, builds, checks, writes the report
dactopus serve    # the MCP server for an AI agent, over stdio
```

## Commands

| Command | What it does |
| --- | --- |
| `dactopus init` | Writes a commented sample configuration |
| `dactopus run` | Loads the sources, builds the packages, runs the checks, writes the report |
| `dactopus check` | Runs the checks and writes the report |
| `dactopus serve` | Serves the models over MCP |

One configuration file describes an installation: the ClickHouse
connection, the databases for raw input and canonical tables, the store's
time zone, the version of the models library, and the packages to run.
`dactopus init` writes it with a comment on every field. Secrets stay out
of it: the file names the environment variables that hold them.

`store.timezone` is the store's time zone, an IANA name such as
`America/New_York` (Settings > General in the Shopify admin), and the days
of the installation are its days. The runner passes it to every package
that takes a time zone (`shopify_timezone` for Shopify, which the
package's `vars` may not set) and to the executor in `serve`, so "today"
in an agent's question is the store's date. With Shopify configured,
`dactopus run` first reads the shop's zone and refuses a `store.timezone`
that differs; another name of the same zone (`UTC` for `Etc/UTC`) passes. It then counts, in the daily tables of each package that
takes a time zone, the rows whose date is not the day of their time in
`store.timezone`. Such a row means the tables were built in another zone:
the Shopify models are incremental, and a build after the zone changed
would leave the orders it does not rebuild on the days of the old one. The
run stops before loading, with the command that rebuilds the package,
`dactopus run --full-refresh shopify`; the agent keeps answering from the
current tables until then.

`dactopus run` loads each source into its package's input tables, then
runs `dbt build` of each package. A load reads only what changed:

| Source | Read from | Next load |
| --- | --- | --- |
| GA4 | The daily `events_YYYYMMDD` tables of the BigQuery export, with the Storage Read API | The days from the latest loaded one minus `ga4_lookback_days`, which replace what was loaded for them |
| Shopify | Orders with their lines, shipping and refunds, 25 per page, then the products, archived ones and drafts included, and the deletions of products from the store's events, from the GraphQL Admin API | The orders and the products updated from an hour before the latest loaded change of their kind, appended; a deleted product as a version with the time of the deletion |

GA4 reads need Google credentials: a service account key named in the
file, or Application Default Credentials (`gcloud auth
application-default login`). The read sessions run in the configured
project, which pays for them. Shopify needs an app with `read_orders`
and `read_products`, installed on the store, and its client id and
secret; without `read_all_orders` Shopify returns the last 60 days of
orders only.

A load writes into the input tables the models library's `input.sql`
creates, and `input.sql` creates a table only if it is missing. When a
release of the library adds a column to a table an earlier one created,
the load stops before reading anything and names the table, the column
and the library's changelog, which says how to add it: the Shopify
refunds' `processed_at` of v0.7.2, the date a refund counts on.

The runner writes the dbt profile the packages name (`profile: dactopus`)
from the configuration, without the password, which dbt reads from the
environment. The profile, dbt's output and the report go to `.dactopus/`
next to the configuration file.

`dactopus run --full-refresh [PACKAGE ...]` loads the sources as usual,
then builds the named packages (every configured package when none is
named) with `dbt build --full-refresh`: the canonical tables are rebuilt
from the input the runner already holds, which keeps every version of
every row. The input is not reloaded from the start: the usual
incremental load adds only what is new. Run it once when a
release of the models library changes a model's columns (the next
incremental build stops with an error, and the report names the table and
this command) or when the store's time zone changes. Each table exists
twice on disk until its rebuild ends: dbt builds the new table next to the
old one and swaps them when it is complete, so `serve` keeps answering
from the old table meanwhile.

## Checks and the report

After the build, `dactopus run` measures the freshness of the input
(`dbt source freshness`, with the thresholds the package sets), runs the
checks and writes the report, to the terminal and to `.dactopus/report.md`.
`dactopus check` does the same over the tables as they are, with `dbt test`
in place of the build. The exit code is 1 when a check failed or could not
run, or when no check ran at all, for cron.

The report answers three questions first: how fresh the data is, whether
every check passed and in words which did not, and how many rows each
canonical table got per day over its last seven days that are over against
the four weeks before. A Shopify day is over at midnight in `store.timezone`;
a GA4 daily table is a day that is over. Every
finding follows below.

The built-in checks:

| Check | Reads | Finding |
| --- | --- | --- |
| `dbt` | `run_results.json` and `manifest.json` of the last dbt run | Each model, test and unit test of the package, as a sentence with the count of failing rows |
| `freshness` | `sources.json` of `dbt source freshness` | The age of the latest input row against the package's `warn_after` and `error_after` |
| `rows_per_day` | The canonical tables | A day without rows in the last seven days that are over of a table the package marks as a gap in its `deployment.yml` (`events` for GA4, `orders` for Shopify), unless the same weekday had none in the four weeks before |
| `time_zone` | The daily tables with a time column in `deployment.yml` | Rows whose date is not the day of their time in `store.timezone`: a `fail` for a package built in it (Shopify), a `warn` with the share of rows for one whose days are its source's (GA4, the property's zone), counted over the days `rows_per_day` reads |

A check is a function of a ClickHouse client and the configuration that
returns findings; a finding names the check, the object, a severity
(`pass`, `skip`, `warn`, `fail`, `error`), what was observed, what was
expected, and one sentence for the store owner. A package registers a check
through the entry point group `dactopus.checks`; the entry point's name is
the check's name, and a check that raises becomes an `error` finding
without stopping the others:

```toml
[project.entry-points."dactopus.checks"]
reconciliation = "my_checks:reconcile_with_shopify"
```

## Serving the models

`dactopus serve` runs the MCP server of dactopus-ossie-clickhouse over
stdio, for an AI agent to start as a subprocess. It serves the model
file each configured package names in its `deployment.yml` in the
library, read as one model:

| Package | Model file of the library |
| --- | --- |
| GA4 | `entities/web_analytics.yaml` |
| Shopify | `entities/commerce.yaml` |

The models name their tables without a database; the executor reads them
in `clickhouse.database`. The agent sees and runs only what its ClickHouse
user may: the executor leaves out of the model what the user's grants do
not cover. The user of `clickhouse.url` writes the tables, so give the
agent a user of its own in the `serve` section, on the same host:

```yaml
serve:
  user: agent
  password_env: AGENT_CLICKHOUSE_PASSWORD
```

```sql
CREATE USER agent IDENTIFIED BY '...';
GRANT SELECT ON dactopus.* TO agent;
```

Read grants on the canonical database are enough. Each command reads only
the password it connects with: `serve` needs the variable
`serve.password_env` names, `run` and `check` the one
`clickhouse.password_env` names. Without the `serve` section, `serve` runs
as the user of `clickhouse.url`.

An agent starts the command from a directory of its own and with a short
`PATH`, so give absolute paths to the command and the configuration, and
set the password variable in the server's environment. For
[Claude Code](https://docs.claude.com/en/docs/claude-code/mcp):

```bash
claude mcp add dactopus -e AGENT_CLICKHOUSE_PASSWORD=... -- \
  /ABSOLUTE/PATH/TO/dactopus serve --config /ABSOLUTE/PATH/TO/dactopus.yaml
```

For Claude Desktop, in `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "dactopus": {
      "command": "/ABSOLUTE/PATH/TO/dactopus",
      "args": ["serve", "--config", "/ABSOLUTE/PATH/TO/dactopus.yaml"],
      "env": {"AGENT_CLICKHOUSE_PASSWORD": "..."}
    }
  }
}
```

The tools the agent gets and how to check the server without one:
[MCP setup](https://github.com/Dactopus/dactopus-ossie-clickhouse/blob/main/docs/mcp-setup.md)
of the executor.

## Scope

The runner does not:

- load sources other than GA4 and Shopify, or run a package it has no
  loader for: only `ga4` and `shopify` are accepted under `packages`. A
  database (WooCommerce on MySQL) is to reach ClickHouse by CDC the
  client sets up, with the runner reading the tables it finds;
- schedule runs, send alerts or keep a run history; runs are started by
  hand or by cron;
- link GA4 sessions to Shopify orders; the two models answer apart;
- migrate a client's dbt or Looker;
- ship as a Docker image or an installer.

It was accepted on the public GA4 sample, on a 2026 GA4 export with
synthetic e-commerce events and on a development store with a scripted
history of 440 orders; a store with tens of thousands of orders
and a server that terminates TLS are untested. The decisions behind the
runner and what each was verified on:
[docs/design.md](https://github.com/Dactopus/dactopus/blob/main/docs/design.md).

## Development

Python 3.11 or later and [uv](https://docs.astral.sh/uv/).

```bash
uv sync
uv run pytest
uv run ruff check src tests
```

See [CONTRIBUTING.md](https://github.com/Dactopus/dactopus/blob/main/CONTRIBUTING.md)
and [AGENTS.md](https://github.com/Dactopus/dactopus/blob/main/AGENTS.md).

## License

[Apache 2.0](https://github.com/Dactopus/dactopus/blob/main/LICENSE).
