Metadata-Version: 2.4
Name: cortexdb-connectors
Version: 0.3.0
Summary: Data connectors for CortexDB — ingest business-system data into the v1 memory API.
Author-email: CortexDB Team <team@cortexdb.ai>
License-Expression: Apache-2.0
Project-URL: Homepage, https://cortexdb.ai
Project-URL: Documentation, https://cortexdb.ai/docs/connectors/slack
Project-URL: Support, https://cortexdb.ai/contact
Keywords: ai,memory,connectors,etl,slack,github,jira,cortexdb
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.0
Requires-Dist: croniter>=2.0
Provides-Extra: slack
Requires-Dist: slack-sdk>=3.0; extra == "slack"
Provides-Extra: github
Requires-Dist: pygithub>=2.0; extra == "github"
Provides-Extra: pagerduty
Requires-Dist: pdpyras>=5.0; extra == "pagerduty"
Provides-Extra: jira
Requires-Dist: jira>=3.0; extra == "jira"
Requires-Dist: tzdata; extra == "jira"
Provides-Extra: confluence
Requires-Dist: atlassian-python-api>=3.0; extra == "confluence"
Provides-Extra: notion
Requires-Dist: notion-client>=2.0; extra == "notion"
Provides-Extra: linear
Provides-Extra: discord
Requires-Dist: aiohttp>=3.9; extra == "discord"
Provides-Extra: teams
Requires-Dist: msal>=1.28; extra == "teams"
Provides-Extra: google-workspace
Requires-Dist: google-api-python-client>=2.0; extra == "google-workspace"
Requires-Dist: google-auth>=2.0; extra == "google-workspace"
Provides-Extra: salesforce
Requires-Dist: simple-salesforce>=1.12; extra == "salesforce"
Provides-Extra: hubspot
Requires-Dist: hubspot-api-client>=9.0; extra == "hubspot"
Provides-Extra: zendesk
Provides-Extra: intercom
Provides-Extra: servicenow
Provides-Extra: gitlab
Requires-Dist: python-gitlab>=4.0; extra == "gitlab"
Provides-Extra: webhooks
Requires-Dist: starlette>=0.37; extra == "webhooks"
Requires-Dist: uvicorn>=0.29; extra == "webhooks"
Requires-Dist: python-multipart>=0.0.9; extra == "webhooks"
Provides-Extra: all
Requires-Dist: slack-sdk>=3.0; extra == "all"
Requires-Dist: pygithub>=2.0; extra == "all"
Requires-Dist: pdpyras>=5.0; extra == "all"
Requires-Dist: jira>=3.0; extra == "all"
Requires-Dist: tzdata; extra == "all"
Requires-Dist: atlassian-python-api>=3.0; extra == "all"
Requires-Dist: notion-client>=2.0; extra == "all"
Requires-Dist: aiohttp>=3.9; extra == "all"
Requires-Dist: msal>=1.28; extra == "all"
Requires-Dist: google-api-python-client>=2.0; extra == "all"
Requires-Dist: google-auth>=2.0; extra == "all"
Requires-Dist: simple-salesforce>=1.12; extra == "all"
Requires-Dist: hubspot-api-client>=9.0; extra == "all"
Requires-Dist: python-gitlab>=4.0; extra == "all"
Requires-Dist: PyYAML>=6.0; extra == "all"
Requires-Dist: starlette>=0.37; extra == "all"
Requires-Dist: uvicorn>=0.29; extra == "all"
Requires-Dist: python-multipart>=0.0.9; extra == "all"
Dynamic: license-file

# cortexdb-connectors

Managed data connectors for [CortexDB](https://cortexdb.ai). Run them yourself, or let CortexDB host them for you on the [Starter tier](https://cortexdb.ai/pricing).

Each connector pulls from a third-party system (Slack, GitHub, Jira, …) and writes every event into CortexDB as a v1 experience.

## Plan availability (audit BLK-3)

| Mode | Plan | What it means |
|---|---|---|
| **Self-hosted** (`cortexdb-sync` CLI) | 🟢 **Free** (this package) | You run the poller / webhook receiver yourself. Cursor state lives in `~/.cortexdb/`. Every connector below works in this mode. |
| **Managed sync** (dashboard) | 🔒 **Starter+** ($29/mo) | CortexDB hosts the syncer, manages credentials, and surfaces health in the dashboard. The "Add Connector" button in the dashboard is gated on this plan. |

The free self-hosted path is always available; the managed dashboard path is what the pricing page calls "Starter+". The badge below appears on each connector's setup page for the same reason.

## Install

```bash
# core only — no third-party SDK dependencies
pip install cortexdb-connectors

# with one source's SDK
pip install 'cortexdb-connectors[slack]'

# everything
pip install 'cortexdb-connectors[all]'
```

Requires Python 3.10+.

## Auth

The connectors talk to the CortexDB v1 API. Two things you need:

- **A PASETO bearer token** (`Authorization: Bearer …`)
- **An actor id** that matches the token's `sub` claim (`X-Cortex-Actor: …`)

The fastest way to get both is to install [`cortexdb-cli`](https://pypi.org/project/cortexdb-cli/) and run `cortexdb init` — anonymous signup, no email or card, 7-day free-tier token. The connectors read `~/.cortexdb/state.json` automatically.

Or set env vars:

```bash
export CORTEXDB_URL=https://api-v1.cortexdb.ai
export CORTEXDB_API_KEY=v4.public...
export CORTEXDB_ACTOR=user:u_019e...
export CORTEXDB_TENANT_ID=acme                            # optional assertion; verified via whoami
export CORTEXDB_SCOPE_TEMPLATE='org:acme/source:{source}'   # optional
export CORTEXDB_SOURCE_INSTANCE_ID='acme-slack-workspace'   # see below
```

`{source}` (and `{scope}`, `{namespace}`, `{actor}`, `{entity.<type>}`) get filled in per event. (`{tenant}` is still accepted as a legacy alias of `{scope}`.)

**Self-hosted server with a static `CORTEX_API_KEY`:** that key has no `sub`
claim, and the server runs a request without `X-Cortex-Actor` as `user:local`.
When `CORTEXDB_ACTOR` is unset, `cortexdb-sync sync|watch` and the Docker
worker ask `GET /v1/auth/whoami` (without the header) and adopt `user:local`
only when the server confirms a static key (0.2.22+). With a PASETO/JWT token
the actor stays explicit and must equal the token's `sub`. When no actor can
be determined the sync stops before fetching anything, with one error naming
`CORTEXDB_ACTOR`, and the cursor is left untouched.

`CORTEXDB_SOURCE_INSTANCE_ID` is an optional stable, non-secret account/site
identifier. Set it (or `--source-instance-id`) whenever two accounts of the
same provider write to the same CortexDB endpoint, actor, and scope. Its digest
partitions canonical source ids and the lifecycle ledger; never put an access
token or other credential in it. The empty default preserves existing
single-account identities. Connector-specific YAML may set
`source_instance_id` under that connector as well.

Before it opens durable lifecycle state, the connector resolves the billing
tenant from `GET /v1/auth/whoami`. `CORTEXDB_TENANT_ID` (or
`--cortex-tenant-id`) is an optional expected value and must match that
authenticated response; it never bypasses verification. Tenant identity is
part of the opaque ledger namespace, so credentials for a different tenant
cannot reuse completed source rows.

## Run

```bash
# one shot
cortexdb-sync sync slack

# poll loop, 60s between cycles
cortexdb-sync watch slack --interval 60

# resume from a specific point
cortexdb-sync sync github --since 2026-05-01T00:00:00Z

# see what's available
cortexdb-sync list

# check resolved CortexDB creds
cortexdb-sync auth

# cursor state across every connector
cortexdb-sync status

# move a tenant's items onto containers, once (see "Container mode")
cortexdb-sync migrate-containers slack --dry-run

# give up an incomplete migration (its unmoved items stay listed)
cortexdb-sync migrate-containers slack --abandon
```

## Available connectors

All 18 connectors are **🟢 Free** to self-host and **🔒 Starter+** ($29/mo) to use via the managed dashboard.

```
slack            SLACK_BOT_TOKEN
github           GITHUB_TOKEN, GITHUB_REPOS
gitlab           GITLAB_TOKEN
jira             JIRA_URL, JIRA_EMAIL, JIRA_API_TOKEN   (incl. crash-safe keyed attachments + multimodal extraction, 0.2.10+)
freshdesk        FRESHDESK_DOMAIN, FRESHDESK_API_KEY    (incl. crash-safe keyed attachments + multimodal extraction, 0.2.20+)
tldv             TLDV_API_KEY
linear           LINEAR_API_KEY
confluence       CONFLUENCE_URL, CONFLUENCE_EMAIL, CONFLUENCE_API_TOKEN
notion           NOTION_TOKEN
pagerduty        PAGERDUTY_API_KEY
discord          DISCORD_BOT_TOKEN
teams            TEAMS_TENANT_ID, TEAMS_CLIENT_ID, TEAMS_CLIENT_SECRET
google-workspace GW_SERVICE_ACCOUNT_KEY, GW_DELEGATED_USER
salesforce       SF_INSTANCE_URL, SF_CLIENT_ID, SF_CLIENT_SECRET, SF_USERNAME, SF_PASSWORD
hubspot          HUBSPOT_TOKEN
zendesk          ZENDESK_SUBDOMAIN, ZENDESK_EMAIL, ZENDESK_TOKEN
intercom         INTERCOM_TOKEN
servicenow       SNOW_INSTANCE, SNOW_USERNAME, SNOW_PASSWORD
```

`cortexdb-sync list` prints the same required variables plus each connector's optional ones.

### GitHub

```bash
export GITHUB_TOKEN=ghp_...                  # PAT or GitHub App installation token
export GITHUB_REPOS=acme/api,acme/web        # required: owner/repo, comma-separated (no discovery)
export GITHUB_EVENTS=pull_request,issues     # optional; default: all six kinds below
export GITHUB_BACKFILL_DAYS=30               # optional; first-sync history, 0 disables
cortexdb-sync sync github
```

- **What is stored (0.2.22+):** PRs as `PR #n [state]: title` plus the full
  description; issues with their full body; issue comments, review bodies and
  inline review comments (file, line and diff hunk) in full; pushes with the
  newest 20 commit messages and the total count. Nothing is truncated — text
  over 1 MiB is split losslessly into source-bound fragments.
- **`GITHUB_EVENTS` kinds:** `push`, `pull_request`, `issues`,
  `issue_comment`, `pull_request_review`, `pull_request_review_comment`. Raw
  Events API names (`PullRequestEvent`, `PullRequestReviewCommentEvent`, …)
  are accepted too, case-insensitively.
- **Sources:** PRs, issues, issue comments and review comments are read from
  the REST list endpoints on every sync (the first sync reaches back
  `GITHUB_BACKFILL_DAYS`, default 30, as does the first sync of a
  repository added to `GITHUB_REPOS` later); the repository Events feed, which
  GitHub caps at 300 events / 30 days, supplies pushes and reviews. Each
  record is keyed by the GitHub object — the same identity the webhook
  receiver uses — so a PR re-read by the poller or delivered by a webhook
  updates one record instead of adding another.
- **Persist `~/.cortexdb` between runs:** it holds the cursor (with the
  work a run deferred — pushes and PR file lists / review listings held
  when the API budget ran low, and a new repository's backfill items that
  failed to ingest — retried on the next sync) and the
  lifecycle ledger that retires superseded versions (or point
  `CORTEXDB_STATE_DIR` at a volume for the ledger, and `--state-file` /
  the worker's `CORTEX_SYNC_STATE_PATH` for the cursor). A throwaway
  `docker run --rm` loses both.

Per-connector setup pages live at https://cortexdb.ai/docs/connectors/ (e.g. [Slack](https://cortexdb.ai/docs/connectors/slack)). **The "Add Connector" button on those pages is the managed (Starter+) path** — the CLI on this page is the free path.

For deep self-hosted operation — auth lifetimes, cursor state, retries, OAuth setup for Salesforce/Teams/Google Workspace — see [`SELF_HOSTING.md`](./SELF_HOSTING.md) in this directory.

## YAML config

If env vars get unwieldy, drop a `cortexdb-connectors.yaml` next to your sync invocation:

```yaml
slack:
  slack_bot_token: xoxb-...
  source_instance_id: acme-slack-workspace
  channels: [C01ABCDEF, C02GHIJKL]

github:
  github_token: ghp_...
  repos: [acme/api, acme/web]
  events: [pull_request, issue_comment]
```

## Programmatic use

Skip the CLI entirely:

```python
import asyncio
from cortexdb_connectors.slack import SlackConnector

connector = SlackConnector(
    cortex_url="https://api-v1.cortexdb.ai",
    cortex_api_key="v4.public...",
    scope="org:acme/source:slack",
    slack_bot_token="xoxb-...",
    channels=["C01ABC", "C02DEF"],
)
# actor + a richer scope_template are set after construction:
connector.bind(
    actor="user:u_019e...",
    scope_template="org:acme/source:slack/channel:{entity.channel}",
    source_instance_id="acme-slack-workspace",
    cortex_tenant_id="acme",  # optional assertion, verified with whoami
)

result = asyncio.run(connector.sync())
print(result.episodes_ingested, "ingested,", len(result.errors), "errors")
```

## Webhooks

The `cortexdb-connectors[webhooks]` extra ships a Starlette-based webhook receiver for sources that push (Slack Events API, GitHub Apps, Jira webhooks, …) instead of polling. See `cortexdb_connectors/webhooks.py`.

## Durable lifecycle ledger

Every canonical text or blob source is recorded in a local SQLite ledger before
a sync cursor advances or a webhook is acknowledged. It stores only opaque
source/version/idempotency digests, random upload tokens, and CortexDB
event/blob ids—never source text or file bytes—and, in container mode, the
scope each item's events live at, for an item a container migration has not
moved yet its scopes, its raw external id when it failed before it normalized,
and the last error, and for a copy left under another tenant root its scope
and event ids. The default is
`~/.cortexdb/connector_event_ledger.sqlite3` (or under `CORTEXDB_STATE_DIR`);
override it with `CORTEX_CONNECTOR_EVENT_LEDGER_PATH`.
The authenticated billing tenant is included in its opaque namespace and an
optional configured tenant assertion must match `/v1/auth/whoami`.

Keep this database on durable local storage and back it up with SQLite's online
backup API (or stop writers and copy the database together with its `-wal` and
`-shm` files). A replica needs the same ledger to suppress replay beyond the
server idempotency window and to erase every historical source version, so run
one active connector writer per source namespace unless all writers share a
correctly coordinated ledger. Same-source generation changes use bounded,
crash-released OS advisory-lock stripes beside the SQLite file; network
filesystems are supported only when they provide equivalent locking semantics.
Accepted replacements retire older generations through exact derived-then-raw
erasure before the sync cursor advances; a failed cleanup remains durable for
retry. Rows are removed after confirmed source erasure or supersession;
monitor disk usage as part of connector operations.

New binary attachments use only CortexDB's versioned keyed-blob protocol. The
ledger commits a random, non-secret upload token before any blob network request,
then preflights and uploads the bytes through `/v1/blobs/idempotent`. A timeout
or restart reuses that same token and reconciles the authoritative blob ID, so
it cannot create a second physical object. Deletion tombstones the token through
the matching keyed endpoint before removing the local mapping. A missing,
disabled, malformed, or older server implementation is a hard sync failure;
connectors never fall back to an unkeyed upload. Existing blob-ID-only ledger
rows from releases before keyed uploads remain deletable through the legacy
ID route as a migration path, but no new rows are created that way.
Binary payloads over the 32 MiB source ceiling, malformed descriptors, and
invalid envelope metadata are rejected before whoami, ledger allocation, or
upload. Text-shaped attachments are strictly decoded and losslessly fragmented
at UTF-8 boundaries instead of taking the binary path.

## Container mode (source permission units)

For a tenant that is an org tree, set `CORTEXDB_CONTAINER_ROOT` to its root
(`org:acme`) and the Slack, Teams, Discord, Google Drive, Confluence and Notion
connectors write each item into the scope of its source permission unit instead
of the scope template, so who reads it follows the source's own ACL:

- a channel, folder, space or page tree is a container,
  `org:acme/source:<system>/container:<key>`; an item shared differently from
  its folder or page tree goes into a container keyed by the fingerprint of its
  ACL;
- a public unit is read through `org:acme/set:<system>-members` (the source's
  full members), a private one through its own set, and a DM or group DM is a
  private group `org:acme/group:dm-<system>-<key>` read by its participants;
- a source account reaches a roster only once the tenant directory links it to
  a person (`POST /v1/access/accounts/resolve`); one it does not is counted, and
  a unit whose ACL the connector could not read is a container nobody reads.

The directory links accounts per system, so link each person's account under
the system its connector names:

| Connector | Account system | Account |
|---|---|---|
| Slack | `slack:<team id>` | user id |
| Microsoft Teams | `microsoft:<Entra tenant id>` | user object id |
| Discord | `discord` | user id |
| Google Drive | `google:<domain>` | email address, Unicode NFC, lowercased |
| Confluence | `atlassian` | Atlassian account id |
| Notion | `notion` | user id |

The connector's actor (`CORTEXDB_ACTOR`, a `service:` principal) owns every
record it writes, and must be one of the tenant root companion's
`access_sources` (or an Owner of the root) with Reader at the root for
`GET /v1/access/limits`. Each sync registers what changed through change sets,
the narrowing one first; moves an item whose sharing changed
(`POST /v1/access/rehome`; an event the server erased stays erased: it is
counted, and never captured again); leaves out the containers past the source's
`containers_per_source` limit (their items are not ingested, and are counted);
and, when it changed something or the source has no watermark yet, proposes
the source's watermark. The records the connector registered are kept with the
cursor; a first sync, or one whose saved state was lost, reads them back from
the registry (`GET /v1/access/records`) before it writes, so it never creates a
record that exists, and a set an operator filled (`set:<system>-apps`,
`set:<system>-services`) keeps its entries. `cortexdb-sync status` shows each
source's counts. Each connector's module docstring says which ACLs its source
API exposes and what it does when one is missing.

**Items written before container mode.** Within a tenant, a sync never captures
an item fresh while a copy of it is still live at its template scope: such a
copy would keep a private channel's content readable in the shared template
scope, the leak containers exist to close. So every container-mode ingest of an
item with a unit (a sync, a sync's retry of a listed item, a migration, or a
direct `ingest_episode_events` or `ingest_episode` call) first checks whether
the ledger still records the item under its template scope's namespace, whether
or not a migration listed it, and if so moves it as `migrate-containers` does:
`POST /v1/access/rehome` to its home and its ledger rows re-keyed, so the
ingest finds it recorded. Sealed or expired events are
erased at the template scope and captured again at the home, absent ones stay
erased, and a DM message is captured at its group, then its template copy
erased. A copy under another tenant root (a template such as `user:default`
outside the container root) cannot move, since no rehome crosses tenants: the
item is captured at its home and the copy is recorded, with its source id, its
scope and the event ids the ledger holds there, for that tenant's operator to
erase. `cortexdb-sync status` lists these records until the copy is erased
through the connector or an operator clears them with
`cortexdb-sync migrate-containers <connector> --clear-other-tenant-copies
[SOURCE_ID ...]`. Clearing makes the ledger forget the copy and erases nothing
at the server.

**Raising `containers_per_source`.** Items of a container left out at the cap
are not ingested, and the cursor moves past them as past any other item, so
they do not hold every later sync back. After the limit is raised, backfill
them: `cortexdb-sync sync <connector> --since <time of the oldest left-out
item>`. The source's watermark says how many containers it left out.

**Webhooks.** A Slack delivery carries no channel ACL, so the Slack webhook
receiver writes to the scope template, and in container mode it would put
private channels where the template's readers see them. With a container root
and `SLACK_SIGNING_SECRET` set, `cortexdb-sync serve` exits with an error
instead of starting, and a Slack handler bound to a root answers every
delivery 503. Run the poll connector (`sync` or `watch`) for Slack instead. The
GitHub, Jira, Freshdesk and tl;dv receivers serve sources that do not write
containers, and run as before.

### Moving a tenant onto containers

A tenant whose connectors wrote to the scope template moves onto containers
once, with its owner, as a breaking change for its readers:

1. Make the tenant an org tree (its root registered with kind `org`,
   `PUT /v1/scopes/tree`). Walking views at the root, at sources and at
   containers then answer 422 `VIEW_NOT_ALLOWED_AT_SCOPE`, so move the
   tenant's clients to `mounts` or `extra_scopes` first.
2. Set `CORTEXDB_CONTAINER_ROOT` and preview the move:
   `cortexdb-sync migrate-containers <connector> --dry-run`. It reads the
   source, the limits and the registry, prints per home the items and events
   that would move, and writes nothing.
3. Before the first container-mode sync, run
   `cortexdb-sync migrate-containers <connector>` (with the `--scope` the syncs
   use, and `--since` as far back as the first sync when the connector's own
   backfill is shorter: it learns each item's unit from the source). It
   registers the units' records exactly as a sync does, then moves each item's
   events, the ids its ledger holds, to its home with `POST /v1/access/rehome`
   in batches of at most 1,024: the events keep their ids and derived rows, and
   nothing is captured again. The syncs that follow find every moved item
   recorded and post nothing; the first proposes the source's watermark.

What it does not move: an item whose events the server skips as sealed or
expired is erased at the template scope and captured again at its home; a DM
message whose author the directory resolves is captured again at its group,
since its observed actor must be that person for a person's erasure to reach
it; an `absent` event (already erased) is reported and never captured again;
the items of a container left out at the cap stay where they are, listed (raise
the limit: the next sync that reads them moves them, or run it again); items
with no unit keep the template. A copy at an item's home beside its template
copy (written by an earlier connector version's container-mode sync, or a DM
message whose template copy could not be erased) makes the template copy a
duplicate, which it erases. A template scope under another tenant root cannot
be moved from: the run names each such copy (source id, scope, the event ids
the ledger holds there) and records it as a sync does, and the next sync
captures the item at its home. The rehome needs the connector's actor to be
an Owner, with `policy.administer.scope`, of the template scope the items are
at now as well as of the containers. A rerun moves nothing (the server answers
`unchanged`).

The migration moves everything at once; a sync moves only the items it reads.
An item the migration did not read (outside its window) is still moved, not
captured beside its copy, by the first sync that reads it.

A run marks the sync state before it writes anything, and clears the mark only
when it completes: every rehome answered, every moved item's ledger rows
moved with it. While the mark is there, `sync`, `watch` and the worker refuse
to start (an item whose rehome failed is still recorded at the template scope,
and a sync would capture it a second time at its home), naming the command
that finishes the run; `cortexdb-sync status` lists it. Run
`cortexdb-sync migrate-containers <connector>` again: it resumes the window
the interrupted run covered and clears the mark.

An item that fails on its own (it could not be read or planned, captured again,
captured at its DM group, or its template copy could not be erased) does not
hold the syncs: the run reports it under "failed" (its source id, or its raw
external id when it failed before it normalized, its template scope and home,
the stage and the error), exits 1, and keeps it listed in the ledger;
`cortexdb-sync status` shows how many are listed and the first of them. The
next sync that reads a listed item moves it the way the migration would have
(rehome and re-keyed ledger rows, sealed or expired events erased and captured
again at the home, absent ones kept erased; a template copy beside a copy at
the home erased; a DM message captured at its group, then its template copy
erased), never by a fresh capture beside its template copy. Once it has moved
it leaves the list; if the move fails again it stays listed with the new error
and is that sync's failure, holding the cursor as any failed item does. Listed
items a sync does not read stay listed for a later sync or a rerun, and a
rerun clears what it settles.

To give up an incomplete run instead,
`cortexdb-sync migrate-containers <connector> --abandon` clears its mark so
the syncs run again. It reads only the ledger and the sync state (nothing is
fetched from the source or sent to CortexDB), logs how many items the run
leaves at their template scopes and their ids, and keeps them listed, so
`status` shows them and the next sync that reads each one moves it. Until
then, an item of a private unit left at a pooled template scope stays readable
there. Without an incomplete run it says so and changes nothing, and it does
not combine with `--dry-run`.

## License

Apache-2.0
