Metadata-Version: 2.4
Name: trustops-security-data-lake
Version: 0.2.8
Summary: Open-source, self-hosted trust operations — SOC 2, NIST AI RMF, FedRAMP, ISO, HIPAA, PCI DSS. Read-only evidence collection, deterministic control tests, and audit-ready proof with an API-first, MCP-native design.
Author: Mohamed Saad
License: Apache-2.0
Project-URL: Homepage, https://github.com/msaad00/trustops-security-data-lake
Project-URL: Documentation, https://github.com/msaad00/trustops-security-data-lake/tree/main/docs
Project-URL: Repository, https://github.com/msaad00/trustops-security-data-lake
Project-URL: Changelog, https://github.com/msaad00/trustops-security-data-lake/blob/main/CHANGELOG.md
Project-URL: Bug Tracker, https://github.com/msaad00/trustops-security-data-lake/issues
Keywords: compliance,soc2,nist,fedramp,hipaa,pci-dss,iso27001,grc,devsecops,audit,security,evidence,controls,mcp,agents,data-lake
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: server
Requires-Dist: fastapi>=0.141.1; extra == "server"
Requires-Dist: uvicorn>=0.30; extra == "server"
Requires-Dist: sqlalchemy>=2.0; extra == "server"
Requires-Dist: alembic>=1.13; extra == "server"
Requires-Dist: psycopg[binary]>=3.1; extra == "server"
Requires-Dist: authlib>=1.3; extra == "server"
Requires-Dist: itsdangerous>=2.2; extra == "server"
Requires-Dist: python3-saml>=1.16; extra == "server"
Requires-Dist: reportlab>=5.0.1; extra == "server"
Requires-Dist: redis>=8.1.0; extra == "server"
Provides-Extra: dev
Requires-Dist: jsonschema>=4.23; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.8.0; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Requires-Dist: pre-commit>=3.7; extra == "dev"
Requires-Dist: commitizen>=3.27; extra == "dev"
Requires-Dist: fastapi>=0.141.1; extra == "dev"
Requires-Dist: uvicorn>=0.30; extra == "dev"
Requires-Dist: httpx>=0.28.1; extra == "dev"
Requires-Dist: sqlalchemy>=2.0; extra == "dev"
Requires-Dist: alembic>=1.13; extra == "dev"
Requires-Dist: authlib>=1.3; extra == "dev"
Requires-Dist: itsdangerous>=2.2; extra == "dev"
Requires-Dist: python3-saml>=1.16; extra == "dev"
Requires-Dist: reportlab>=5.0.1; extra == "dev"
Requires-Dist: redis>=8.1.0; extra == "dev"
Requires-Dist: pandas>=2.0; extra == "dev"
Provides-Extra: analytics
Requires-Dist: duckdb>=1.0; extra == "analytics"
Provides-Extra: cloud
Requires-Dist: boto3>=1.34; extra == "cloud"
Requires-Dist: azure-identity>=1.17; extra == "cloud"
Requires-Dist: azure-mgmt-authorization>=4.0; extra == "cloud"
Requires-Dist: azure-mgmt-resource<24,>=23.0; extra == "cloud"
Requires-Dist: google-cloud-asset>=3.20; extra == "cloud"
Requires-Dist: google-cloud-resource-manager>=1.12; extra == "cloud"
Requires-Dist: google-cloud-org-policy>=1.11; extra == "cloud"
Requires-Dist: snowflake-connector-python[pandas]>=4.7.1; extra == "cloud"
Requires-Dist: clickhouse-connect>=0.7; extra == "cloud"
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.2.0; extra == "mcp"
Provides-Extra: sdk
Requires-Dist: httpx>=0.28.1; extra == "sdk"
Provides-Extra: agents
Requires-Dist: langgraph>=0.2.0; extra == "agents"
Dynamic: license-file

<p align="center">
  <img src="docs/images/trustops-readme-banner.svg" alt="TrustOps — Collect. Evaluate. Resolve. Export." width="100%">
</p>

<p align="center"><strong>Open, self-hosted GRC for cloud and AI.</strong></p>

<p align="center">
  <a href="https://pypi.org/project/trustops-security-data-lake/"><img src="https://img.shields.io/pypi/v/trustops-security-data-lake?color=2b7bba&label=PyPI" alt="PyPI version"></a>
  <a href="https://pypi.org/project/trustops-security-data-lake/"><img src="https://img.shields.io/badge/python-3.11%2B-blue" alt="Python 3.11+"></a>
  <a href="https://github.com/msaad00/trustops-security-data-lake/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/msaad00/trustops-security-data-lake/ci.yml?branch=main&amp;label=CI" alt="CI status"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue" alt="License: Apache 2.0"></a>
</p>

<p align="center">
  <a href="#quick-start">Quick start</a> ·
  <a href="#how-it-works">How it works</a> ·
  <a href="#explore">Explore</a> ·
  <a href="#develop-and-verify">Develop & verify</a>
</p>

TrustOps collects security evidence, evaluates controls, tracks follow-up work,
and exports assessments for review. **Built for humans and agents:** use the
console for investigation and review, or API, CLI, MCP, and CI for automation.
Deploy it in your environment. Data access and egress depend on your configured
connectors, sinks, and model integrations.

## Quick start

**Try the console with fixture data.** From a cloned repository, use Python 3.11+,
[uv](https://docs.astral.sh/uv/), and Node 22+:

```bash
uv sync --frozen --extra dev --extra server
make demo-local
```

Open [localhost:8787/console/dashboard/](http://127.0.0.1:8787/console/dashboard/).
The command builds the console, loads the golden fixture, migrates the local
database, and starts the server. This local demo disables authentication; use
[authenticated deployment](deploy/README.md) for a shared environment.

<details>
<summary><strong>Other setup paths</strong> — pip, CLI-only, and deployment</summary>

For a source install without uv:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,server]"
make web-install web-build
security-lakehouse fixtures load --company golden --out build/lakehouse
security-lakehouse db upgrade --lake build/lakehouse
security-lakehouse serve --lake build/lakehouse --server --allow-insecure-no-auth --port 8787
```

For the CLI and local lake only:

```bash
pip install trustops-security-data-lake
security-lakehouse fixtures load --company golden --out ./lake
```

[Docker, Helm, and production configuration](deploy/README.md) ·
[Server authentication](docs/SERVER_AUTH.md)

</details>

## How it works

| Step         | What you do                                          | What you get                                        |
| ------------ | ---------------------------------------------------- | --------------------------------------------------- |
| **Collect**  | Connect a source with read-only access.              | Evidence with source, freshness, and provenance.    |
| **Evaluate** | Apply deterministic control rules.                   | Results tied to evidence and the evaluated catalog. |
| **Resolve**  | Assign findings, track fixes, and review exceptions. | Ownership and a record of follow-up decisions.      |
| **Export**   | Freeze an assessment and share reports.              | Evidence and assessment history for reviewers.      |

The [Common Control Framework](docs/COMMON_CONTROL_FRAMEWORK.md) reuses safeguards
across framework mappings. A mapping does not itself establish compliance.
Models may summarize or propose actions; deterministic rules decide control results.

## Explore

<details>
<summary><strong>01 · Product tour</strong> — overview, evidence, and audit room</summary>

The images below show the stored demo fixture, not live customer evidence.

|                              Overview                              |                             Evidence                              |
| :----------------------------------------------------------------: | :---------------------------------------------------------------: |
| ![TrustOps demo overview](docs/images/trustops-demo-dashboard.png) | ![TrustOps demo evidence](docs/images/trustops-demo-evidence.png) |

[Full walkthrough](docs/PRODUCT_WALKTHROUGH.md) ·
[Connectors](docs/images/trustops-demo-connectors.png) ·
[Frameworks](docs/images/trustops-demo-frameworks.png) ·
[Findings](docs/images/trustops-demo-findings.png) ·
[Triage](docs/images/trustops-demo-triage.png) ·
[Remediation](docs/images/trustops-demo-remediation.png) ·
[Audit room](docs/images/trustops-demo-audit-room.png) ·
[Workflows](docs/images/trustops-demo-workflows.png)

</details>

<details>
<summary><strong>02 · Connect sources</strong> — cloud, identity, code, and existing lakes</summary>

In the console, open **Connectors → choose a source → Discover → Test → Enable → Sync**.
No pre-existing data lake is required. For automation, use the
[headless setup playbook](docs/playbooks/HEADLESS_CONNECTOR_SETUP.md).

Sources include AWS, Azure, GCP, GitHub, GitLab, Okta, Snowflake, and ClickHouse.
Check the [connector catalog](docs/CONNECTORS.md) for each integration's scope and status.

No connector requires pasted long-lived cloud keys.

- **AWS** uses STS AssumeRole, one External ID per deployed role, short-lived session credentials, and read-only IAM posture APIs. Temporary credentials expire after each session; TrustOps stores no long-lived access keys. Scale rollout with CloudFormation StackSets or Terraform workspaces; Bulk account import is the next operator surface. See the [cloud setup guide](docs/LIVE_CLOUD_POC.md).
- **Azure** supports a customer-owned Entra application, managed identity, or federated workload identity with Reader scope.
- **Snowflake** uses a read-only service identity with a key-pair or OAuth token reference. TrustOps stores identifiers, not passwords or private-key contents. Snowflake is the existing security-data-lake path.

[AWS credential lifecycle diagram](docs/images/trustops-aws-sts-lifecycle.svg) ·
[Continuous ingestion](docs/CONTINUOUS_INGESTION.md)

</details>

<details>
<summary><strong>03 · Controls and workflows</strong> — safeguards, frameworks, and ownership</summary>

| Area                                                 | Read more                                                                                       |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Safeguards and executable rules                      | [Common Control Framework](docs/COMMON_CONTROL_FRAMEWORK.md)                                    |
| Framework mappings and coverage                      | [Framework coverage](docs/FRAMEWORK_COVERAGE.md)                                                |
| Findings, reviews, exceptions, and audit preparation | [Product walkthrough](docs/PRODUCT_WALKTHROUGH.md) · [Audit readiness](docs/AUDIT_READINESS.md) |
| Implemented, partial, and planned capabilities       | [Product status](docs/PRODUCT_SHAPE.md) · [Roadmap](ROADMAP.md)                                 |

Catalog size, mapped coverage, and evaluated results are separate measures. Inspect
the current safeguard catalog with `security-lakehouse frameworks safeguards`.

</details>

<details>
<summary><strong>04 · Deployment and interoperability</strong> — local, cloud, and evidence storage</summary>

```text
Source → Raw evidence → Normalized facts → Control evaluation → Assessment
                                                ↓                  ↓
                                           Owned findings    Review / export
```

| Layer                   | Current boundary                                                         |
| ----------------------- | ------------------------------------------------------------------------ |
| Evidence and evaluation | Local JSONL, deterministic rules, and verified assessment generations.   |
| Local analytics         | SQLite mart; DuckDB is optional.                                         |
| Operational state       | Application database and local state for jobs, assignments, and reviews. |
| External storage        | Snowflake and ClickHouse integrations; verify the configured deployment. |
| Open table formats      | Parquet, Iceberg REST, and Polaris interoperability are planned.         |

Run locally with Python or Docker, or deploy the Helm chart in your own cloud.
The current assessment writer needs durable local POSIX storage and one writer
per lake. Snowflake and ClickHouse integrations are evidence backends; they do
not host the TrustOps application. Snowflake Native App packaging and Databricks
support are planned. See the [deployment guide](deploy/README.md).

[Architecture guide](docs/ARCHITECTURE.md) ·
[Architecture diagram](docs/images/trustops-assessment-architecture.svg) ·
[Assessment publication and failure contracts](docs/ASSESSMENT_GENERATIONS.md)

</details>

<details>
<summary><strong>05 · API, agents, and CI</strong> — use the same assessment engine headlessly</summary>

| Surface                                 | Purpose                                                          |
| --------------------------------------- | ---------------------------------------------------------------- |
| Console                                 | Browse posture, evidence, findings, and reviews.                 |
| [API](docs/api/AGENT_API.md)            | Versioned `/api/v1` access for integrations.                     |
| CLI                                     | Collect, evaluate, verify, export, and run the local server.     |
| [MCP](docs/HEADLESS_GRC.md)             | Read assessments and propose actions through governed tools.     |
| [CI](docs/playbooks/CI_POSTURE_GATE.md) | Apply posture and control-test thresholds to delivery workflows. |

[TrustOps operator skill](agent-skills/trustops-operator/SKILL.md) ·
[Specialist skills](agent-skills/FRAMEWORK_SKILLS.md) ·
[Agent workflow catalog](docs/api/AGENT_SKILLS.md) ·
[AI bill of materials](docs/AIBOM.md)

</details>

## Develop and verify

<details>
<summary><strong>Checks, repository layout, and documentation</strong></summary>

```bash
make smoke       # backend, contracts, docs, brand, pipeline, API
make web-ci      # install, typecheck, production build
make security    # dependency audits and pre-commit checks
```

Regenerate fixture screenshots with `make demo-screenshots-full`.

| Directory                               | Contents                                           |
| --------------------------------------- | -------------------------------------------------- |
| `src/security_lakehouse/`               | Assessment engine, API, auth, connectors, and MCP. |
| `app/web/`                              | Next.js console.                                   |
| `controls/`, `frameworks/`, `mappings/` | Rules, framework catalogs, and mappings.           |
| `deploy/`                               | Deployment and infrastructure examples.            |
| `docs/`                                 | Product, architecture, operations, and API guides. |

[Validation and benchmark plan](docs/BENCHMARKS.md) ·
[Deployment](docs/DEPLOYMENT.md) · [Roadmap](ROADMAP.md) ·
[Third-party assets](docs/THIRD_PARTY_ASSETS.md)

</details>

[Apache-2.0 license](LICENSE).
