Metadata-Version: 2.4
Name: metapod-probes
Version: 0.4.0
Summary: Lightweight metadata extraction probes — 191 probes across 47 data sources
Project-URL: Homepage, https://github.com/mario-ambrosino/metapod
Project-URL: Repository, https://github.com/mario-ambrosino/metapod
Author-email: Mario Ambrosino <ambromar@outlook.it>
License: Apache-2.0
License-File: LICENSE
Keywords: data-catalog,data-lineage,dbt,metadata,oracle,probes,snowflake
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: typer[all]>=0.12
Provides-Extra: all
Requires-Dist: azure-identity<2,>=1.15; extra == 'all'
Requires-Dist: azure-storage-blob<13,>=12.19; extra == 'all'
Requires-Dist: azure-storage-file-datalake<13,>=12.14; extra == 'all'
Requires-Dist: azure-storage-file-share<13,>=12.15; extra == 'all'
Requires-Dist: boto3<2,>=1.34; extra == 'all'
Requires-Dist: databricks-sdk<1,>=0.20; extra == 'all'
Requires-Dist: duckdb<2,>=0.10; extra == 'all'
Requires-Dist: elasticsearch<9,>=8.0; extra == 'all'
Requires-Dist: google-cloud-bigquery<4,>=3.0; extra == 'all'
Requires-Dist: oracledb<4,>=2.0; extra == 'all'
Requires-Dist: psycopg2-binary<3,>=2.9; extra == 'all'
Requires-Dist: pyarrow<19,>=14.0; extra == 'all'
Requires-Dist: pymongo<5,>=4.0; extra == 'all'
Requires-Dist: pymysql<2,>=1.1; extra == 'all'
Requires-Dist: snowflake-connector-python<4,>=3.0; extra == 'all'
Provides-Extra: avro
Requires-Dist: avro<2,>=1.11; extra == 'avro'
Provides-Extra: aws
Requires-Dist: boto3<2,>=1.34; extra == 'aws'
Provides-Extra: azure
Requires-Dist: azure-identity<2,>=1.15; extra == 'azure'
Requires-Dist: azure-storage-blob<13,>=12.19; extra == 'azure'
Requires-Dist: azure-storage-file-datalake<13,>=12.14; extra == 'azure'
Requires-Dist: azure-storage-file-share<13,>=12.15; extra == 'azure'
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery<4,>=3.0; extra == 'bigquery'
Provides-Extra: databricks
Requires-Dist: databricks-sdk<1,>=0.20; extra == 'databricks'
Provides-Extra: db2
Requires-Dist: ibm-db<4,>=3.0; extra == 'db2'
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: duckdb
Requires-Dist: duckdb<2,>=0.10; extra == 'duckdb'
Provides-Extra: elasticsearch
Requires-Dist: elasticsearch<9,>=8.0; extra == 'elasticsearch'
Provides-Extra: hana
Requires-Dist: hdbcli<3,>=2.0; extra == 'hana'
Provides-Extra: mongodb
Requires-Dist: pymongo<5,>=4.0; extra == 'mongodb'
Provides-Extra: mysql
Requires-Dist: pymysql<2,>=1.1; extra == 'mysql'
Provides-Extra: oracle
Requires-Dist: oracledb<4,>=2.0; extra == 'oracle'
Provides-Extra: parquet
Requires-Dist: pyarrow<19,>=14.0; extra == 'parquet'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary<3,>=2.9; extra == 'postgres'
Provides-Extra: presidio
Requires-Dist: pandas<3,>=2.0; extra == 'presidio'
Requires-Dist: presidio-analyzer<3,>=2.2; extra == 'presidio'
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python<4,>=3.0; extra == 'snowflake'
Provides-Extra: soda
Requires-Dist: soda-core<4,>=3.3; extra == 'soda'
Provides-Extra: teradata
Requires-Dist: teradatasql<21,>=20.0; extra == 'teradata'
Provides-Extra: trino
Requires-Dist: trino<1,>=0.320; extra == 'trino'
Provides-Extra: ydata
Requires-Dist: pandas<3,>=2.0; extra == 'ydata'
Requires-Dist: ydata-profiling<5,>=4.6; extra == 'ydata'
Description-Content-Type: text/markdown

# metapod

[![PyPI version](https://img.shields.io/pypi/v/metapod-probes.svg)](https://pypi.org/project/metapod-probes/)
[![Python](https://img.shields.io/pypi/pyversions/metapod-probes.svg)](https://pypi.org/project/metapod-probes/)
[![Tests](https://img.shields.io/badge/tests-131%20passing-brightgreen)](https://github.com/mario-ambrosino/metapod)
[![Probes Tested](https://img.shields.io/badge/probes%20validated-73%20real-green)](https://github.com/mario-ambrosino/metapod)
[![Docs](https://readthedocs.org/projects/metapod/badge/?version=latest)](https://metapod.readthedocs.io/)
[![License](https://img.shields.io/pypi/l/metapod-probes.svg)](https://github.com/mario-ambrosino/metapod/blob/main/LICENSE)
[![Probes](https://img.shields.io/badge/probes-185-blue)](https://github.com/mario-ambrosino/metapod)
[![Sources](https://img.shields.io/badge/sources-42-blue)](https://github.com/mario-ambrosino/metapod)

**Lightweight metadata extraction probes.** 185 probes across 42 data sources in 9 categories — metadata, governance, quality, profiling, operations, compliance, cost, schema history, and lineage.

Runs inside your network. Extracts only metadata — no raw data leaves the perimeter.

## Install

```bash
pip install metapod-probes                     # Core (file scanners, BI tools)
pip install 'metapod-probes[oracle]'           # + Oracle
pip install 'metapod-probes[snowflake]'        # + Snowflake
pip install 'metapod-probes[azure]'            # + Azure (ADLS, Blob, ADF)
pip install 'metapod-probes[all]'              # Everything
```

## Quick start

```bash
metapod init                                    # Generate config.yaml
metapod list-probes                             # Show all 155 probes
metapod run "oracle.*" --schema RISK_MGMT       # Extract Oracle metadata
metapod push --target https://your-platform.com --project myproject  # Send results
metapod serve --port 9090                       # Pull-mode daemon
```

## Supported sources (40)

### Databases (15)

| Source | Probes | Driver |
|--------|--------|--------|
| Oracle | 18 | oracledb |
| PostgreSQL | 10 | psycopg2 |
| Snowflake | 5 | snowflake-connector |
| BigQuery | 5 | google-cloud-bigquery |
| SQL Server / Azure SQL | 5 | pyodbc |
| MySQL | 4 | pymysql |
| IBM DB2 | 4 | ibm_db |
| MariaDB | 2 | pymysql |
| SAP HANA | 2 | hdbcli |
| Teradata | 2 | teradatasql |
| DuckDB | 3 | duckdb |
| Trino | 3 | trino |
| SQLite | 2 | stdlib |
| MongoDB | 1 | pymongo |
| Elasticsearch | 1 | elasticsearch |

### Cloud platforms (7)

| Source | Probes | API |
|--------|--------|-----|
| Databricks Unity Catalog | 6 | REST API |
| Azure Data Factory | 5 | Azure Management API |
| ADLS Gen2 | 4 | azure-storage-file-datalake |
| Azure Blob Storage | 3 | azure-storage-blob |
| Azure File Share | 3 | azure-storage-file-share |
| AWS S3 | 3 | boto3 |
| Oracle Cloud (OCI) | 3 | oci-python-sdk |

### BI tools (4)

| Source | Probes | Method |
|--------|--------|--------|
| Power BI | 6 | .pbit/.pbix ZIP parsing |
| Excel | 6 | .xlsx OOXML (table detector + formula lineage) |
| Tableau | 5 | .twb/.twbx XML parsing |
| Qlik Sense/View | 3 | .qvf SQLite / .qvw binary |

### ETL / Orchestration (6)

| Source | Probes | Method |
|--------|--------|--------|
| Informatica PowerCenter | 5 | Repository XML export |
| Informatica Cloud (IICS) | 4 | REST API v2/v3 |
| Oracle Data Integrator | 3 | Smart Export XML |
| Airflow | 3 | REST API |
| Alteryx | 2 | .yxmd XML parsing |
| dbt | 3 | manifest.json parsing |

### Governance platforms (4)

| Source | Probes | Direction |
|--------|--------|-----------|
| Collibra | 3 | Bidirectional |
| Microsoft Purview | 3 | Bidirectional |
| Blindata | 3 | Bidirectional |
| Witboost | 2 | Bidirectional |

### Files / Other (4)

| Source | Probes | Method |
|--------|--------|--------|
| File Scanner | 5 | Local filesystem (auto-dispatch + CSV/JSON/Parquet schema) |
| SharePoint / OneDrive | 3 | Local sync / REST / Graph API |
| Python Transforms | 1 | AST-based pandas/PySpark detection |
| Avro | 1 | Header-only schema reading |

## Architecture

```
Customer network (VPN / Azure / on-prem)
┌─────────────────────────────────────────────┐
│  metapod                                     │
│  ├── 155 probes across 40 sources           │
│  ├── Push mode: metapod push --target URL    │
│  ├── Pull mode: metapod serve --port 9090    │
│  └── File mode: metapod run → ./output/      │
└──────────────────┬──────────────────────────┘
                   │ outbound HTTPS only
                   ▼
         Any metadata consumer
```

- **Zero inbound access** — only outbound HTTPS
- **No raw data** — only metadata (schema, statistics, descriptions)
- **Local-first** — results written to `./output/` before push
- **Config-driven** — YAML with `${ENV_VAR}` substitution

## Commands

```bash
metapod run "oracle.*"                          # Extract
metapod push --target URL --project ID          # Push to platform
metapod serve --port 9090                       # Pull-mode daemon
metapod list-probes                             # Show all probes
metapod diff                                    # Compare extractions
metapod schedule "oracle.*" --cron "0 6 * * *"  # Cron schedule
metapod init                                    # Generate config
```

## Adding a probe

```python
from core.base_probe import BaseProbe
from core.registry import register

@register
class MyProbe(BaseProbe):
    @staticmethod
    def probe_name() -> str:
        return "mydb.metadata.tables"

    @staticmethod
    def description() -> str:
        return "Extract table metadata"

    def execute(self, connection, schema: str, **kwargs) -> list[dict]:
        # Your extraction logic here
        return [{"table_name": "example"}]
```

## License

Apache 2.0
