Metadata-Version: 2.4
Name: metapod-probes
Version: 0.1.1
Summary: Lightweight metadata extraction probes — 135 probes across 35 data sources
Project-URL: Homepage, https://github.com/mario-ambrosino/metapod
Project-URL: Repository, https://github.com/mario-ambrosino/metapod
Author-email: Mario Ambrosino <ambromar@outlook.it>
License: Apache-2.0
Keywords: data-catalog,data-lineage,dbt,metadata,oracle,probes,snowflake
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: typer[all]>=0.12
Provides-Extra: all
Requires-Dist: azure-identity>=1.15; extra == 'all'
Requires-Dist: azure-storage-blob>=12.19; extra == 'all'
Requires-Dist: azure-storage-file-datalake>=12.14; extra == 'all'
Requires-Dist: azure-storage-file-share>=12.15; extra == 'all'
Requires-Dist: boto3>=1.34; extra == 'all'
Requires-Dist: databricks-sdk>=0.20; extra == 'all'
Requires-Dist: duckdb>=0.10; extra == 'all'
Requires-Dist: elasticsearch>=8.0; extra == 'all'
Requires-Dist: google-cloud-bigquery>=3.0; extra == 'all'
Requires-Dist: oracledb>=2.0; extra == 'all'
Requires-Dist: psycopg2-binary>=2.9; extra == 'all'
Requires-Dist: pyarrow>=14.0; extra == 'all'
Requires-Dist: pymongo>=4.0; extra == 'all'
Requires-Dist: pymysql>=1.1; extra == 'all'
Requires-Dist: snowflake-connector-python>=3.0; extra == 'all'
Provides-Extra: avro
Requires-Dist: avro>=1.11; extra == 'avro'
Provides-Extra: aws
Requires-Dist: boto3>=1.34; extra == 'aws'
Provides-Extra: azure
Requires-Dist: azure-identity>=1.15; extra == 'azure'
Requires-Dist: azure-storage-blob>=12.19; extra == 'azure'
Requires-Dist: azure-storage-file-datalake>=12.14; extra == 'azure'
Requires-Dist: azure-storage-file-share>=12.15; extra == 'azure'
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery>=3.0; extra == 'bigquery'
Provides-Extra: databricks
Requires-Dist: databricks-sdk>=0.20; extra == 'databricks'
Provides-Extra: db2
Requires-Dist: ibm-db>=3.0; extra == 'db2'
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: duckdb
Requires-Dist: duckdb>=0.10; extra == 'duckdb'
Provides-Extra: elasticsearch
Requires-Dist: elasticsearch>=8.0; extra == 'elasticsearch'
Provides-Extra: hana
Requires-Dist: hdbcli>=2.0; extra == 'hana'
Provides-Extra: mongodb
Requires-Dist: pymongo>=4.0; extra == 'mongodb'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.1; extra == 'mysql'
Provides-Extra: oracle
Requires-Dist: oracledb>=2.0; extra == 'oracle'
Provides-Extra: parquet
Requires-Dist: pyarrow>=14.0; extra == 'parquet'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python>=3.0; extra == 'snowflake'
Provides-Extra: teradata
Requires-Dist: teradatasql>=20.0; extra == 'teradata'
Provides-Extra: trino
Requires-Dist: trino>=0.320; extra == 'trino'
Description-Content-Type: text/markdown

# metapod

Lightweight metadata and quality probes for [the target platform](https://github.com/mario-ambrosino/metapod). Runs inside your network (VPN, Azure, on-prem) and extracts only metadata — no raw data leaves the perimeter.

## Quick start (Python)

```bash
# Install
pip install -e ".[oracle]"

# Create config
python cli.py init
# Edit config.yaml with your Oracle credentials

# List available probes
python cli.py list-probes

# Run all Oracle metadata probes
python cli.py run "oracle.metadata.*" --schema RISK_MGMT

# Run all probes (metadata + quality)
python cli.py run "oracle.*" --schema RISK_MGMT

# Run a single probe
python cli.py run oracle.metadata.tables --schema RISK_MGMT

# Run and push results to target platform
python cli.py run "oracle.*" --schema RISK_MGMT --push
```

## Quick start (Docker)

```bash
docker build -t metapod .
docker run -v $(pwd)/config.yaml:/app/config.yaml -v $(pwd)/output:/app/output \
  metapod "oracle.metadata.*"
```

## Available probes

### Oracle metadata (9 probes)

| Probe | Source | Extracts |
|-------|--------|----------|
| `oracle.metadata.tables` | ALL_TABLES | Table names, row counts, tablespace |
| `oracle.metadata.columns` | ALL_TAB_COLUMNS | Column names, types, nullable, position |
| `oracle.metadata.views` | ALL_VIEWS | View SQL definitions |
| `oracle.metadata.source_code` | ALL_SOURCE | PL/SQL: procedures, functions, packages, triggers |
| `oracle.metadata.synonyms` | ALL_SYNONYMS | Private + public synonym mappings |
| `oracle.metadata.db_links` | ALL_DB_LINKS | Database link definitions |
| `oracle.metadata.constraints` | ALL_CONSTRAINTS | PK, FK, unique constraints with columns |
| `oracle.metadata.indexes` | ALL_INDEXES | Index types and column compositions |
| `oracle.metadata.comments` | ALL_TAB/COL_COMMENTS | Business descriptions |

### Oracle quality (5 probes)

| Probe | Method | Measures |
|-------|--------|----------|
| `oracle.quality.null_ratio` | Sampled COUNT | NULL % per column |
| `oracle.quality.distinct_count` | Oracle stats | Cardinality per column |
| `oracle.quality.min_max` | Oracle stats | Value ranges for numeric/date |
| `oracle.quality.freshness` | LAST_ANALYZED | Data staleness in days |
| `oracle.quality.pattern_check` | REGEXP_LIKE | PII patterns (email, phone, SSN, IBAN) |

## Architecture

```
Customer network (VPN / Azure / on-prem)
┌─────────────────────────────────────────┐
│  metapod                          │
│  ├── oracle.metadata.* → local JSON     │
│  ├── oracle.quality.*  → local JSON     │
│  └── --push → HTTPS to target platform API     │
└────────────────┬────────────────────────┘
                 │ outbound HTTPS only
                 ▼
┌─────────────────────────────────────────┐
│  the target platform Platform                       │
│  └── Metastore (MinIO)                  │
│      └── probes/oracle/metadata/*.json  │
└─────────────────────────────────────────┘
```

- **Zero inbound access** — only outbound HTTPS to target platform API
- **No raw data** — only metadata (schema structure, statistics, descriptions)
- **Local-first** — results written to `./output/` before optional push
- **Config-driven** — YAML file, no code changes per deployment

## Configuration

```yaml
source:
  type: oracle
  host: prod-db.example.com
  port: 1521
  service_name: ORCLPDB1
  username: metadata_reader
  password: "${ORACLE_PASSWORD}"
  schema_name: RISK_MGMT

target:
  url: https://your-platform.example.com
  api_key: "${METAPOD_API_KEY}"
  project: default

output_dir: ./output
```

## Output format

Every probe produces a standard JSON envelope:

```json
{
  "probe": "oracle.metadata.tables",
  "source": {"type": "oracle", "id": "prod-db:1521/ORCLPDB1/RISK_MGMT"},
  "timestamp": "2026-03-27T23:00:00Z",
  "records": [...],
  "record_count": 42,
  "duration_ms": 350
}
```

## Adding probes

Create a new file in `probes/oracle/metadata/` or `probes/oracle/quality/`:

```python
from core.base_probe import BaseProbe
from core.registry import register

@register
class MyProbe(BaseProbe):
    @staticmethod
    def probe_name() -> str:
        return "oracle.metadata.my_probe"

    @staticmethod
    def description() -> str:
        return "What this probe extracts"

    def execute(self, connection, schema: str, **kwargs) -> list[dict]:
        with connection.cursor() as cur:
            cur.execute("SELECT ... FROM all_... WHERE owner = :schema", schema=schema.upper())
            return [{"col": row[0]} for row in cur.fetchall()]
```

## License

Apache 2.0
