Metadata-Version: 2.4
Name: meridian-schema-forge
Version: 0.1.2
Summary: Build a processing pipeline from any schema, in any format: normalize a schema (XML/XSD, JSON/JSON-Schema, YAML, SQL DDL, delimited), classify field roles, then profile (exact tiktoken counts) or de-identify — from files or SQL.
Author: Meridian Intelligence
License: Proprietary — Meridian Intelligence, engagement-scoped (see LICENSE)
Project-URL: Documentation, https://github.com/Meridian-Int/meridian-schema-forge#readme
Keywords: schema,de-identification,pseudonymization,profiling,tiktoken,tokens,data-pipeline,xml,json-schema,sql
Classifier: Programming Language :: Python :: 3
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE.pdf
Requires-Dist: PyYAML>=5.4
Requires-Dist: jsonschema>=4.0
Requires-Dist: tiktoken>=0.7
Requires-Dist: SQLAlchemy>=1.4
Provides-Extra: sql
Requires-Dist: pyodbc>=4.0; extra == "sql"
Provides-Extra: ner
Requires-Dist: spacy>=3.4; extra == "ner"
Provides-Extra: stream
Requires-Dist: ijson>=3.0; extra == "stream"
Provides-Extra: pandas
Requires-Dist: pandas>=1.3; extra == "pandas"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

# schemaforge

schemaforge turns a schema plus a data source into an inspectable processing
plan, then uses that plan to profile or de-identify the data.

Use it when you need to:

- Normalize a schema from SQL DDL, XSD/XML, JSON/JSON-Schema, YAML, JSONL, or a
  delimited header.
- Classify fields into auditable roles such as identifier, quasi-identifier,
  free text, date, numeric, categorical, boolean, and structural.
- Profile data with exact token counts and useful per-field metrics.
- De-identify files or SQL tables while preserving keys and joins.

The central artifact is `plan.yaml`. Generate it, review it, edit any field
roles or processor options, then re-run from that same plan.

## Install

Python 3.9 or newer is required.

Install from PyPI:

```bash
python3 -m pip install meridian-schema-forge
```

Or install from a wheel supplied to you:

```bash
python3 -m pip install meridian_schema_forge-0.1.2-py3-none-any.whl
```

Or install from a source checkout:

```bash
python3 -m pip install .
```

For local development:

```bash
python3 -m pip install -e ".[dev]"
```

Optional extras:

| Extra | Installs | Use when |
| --- | --- | --- |
| `sql` | `pyodbc` | Connecting to Microsoft Fabric, Azure SQL, or SQL Server |
| `stream` | `ijson` | Working with very large JSON arrays |
| `ner` | `spacy` | Adding statistical name detection in free text |
| `pandas` | `pandas` | Using DataFrame convenience workflows |
| `dev` | `pytest` | Running the test suite |

Verify the CLI is available:

```bash
schemaforge --version
```

If you are running directly from a checkout without installing, use:

```bash
PYTHONPATH=src python3 -m schemaforge --version
```

## Quick Start

The source checkout includes a quickstart with two CSV files and a matching SQL
schema.

```bash
cd examples/quickstart

# 1. Inspect the normalized schema and classifier output.
schemaforge inspect-schema --schema schema.sql --classify

# 2. Build the editable plan.
schemaforge plan --schema schema.sql --source ./data --out plan.yaml

# 3. Profile the data.
schemaforge profile --plan plan.yaml --out-json profile.json

# 4. Generate a secret and de-identify the data.
schemaforge init-secret --out secret.key
schemaforge deidentify --plan plan.yaml --out ./clean --secret-file secret.key

# 5. Validate the profile output contract.
schemaforge validate --json profile.json --kind profile
```

Expected outputs:

- `plan.yaml`: schema, source binding, roles, and processor options.
- `profile.json`: token counts and per-field profile metrics.
- `clean/*.jsonl`: de-identified records, one JSONL file per entity.
- `clean-PRIVATE/`: private mapping and audit files. Do not deliver this
  directory with de-identified data.
- `secret.key`: the HMAC secret used for deterministic pseudonymization. Keep it
  private.

## Standard Workflow

1. Inspect the schema.

   ```bash
   schemaforge inspect-schema --schema schema.sql --classify
   ```

2. Build a plan from the schema and source data.

   ```bash
   schemaforge plan --schema schema.sql --source ./data --out plan.yaml
   ```

3. Review `plan.yaml`.

   Check that each field has the right `role`. Change any role by hand if the
   classifier guessed incorrectly.

4. Run a processor from the plan.

   ```bash
   schemaforge profile --plan plan.yaml --out-json profile.json
   schemaforge deidentify --plan plan.yaml --out ./clean --secret-file secret.key
   ```

5. Validate outputs before sharing.

   ```bash
   schemaforge validate --json profile.json --kind profile
   ```

## Inputs

### Schema Inputs

schemaforge accepts schema files, raw schema text, and simple data files that can
be used to infer a schema.

| Input | Examples | Notes |
| --- | --- | --- |
| SQL DDL | `.sql`, `.ddl` | Parses `CREATE TABLE`, primary keys, foreign keys, and column types |
| XML / XSD | `.xml`, `.xsd` | XSD gives declared structure; plain XML can be inferred |
| JSON / JSON-Schema | `.json` | JSON-Schema is read directly; plain JSON can be inferred |
| JSONL / NDJSON | `.jsonl`, `.ndjson` | Infers fields from newline-delimited records |
| YAML | `.yaml`, `.yml` | Reads schema-shaped YAML or infers from data |
| Delimited | `.csv`, `.tsv` | Uses the header as fields |

Auto-detection uses extension first and then content. Force a format when needed:

```bash
schemaforge inspect-schema --schema customers.xsd --format xsd
schemaforge inspect-schema --schema schema.ddl --format sql
schemaforge inspect-schema --schema customers.csv --format delimited
```

### Data Sources

Use files or any SQLAlchemy database URL.

For files, point `--source` at a directory, glob, or single file:

```bash
schemaforge plan --schema schema.sql --source ./data --out plan.yaml
schemaforge profile --schema customer.json --source ./customer.jsonl --out-json profile.json
schemaforge profile --schema schema.sql --source "./exports/*.csv" --out-json profile.json
```

Directory sources match schema entities to files by filename stem:

```text
data/
  customer.csv
  order.csv
```

CSV, TSV, TXT, JSONL, JSON, YAML, and XML are supported. CSV, JSONL, and XML are
streamed. JSON and YAML are loaded as files, so prefer JSONL or SQL for very
large data.

For SQL:

```bash
schemaforge plan --schema schema.sql --source "sqlite:///app.db" --out plan.yaml
schemaforge profile --plan plan.yaml --out-json profile.json
```

Any SQLAlchemy URL can be used. For SQL Server, Azure SQL, or Microsoft Fabric,
install the `sql` extra and provide an ODBC URL, for example:

```bash
schemaforge plan \
  --schema schema.sql \
  --source "mssql+pyodbc://user:password@host/database?driver=ODBC+Driver+18+for+SQL+Server" \
  --out plan.yaml
```

## The Plan File

`plan.yaml` is the reproducible contract for a run. It includes:

- The normalized schema.
- The source location.
- One role for every field.
- Processor options.
- The list of roles counted as profile "content".
- A config digest and schemaforge version.

Example:

```yaml
schemaforge_version: 0.1.2
source:
  kind: file
  location: ./data
  group_key: customer_id
content_roles: [categorical, free_text, numeric, quasi_identifier]
processors:
  profile: {}
  deidentify:
    date_strategy: shift
    date_max_days: 365
schema:
  entities:
    - name: customer
      primary_key: [customer_id]
      fields:
        - name: customer_id
          type: integer
          is_primary_key: true
          role: structural
        - name: email
          type: string
          role: identifier
        - name: full_name
          type: string
          role: quasi_identifier
        - name: notes
          type: string
          role: free_text
```

After editing the plan, run processors with `--plan plan.yaml`. You do not need
to regenerate the plan unless the schema, source, or config changed.

## Field Roles

Every field gets exactly one role.

| Role | Meaning | Common examples |
| --- | --- | --- |
| `identifier` | Directly identifies a person, account, or row | email, phone, SSN, UUID, account number |
| `quasi_identifier` | May identify someone when combined with other fields | name, birth date, ZIP, demographic fields |
| `free_text` | Narrative or unstructured text | notes, comments, descriptions |
| `date` | Date or timestamp that should be shifted or generalized | order_date, created_at |
| `numeric` | Quantity or measurement | amount, score, quantity |
| `categorical` | Label, enum, or code with limited values | status, country, type |
| `boolean` | True/false value | active, is_paid |
| `structural` | Keys or structure needed for joins and shape | primary key, foreign key, row index |

Classification is deterministic. The classifier considers, in order:

1. Config overrides.
2. Primary keys, foreign keys, and structural schema facts.
3. Declared field type or format.
4. Built-in name patterns.
5. Sampled values.
6. Type fallback.

If a role is wrong, edit `plan.yaml` directly or use a config file.

```yaml
# config.yaml
roles:
  customer.membership_code: categorical
  order.tracking_ref: identifier

thresholds:
  categorical_max_distinct: 50

content_roles: [free_text, numeric, categorical, quasi_identifier]

processors:
  deidentify:
    date_strategy: shift
    date_max_days: 365
```

Use the config while building the plan:

```bash
schemaforge plan --schema schema.sql --source ./data --config config.yaml --out plan.yaml
```

## Processors

### Profile

`profile` measures corpus size and field shape.

```bash
schemaforge profile --plan plan.yaml --out-json profile.json
```

It reports:

- Record counts by entity.
- Total token counts.
- Content-only token counts based on `content_roles`.
- Structural token counts.
- Per-field coverage, distinct counts, date ranges, numeric min/max/mean, and
  top values.
- Token counts using `o200k_base`, with `cl100k_base` as a cross-check.

### De-Identify

`deidentify` transforms records based on field role.

```bash
schemaforge init-secret --out secret.key
schemaforge deidentify --plan plan.yaml --out ./clean --secret-file secret.key
```

Role behavior:

| Role | De-identification behavior |
| --- | --- |
| `identifier` | Replaced with deterministic keyed surrogates |
| Primary and foreign keys | Replaced with deterministic keyed surrogates that preserve joins |
| `quasi_identifier` | Generalized, such as names to synthetic tokens, ZIPs to prefixes, ages to bands, dates to years |
| `free_text` | Scrubbed for known identifiers and identifier-shaped values |
| `date` | Shifted by group, preserving intervals, or generalized to year |
| `numeric`, `categorical`, `boolean` | Passed through |
| `structural` | Passed through unless it is a primary or foreign key |

The write path is fail-safe. schemaforge writes to a staging directory, runs QA,
and only promotes the output if the gate passes. If a leak is detected, nothing
is written to the requested output directory.

Private artifacts are written to a sibling `*-PRIVATE` directory. Keep that
directory and the secret out of any delivered dataset.

## Command Reference

| Command | Purpose | Example |
| --- | --- | --- |
| `inspect-schema` | Parse and display a normalized schema | `schemaforge inspect-schema --schema schema.sql --classify` |
| `plan` | Build `plan.yaml` from schema, source, and optional config | `schemaforge plan --schema schema.sql --source ./data --out plan.yaml` |
| `profile` | Produce token counts and per-field metrics | `schemaforge profile --plan plan.yaml --out-json profile.json` |
| `deidentify` | Write de-identified JSONL output | `schemaforge deidentify --plan plan.yaml --out ./clean --secret-file secret.key` |
| `validate` | Validate an output contract | `schemaforge validate --json profile.json --kind profile` |
| `run` | Run a registered custom processor | `schemaforge run --plan plan.yaml --processor rowcount` |
| `init-secret` | Generate a private HMAC secret | `schemaforge init-secret --out secret.key` |

Most processor commands can either use an existing plan:

```bash
schemaforge profile --plan plan.yaml --out-json profile.json
```

Or build a plan on the fly from `--schema` and `--source`:

```bash
schemaforge profile --schema schema.sql --source ./data --out-json profile.json
```

Prefer a saved plan for reviewed or repeatable work.

## Python API

```python
import schemaforge as sf

schema = sf.ingest("schema.sql")
plan = sf.build_plan(schema, "./data")

profile = sf.profile(plan)
assert not sf.errors(sf.run_qa(profile))
sf.write_json(profile.output, "profile.json")

secret = sf.generate_secret()
sf.deidentify(plan, out="clean", secret=secret)
```

Useful API entry points:

```python
sf.ingest(source, fmt="auto")
sf.build_plan(schema, source, config=None)
sf.profile(plan)
sf.deidentify(plan, out="clean", secret_file="secret.key")
sf.run(plan, "processor_name")
sf.run_qa(result)
sf.errors(issues)
sf.validate(obj, "profile")
sf.write_json(obj, path)
sf.write_yaml(obj, path)
sf.generate_secret()
```

## Custom Processors

Processors can be registered without changing core files.

```python
import schemaforge as sf

@sf.register
class RowCount(sf.Processor):
    name = "rowcount"

    def run(self, reader, plan):
        rows = sum(1 for entity in reader.entities() for _ in reader.iter_records(entity))
        return sf.ProcessorResult(self.name, {"rows": rows})

sf.run(plan, "rowcount")
```

Run a registered processor from the CLI:

```bash
schemaforge run --plan plan.yaml --processor rowcount
```

## Quality Gates

schemaforge uses hard QA gates. An error exits with code `2`; usage or runtime
errors exit with code `1`; success exits with code `0`.

Gates include:

- Schema has at least one entity and uniquely named fields.
- Every field has exactly one valid role.
- Plans round-trip and validate against the bundled plan contract.
- Profile output has internally consistent token, count, coverage, percentile,
  and distribution metrics.
- De-identification has matching input/output counts, injective mappings, and no
  known identifier or identifier-shaped value surviving verbatim in output.

For de-identification, QA runs before output promotion, so a failed gate writes
nothing to the requested output directory.

## Troubleshooting

`schemaforge: command not found`

Install the package first with `python3 -m pip install .` or run from a checkout
with `PYTHONPATH=src python3 -m schemaforge ...`.

`No module named schemaforge`

The package is not installed in the interpreter you are using. Run
`python3 -m pip install .` from the repository root or activate the environment
where schemaforge was installed.

An entity has zero records

For file sources, confirm each filename stem matches an entity name in the schema,
such as `customer.csv` for entity `customer`. For a single JSON, YAML, or XML
container file, confirm entity arrays are keyed by entity name.

A field has the wrong role

Edit `plan.yaml` and re-run the processor, or add a role override in `config.yaml`
and rebuild the plan.

De-identification fails with a leak gate

Review the reported field/value shape, mark the source field as `identifier` or
`quasi_identifier`, and re-run. No requested output directory is written when the
gate fails.

## Requirements

- Python 3.9+
- PyYAML
- jsonschema
- tiktoken
- SQLAlchemy
- Optional database drivers for non-SQLite SQL sources

## License

Proprietary - Meridian Intelligence, engagement-scoped. This software is provided
solely for the specific engagement for which it is supplied and must be deleted
and cleared after that exercise. See [LICENSE](LICENSE) and
[LICENSE.pdf](LICENSE.pdf).
