Metadata-Version: 2.4
Name: ski-audit-ledger
Version: 3.0.3
Summary: Audit ledger management tool for the SKI Framework.
Author-email: KpiFinity <hello@kpifinity.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/kpifinity/ski-framework
Project-URL: Documentation, https://kpifinity.github.io/ski-framework/
Project-URL: Issues, https://github.com/kpifinity/ski-framework/issues
Keywords: ski-framework,audit,compliance,ledger
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: psycopg[binary]<4,>=3.1.18
Requires-Dist: sqlalchemy<3,>=2.0.27
Requires-Dist: click<9,>=8.1.7
Requires-Dist: pydantic<3,>=2.13.4
Requires-Dist: python-dotenv<2,>=1.2.2
Requires-Dist: jinja2<4,>=3.1.6
Requires-Dist: pyyaml<7,>=6.0.1
Requires-Dist: httpx<0.29,>=0.27
Requires-Dist: tabulate<1,>=0.9

# audit-ledger

> Install from PyPI: `pip install ski-audit-ledger` (publishing starts with the first release after June 2026).

> **⚠ STATUS: EARLY ALPHA (v0.1.0a0).** Alpha-quality tooling. See the
> repo root `README.md` for the project-wide status.

Manage and verify the immutable audit ledger for SKI Framework compliance monitoring.

## v2.1 highlights

- **Real `verify_integrity()`.** Recomputes every entry's hash from the
  documented canonical serialization. Catches in-place tampering, not
  just chain-linkage breakage.
- **Real `backup`.** Invokes `pg_dump` and verifies the dump with
  `pg_restore --list`. No more stub.
- **Five-verdict taxonomy.** `--verdict-filter` now accepts
  `NULL_UNMAPPED` and `NULL_STALE` (the pre-v2.1 `NULL` is gone).
- **No `confidence_level`.** Removed entirely from the data model.
- **Append-only enforcement** lives at the database layer
  (`reference-implementation/src/ledger/append_only.sql`). UPDATE,
  DELETE, and TRUNCATE on `ledger_entries` are refused by triggers.

## Overview

The audit ledger is a hash-chained, immutable record of all compliance
verdicts generated by the SKI Model runtime. This tool provides
utilities for:

- **Verification** — Verify ledger integrity and detect tampering
- **Backup** — Create secure backups with cryptographic verification
- **Export** — Extract ledger data for analysis and reporting
- **Reporting** — Generate compliance reports and summaries

## Features

### Ledger Integrity Verification

Verify the complete hash chain of ledger entries to ensure no records have been tampered with:

```bash
audit-ledger verify --ledger-db postgresql://user:pass@localhost/ledger
```

Output:
```
Ledger Integrity Verification
├── Total entries: 4,327
├── Chain integrity: ✓ VERIFIED
├── Hash verification: ✓ 4,327/4,327 valid
├── Timestamp order: ✓ VALID
└── Last verified: 2026-05-21T15:30:00Z
```

### Secure Backup

Create verified backups with integrity checking:

```bash
audit-ledger backup \
  --source postgresql://user:pass@localhost/ledger \
  --output ./ledger-backup-2026-05-21.sql \
  --verify
```

### Data Export

Export ledger entries for analysis and reporting:

```bash
# Export to JSON
audit-ledger export \
  --source postgresql://user:pass@localhost/ledger \
  --output ledger-data.json \
  --format json

# Export to CSV
audit-ledger export \
  --source postgresql://user:pass@localhost/ledger \
  --output ledger-data.csv \
  --format csv \
  --fields "id,timestamp,verdict,rule_id,telemetry_id"
```

### Compliance Reporting

Generate formatted compliance reports:

```bash
audit-ledger report \
  --source postgresql://user:pass@localhost/ledger \
  --start-date 2026-01-01 \
  --end-date 2026-05-31 \
  --output compliance-report.html
```

## Installation

### From Source

```bash
cd tools/audit-ledger
pip install -e .
```

### Requirements

- Python 3.8+
- PostgreSQL 12+ (for ledger database)
- psycopg2 (PostgreSQL adapter)
- sqlalchemy (ORM)

## Quick Start

### 1. Verify Ledger Integrity

After running compliance monitoring for a period, verify the ledger has not been tampered with:

```bash
audit-ledger verify \
  --ledger-db postgresql://compliance:securepass@localhost:5432/ski_ledger \
  --verbose
```

### 2. Create a Backup

Before major system changes, create a verified backup:

```bash
audit-ledger backup \
  --source postgresql://compliance:securepass@localhost:5432/ski_ledger \
  --output ./backups/ledger-$(date +%Y%m%d).sql.gz \
  --compress \
  --verify
```

### 3. Generate Monthly Report

At month-end, generate a compliance summary:

```bash
audit-ledger report \
  --source postgresql://compliance:securepass@localhost:5432/ski_ledger \
  --start-date 2026-05-01 \
  --end-date 2026-05-31 \
  --output monthly-report-2026-05.html \
  --include-verdicts \
  --include-violations
```

### 4. Export for Audit

Export ledger data for external audit:

```bash
audit-ledger export \
  --source postgresql://compliance:securepass@localhost:5432/ski_ledger \
  --output audit-export.json \
  --format json \
  --date-range 2026-01-01,2026-05-31
```

## Command Reference

### verify

Verify ledger integrity and hash chain validity.

```bash
audit-ledger verify [OPTIONS]

Options:
  --ledger-db TEXT          PostgreSQL connection string [required]
  --verbose                 Show detailed verification output
  --check-timestamps        Verify timestamp ordering
  --repair                  Attempt to repair minor issues (use with caution)
  --output TEXT             Write results to file
  --help                    Show help message
```

### backup

Create a backup of the ledger database.

```bash
audit-ledger backup [OPTIONS]

Options:
  --source TEXT             PostgreSQL connection string [required]
  --output TEXT             Output file path [required]
  --compress                Gzip compress the backup
  --verify                  Verify backup integrity after creation
  --encryption-key TEXT     Encrypt backup (optional)
  --help                    Show help message
```

### export

Export ledger entries for analysis.

```bash
audit-ledger export [OPTIONS]

Options:
  --source TEXT             PostgreSQL connection string [required]
  --output TEXT             Output file path [required]
  --format TEXT             json|csv|jsonl (default: json)
  --fields TEXT             Comma-separated field names to include
  --date-range TEXT         Start,end dates (YYYY-MM-DD,YYYY-MM-DD)
  --verdict-filter TEXT     Filter by verdict (CLEAR|FLAG|NULL|DISCRETIONARY)
  --rule-id TEXT            Filter by rule ID
  --limit INTEGER           Maximum entries to export
  --help                    Show help message
```

### report

Generate a compliance report from ledger data.

```bash
audit-ledger report [OPTIONS]

Options:
  --source TEXT             PostgreSQL connection string [required]
  --output TEXT             Output HTML file path
  --start-date TEXT         Report start date (YYYY-MM-DD)
  --end-date TEXT           Report end date (YYYY-MM-DD)
  --include-verdicts        Include verdict summary statistics
  --include-violations      Include violation details
  --include-timeline        Include timeline visualization
  --include-audit-trail     Include audit trail verification
  --title TEXT              Report title
  --organization TEXT       Organization name for report
  --help                    Show help message
```

## Ledger Structure

The audit ledger stores compliance verdicts in a hash-chained format:

```sql
CREATE TABLE ledger_entries (
    id BIGSERIAL PRIMARY KEY,
    sequence_number BIGINT UNIQUE NOT NULL,
    previous_hash CHAR(64) NOT NULL,
    entry_hash CHAR(64) NOT NULL UNIQUE,
    timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    verdict TEXT NOT NULL CHECK (verdict IN (
        'CLEAR','FLAG','NULL_UNMAPPED','NULL_STALE','DISCRETIONARY'
    )),
    telemetry_id TEXT NOT NULL,
    telemetry_hash CHAR(64) NOT NULL,
    rule_id TEXT,
    knowledge_graph_version TEXT,
    ski_model_version TEXT NOT NULL,
    reasoning TEXT,
    track TEXT CHECK (track IS NULL OR track IN ('symbolic','llm')),
    escalation_status TEXT,
    escalation_notes TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```

> **v2.1 schema notes.** `milm_version` is renamed to `ski_model_version`.
> `confidence_level` has been removed entirely — B3.1 prohibits confidence
> scores in the ledger. The verdict CHECK constraint enforces the
> five-verdict taxonomy.

Each entry's `entry_hash` is `SHA-256` of the canonical serialization
documented in
[`src/audit_ledger/canonical.py`](./src/audit_ledger/canonical.py). The
canonical payload includes `sequence_number`, `previous_hash`, `timestamp`,
`verdict`, `telemetry_id`, `telemetry_hash`, `rule_id`, `kg_version`,
`ski_model_version`, `reasoning`, and `track`. Modifying any one of
these fields invalidates that entry's `entry_hash`.

Append-only enforcement is in
[`reference-implementation/src/ledger/append_only.sql`](../../reference-implementation/src/ledger/append_only.sql)
— `BEFORE UPDATE`, `BEFORE DELETE`, and `BEFORE TRUNCATE` triggers
refuse any modification at the database layer.

## Integrity verification process

The `verify` command checks, for every row:

1. **Chain linkage** — each row's `previous_hash` equals the prior row's `entry_hash`.
2. **Entry-hash recomputation** — re-derive `entry_hash` from the
   canonical payload and compare with what's stored. Catches in-place
   tampering that chain-linkage-only checks miss.
3. **Sequence continuity** — `sequence_number` has no gaps.
4. **Timestamp ordering** — timestamps are monotonic.

Example verification output:

```
Ledger Integrity Verification Report
====================================

Database: postgresql://localhost/ski_ledger
Verification Date: 2026-05-21T15:30:00Z

Chain Analysis:
  Total Entries: 4,327
  Sequence Range: 1 - 4,327
  Time Range: 2026-01-01 to 2026-05-21

Integrity Checks:
  ✓ Chain continuity: All 4,327 entries form valid chain
  ✓ Hash verification: 4,327/4,327 entries (100%)
  ✓ Timestamp ordering: All timestamps chronologically valid
  ✓ Data consistency: All references valid

Verdict Distribution:
  CLEAR: 3,821 (88.3%)
  FLAG: 412 (9.5%)
  DISCRETIONARY: 94 (2.2%)
  NULL: 0 (0.0%)

Risk Assessment:
  ✓ No signs of tampering
  ✓ No missing entries
  ✓ No invalid references
  Status: VERIFIED

Recommendation: Ledger integrity confirmed. Safe for regulatory reporting.
```

## Backup & Recovery

### Creating a Backup

```bash
# Standard backup
audit-ledger backup \
  --source postgresql://compliance:pass@localhost/ski_ledger \
  --output ./ledger-backup-2026-05-21.sql

# Compressed with verification
audit-ledger backup \
  --source postgresql://compliance:pass@localhost/ski_ledger \
  --output ./ledger-backup-2026-05-21.sql.gz \
  --compress \
  --verify
```

### Verifying a Backup

```bash
# Restore to test database
psql test_ledger < ledger-backup-2026-05-21.sql

# Verify the restored ledger
audit-ledger verify \
  --ledger-db postgresql://user:pass@localhost/test_ledger
```

## Export Examples

### Export All Violations

```bash
audit-ledger export \
  --source postgresql://user:pass@localhost/ski_ledger \
  --output violations.json \
  --format json \
  --verdict-filter FLAG
```

Output:
```json
{
  "export_date": "2026-05-21T15:30:00Z",
  "entry_count": 412,
  "entries": [
    {
      "id": 1543,
      "timestamp": "2026-05-15T10:30:00Z",
      "verdict": "FLAG",
      "rule_id": "e001",
      "telemetry_id": "TEL_2026_05_001",
      "reasoning": "SO2 emissions 125 ppm exceeds limit of 100 ppm"
    },
    ...
  ]
}
```

### Export Monthly Audit Trail

```bash
audit-ledger export \
  --source postgresql://user:pass@localhost/ski_ledger \
  --output may-2026-audit.csv \
  --format csv \
  --date-range 2026-05-01,2026-05-31 \
  --fields "timestamp,verdict,rule_id,reasoning,escalation_status"
```

## Reporting

### Generate Monthly Compliance Report

```bash
audit-ledger report \
  --source postgresql://user:pass@localhost/ski_ledger \
  --start-date 2026-05-01 \
  --end-date 2026-05-31 \
  --output may-2026-compliance.html \
  --include-verdicts \
  --include-violations \
  --include-timeline \
  --organization "Acme Energy Corp"
```

Report includes:
- Executive summary of compliance status
- Verdict distribution charts
- Violations identified and status
- Remediation actions taken
- Timeline of escalations
- Audit trail verification status
- Regulatory filing readiness

## Security Considerations

### Database Security

The PostgreSQL ledger database should be:
- Behind a firewall (not internet-accessible)
- Using strong authentication (not default credentials)
- Requiring SSL/TLS for all connections
- Backed up regularly
- Monitored for unauthorized access

### Backup Security

Backups should be:
- Encrypted if stored off-site
- Stored securely (not in version control)
- Verified before relying on them
- Tested for recovery capability

### Access Control

Ledger access should be restricted to:
- Compliance officers
- System administrators
- Authorized auditors

## Troubleshooting

### Ledger Verification Fails

If verification reports issues:

```bash
# Get detailed diagnostic output
audit-ledger verify \
  --ledger-db postgresql://user:pass@localhost/ski_ledger \
  --verbose

# Attempt automatic repair (use with caution)
audit-ledger verify \
  --ledger-db postgresql://user:pass@localhost/ski_ledger \
  --repair
```

### Export Takes Too Long

For large ledgers, use filters:

```bash
# Export only recent violations
audit-ledger export \
  --source postgresql://user:pass@localhost/ski_ledger \
  --output recent-flags.json \
  --verdict-filter FLAG \
  --date-range 2026-05-01,2026-05-31
```

### Backup Verification Fails

Restore to a test database and verify:

```bash
# Create test database
createdb test_ledger

# Restore backup
psql test_ledger < backup.sql

# Verify restoration
audit-ledger verify --ledger-db postgresql://user:pass@localhost/test_ledger
```

## API Usage

### Python API

```python
from audit_ledger import Ledger

# Connect to ledger
ledger = Ledger("postgresql://user:pass@localhost/ski_ledger")

# Verify integrity
result = ledger.verify_integrity()
print(f"Ledger valid: {result.is_valid}")
print(f"Total entries: {result.total_entries}")

# Export entries
entries = ledger.export_entries(
    start_date="2026-05-01",
    end_date="2026-05-31",
    verdict_filter="FLAG"
)

# Generate report
report = ledger.generate_report(
    start_date="2026-05-01",
    end_date="2026-05-31",
    include_verdicts=True,
    include_violations=True
)
ledger.save_report(report, "compliance-report.html")
```

## Production Checklist

- [ ] Configure PostgreSQL with strong authentication
- [ ] Enable SSL/TLS for database connections
- [ ] Set up automated backup schedule
- [ ] Verify backup recovery procedures
- [ ] Document ledger access controls
- [ ] Train staff on ledger commands
- [ ] Set up monitoring for ledger database
- [ ] Plan audit trail retention policy
- [ ] Test export/reporting workflows
- [ ] Schedule regular integrity verification
- [ ] Document escalation procedures for verification failures

## Support

For issues or questions:
- Check individual command help: `audit-ledger COMMAND --help`
- Review ledger structure documentation
- Open an issue on GitHub
- Contact your compliance team

---

**Note:** The audit ledger is critical for regulatory compliance. All operations should be performed by authorized personnel only. Always verify backups and test recovery procedures before relying on them.
