Metadata-Version: 2.4
Name: termscockpit
Version: 1.0.3
Summary: Terms Cockpit is a Python package that enables the user to operate with documents from the ToS;DR project, drawing directly from their GitHub repository.
Author-email: José María Cruz Lorite <josemariacruzlorite@gmail.com>
Project-URL: Homepage, https://github.com/cruzlorite/termscockpit
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tqdm
Requires-Dist: GitPython
Requires-Dist: platformdirs
Provides-Extra: server
Requires-Dist: flask; extra == "server"
Requires-Dist: flask-smorest; extra == "server"
Requires-Dist: readability-lxml; extra == "server"
Requires-Dist: playwright; extra == "server"
Provides-Extra: lexglue
Requires-Dist: transformers; extra == "lexglue"
Requires-Dist: torch; extra == "lexglue"
Dynamic: license-file

# Terms Cockpit

![PyPI Version](https://img.shields.io/pypi/v/termscockpit)
![Python Versions](https://img.shields.io/pypi/pyversions/termscockpit)
![License](https://img.shields.io/pypi/l/termscockpit)
[![Publish to PyPI](https://github.com/cruzlorite/termscockpit/actions/workflows/publish-pypi.yaml/badge.svg)](https://github.com/cruzlorite/termscockpit/actions/workflows/publish-pypi.yaml)

**Terms Cockpit** is a Python package for accessing, indexing, and analysing Terms of Service and Privacy Policy documents from [OpenTermsArchive](https://github.com/OpenTermsArchive) and [ToS;DR](https://github.com/tosdr) repositories.

It ships both a **reusable Python library** and an optional **web application** with a full REST API, interactive document viewer, version history explorer, and UNFAIR_TOS clause classifier.

---

## Table of Contents

- [Features](#features)
- [Installation](#installation)
- [Python Library](#python-library)
  - [TermsCockpit](#termscockpit)
  - [GitChangeIndex](#gitchangeindex)
  - [document\_utils](#document_utils)
  - [lexglue](#lexglue)
- [Web Server](#web-server)
  - [Running the server](#running-the-server)
  - [CLI options](#cli-options)
  - [Known repositories](#known-repositories)
  - [REST API](#rest-api)
- [Architecture](#architecture)
- [License](#license)
- [Acknowledgements](#acknowledgements)

---

## Features

- **Multi-repository support** — load any number of OpenTermsArchive or ToS;DR snapshot repositories simultaneously.
- **Efficient change indexing** — a single `git log --numstat` pass populates a SQLite database; subsequent starts are incremental.
- **Document retrieval** — fetch latest content or any historical version by commit hash.
- **Readability extraction** — strip navigation chrome from HTML with Mozilla Readability.
- **Version diffing** — unified diffs of raw HTML or plain-text (tag-stripped) content between any two adjacent versions.
- **Text-change detection** — identify which versions have actual visible-text changes, ignoring formatting-only commits.
- **UNFAIR_TOS classification** — stream per-paragraph clause analysis via Server-Sent Events using fine-tuned LegalBERT, zero-shot NLI, or cosine-similarity backends.
- **Interactive web UI** — service browser, document viewer, version history chart, diff viewer, evolution sparklines, and LexGlue analysis page.
- **OpenAPI / Swagger UI** — auto-generated API documentation at `/api/docs`.

---

## Installation

```bash
# Core library only
pip install termscockpit

# Core + web server
pip install 'termscockpit[server]'

# Core + web server + LexGlue classifier
pip install 'termscockpit[server,lexglue]'

# From source
pip install git+https://github.com/cruzlorite/termscockpit.git
```

---

## Python Library

All business logic is available as a standalone Python package, independent of the web server.

### TermsCockpit

The main entry point. Clones (or pulls) a remote snapshot repository, enumerates services and documents, and provides content and history query methods.

```python
from termscockpit import TermsCockpit

# Uses the OpenTermsArchive Community repo by default
tos = TermsCockpit(track_changes=True)

# Enumerate
print(tos.services)                        # ['Google', 'Facebook', ...]
print(tos.documents['Google'])             # ['Google/Privacy Policy.html', ...]
print(tos.list_all_documents())            # flat list of all document paths

# Latest content
html = tos.get_document_content('Google/Privacy Policy.html')

# Content at a specific commit
html = tos.get_document_content('Google/Privacy Policy.html', commit_hash='abc123')

# Readability-extracted content
result = tos.get_document_readability('Google/Privacy Policy.html')
# {'title': '...', 'short_title': '...', 'content': '<html>...'}

# Version history
changes = tos.list_document_changes('Google/Privacy Policy.html')
# [(commit_hash, author, timestamp, insertions, deletions, blob_sha), ...]

# Summaries for a list of documents (batch-efficient)
summaries = tos.list_document_summaries(['Google/Privacy Policy.html'])
# {'Google/Privacy Policy.html': (version_count, last_timestamp)}

# Unified diff — raw HTML
diff = tos.diff_document_between_commits('Google/Privacy Policy.html', idx=5)

# Unified diff — visible text only (tags stripped)
diff = tos.diff_document_text('Google/Privacy Policy.html', idx=5)

# Indices of versions with actual text-content changes
indices = tos.list_text_change_versions('Google/Privacy Policy.html')
# [1, 3, 7, ...]

tos.close()
```

**Constructor parameters**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `track_changes` | `bool` | `True` | Build a SQLite change index on startup. Disable for faster startup when history is not needed. |
| `repo_url` | `str` | OTA Community | URL of the Git repository to clone. |

**Class constants**

| Constant | Value |
|---|---|
| `TermsCockpit.OTA_REPO_URL` | `https://github.com/OpenTermsArchive/contrib-snapshots.git` |
| `TermsCockpit.TOSDR_REPO_URL` | `https://github.com/tosdr/tosdr-snapshots.git` |

---

### GitChangeIndex

Low-level, reusable SQLite index over any Git repository. Can be used independently of `TermsCockpit`.

```python
from termscockpit import GitChangeIndex
from pathlib import Path

index = GitChangeIndex(
    repo_path=Path('/path/to/repo'),
    index_path=Path('/path/to/index.db'),
)

# Per-file summary (batch query)
summary = index.list_files_summary(['service/doc.html'])
# {'service/doc.html': (version_count, last_timestamp)}

# Full history for one file
changes = index.list_file_changes('service/doc.html')
# [(commit_hash, author, timestamp, insertions, deletions, blob_sha), ...]

# File content at a commit
content = index.get_file_content('abc123def', 'service/doc.html')

# Unified diff between adjacent versions
diff_lines = index.diff_adjacent('service/doc.html', idx=3)

index.close()
```

The index is built with a single `git log --numstat` subprocess (orders of magnitude faster than per-commit GitPython API calls) and updated incrementally on subsequent runs.

---

### document\_utils

Pure-Python HTML utilities, usable without the server.

```python
from termscockpit.document_utils import html_to_text, text_hash, apply_readability

# Strip tags and normalise whitespace
text = html_to_text(html)

# MD5 fingerprint of visible text (useful for change detection)
h = text_hash(html)

# Mozilla Readability extraction
result = apply_readability(html)
# {'title': '...', 'short_title': '...', 'content': '<html>...'}
```

---

### lexglue

UNFAIR_TOS classification engine, usable without the server.

```python
from termscockpit.lexglue import (
    UNFAIR_CATEGORIES, FAIR_LABEL, DEFAULT_MODEL,
    detect_backend, load_pipeline, score_batch,
    extract_blocks_annotated,
)

# Extract annotated segments from an HTML document
segments, annotated_html = extract_blocks_annotated(html, readability=True)

# Load inference pipeline (cached after first call)
pipe = load_pipeline('Agreemind/lexglue-legalbert-unfair-tos')

# Score a batch of text segments
scores = score_batch(pipe, 'Agreemind/lexglue-legalbert-unfair-tos', segments)
# [{'limitation of liability': 0.82, 'fair and unproblematic clause': 0.18, ...}, ...]
```

**Supported backends** (auto-detected from model config):

| Backend | Description | Example model |
|---|---|---|
| `clf` | Fine-tuned multi-label classifier | `Agreemind/lexglue-legalbert-unfair-tos` |
| `nli` | Zero-shot NLI via entailment | `cross-encoder/nli-deberta-v3-small` |
| `sim` | Cosine similarity on CLS embeddings | `nlpaueb/legal-bert-base-uncased` |

**UNFAIR_TOS categories**

1. Limitation of liability
2. Unilateral changes to the terms
3. Content removal by the provider
4. Jurisdiction clause
5. Choice of law
6. Mandatory arbitration
7. Unilateral termination by the provider
8. Contract by using the service

---

## Web Server

### Running the server

```bash
# Load all known repositories (clones on first run, ~several GB)
python -m termscockpit.server

# Load a subset of repositories
python -m termscockpit.server --repos contrib genai-contrib tosdr

# Faster startup — disable change tracking (history features unavailable)
python -m termscockpit.server --repos contrib --no-changes

# Custom host and port
python -m termscockpit.server --host 0.0.0.0 --port 8080

# Set the default active repository
python -m termscockpit.server --repo tosdr --repos tosdr contrib
```

The server starts immediately and loads repositories in the background. The UI shows each collection's loading progress; pages for a collection become interactive as soon as that repository is ready.

### CLI options

| Option | Default | Description |
|---|---|---|
| `--host` | `127.0.0.1` | Bind address |
| `--port` | `5000` | Bind port |
| `--repo` | `contrib` | Default active repository name |
| `--repos` | all | Whitespace-separated list of repository names to load |
| `--no-changes` | off | Disable git change indexing (faster startup) |
| `--debug` | off | Enable Flask debug mode |

### Known repositories

| Name | Label | Group |
|---|---|---|
| `contrib` | Community | OpenTermsArchive |
| `genai-contrib` | GenAI Community | OpenTermsArchive |
| `india` | India | OpenTermsArchive — Regions |
| `kenya` | Kenya | OpenTermsArchive — Regions |
| `cote-divoire` | Côte d'Ivoire | OpenTermsArchive — Regions |
| `dating` | Dating | OpenTermsArchive — Topics |
| `p2b-compliance` | Platform-to-Business Compliance | OpenTermsArchive — Topics |
| `pga` | Professional Gaming | OpenTermsArchive — Topics |
| `dsa-reports` | DSA Reports | OpenTermsArchive — Topics |
| `genai-eu` | GenAI EU | OpenTermsArchive — Topics |
| `demo` | Demo | OpenTermsArchive — Other |
| `sandbox` | Sandbox | OpenTermsArchive — Other |
| `tosdr` | ToS;DR Snapshots | ToS;DR |

### REST API

Interactive API documentation (Swagger UI) is available at `/api/docs` when the server is running.

**Base URL:** `/api/<repo>/`

#### Services

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/<repo>/services/` | List all services |
| `GET` | `/api/<repo>/services/<service>/documents` | List documents for a service (with version counts and last-modified dates) |

#### Documents

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/<repo>/documents/<path>` | Latest raw HTML content |
| `GET` | `/api/<repo>/documents/<path>/at/<commit>` | Raw HTML at a specific commit |
| `GET` | `/api/<repo>/documents/<path>/readability` | Latest readability-extracted content |
| `GET` | `/api/<repo>/documents/<path>/readability/at/<commit>` | Readability content at a specific commit |
| `GET` | `/api/<repo>/documents/<path>/changes` | Full version history |
| `GET` | `/api/<repo>/documents/<path>/diff/<idx>` | Unified diff at version index (`?text=1` for plain-text diff) |
| `GET` | `/api/<repo>/documents/<path>/text-change-versions` | Indices of versions with actual text changes |

#### Repositories

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/repos/` | List all known repositories and their loading status |
| `POST` | `/api/repos/switch` | Switch active repository (body: `{"name": "<repo>"}`) |
| `GET` | `/api/repos/status/<name>` | Poll loading status for a specific repository |

#### LexGlue Analysis

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/<repo>/lexglue/<path>/analyze` | Stream UNFAIR_TOS analysis via Server-Sent Events |

Query parameters for the analyze endpoint:

| Parameter | Default | Description |
|---|---|---|
| `model` | `Agreemind/lexglue-legalbert-unfair-tos` | HuggingFace model identifier |
| `commit` | latest | Analyse a specific document version |
| `readability` | `1` | Strip navigation chrome before analysis |

SSE events: `html`, `status`, `paragraph`, `error`, `done`.

---

## Architecture

```
termscockpit/
├── termscockpit.py       # TermsCockpit — main library class
├── git_change_index.py   # GitChangeIndex — SQLite-backed git history index
├── document_utils.py     # HTML utilities (html_to_text, text_hash, apply_readability)
├── lexglue.py            # UNFAIR_TOS classification engine (clf / nli / sim backends)
└── server/
    ├── app.py            # Flask app factory, multi-repo background loading
    ├── views.py          # Page routes (thin — serve templates only)
    ├── api.py            # REST API blueprints (thin wrappers over the library)
    ├── lexglue_api.py    # SSE streaming endpoint (thin wrapper over lexglue.py)
    ├── repos_api.py      # Repository management endpoints
    ├── __main__.py       # CLI entry point
    ├── static/
    │   ├── css/style.css
    │   └── js/app.js
    └── templates/
        ├── base.html
        ├── index.html      # Services listing + collection switcher
        ├── service.html    # Documents listing for a service
        ├── document.html   # Document viewer with version history
        ├── versions.html   # Version history chart + diff viewer
        ├── evolution.html  # Service-level document evolution
        └── lexglue.html    # UNFAIR_TOS analysis
```

**Design principles:**

- The server API is a thin serialisation layer — all business logic lives in the Python package and can be used programmatically without Flask.
- Repositories load in a background thread; the server accepts requests immediately and returns `503` for collections that are not yet ready.
- The SQLite change index is built with a single subprocess call (`git log --numstat`) and updated incrementally, making restarts fast even for repositories with hundreds of thousands of commits.

---

## License

This project is licensed under the [MIT License](https://opensource.org/license/mit).

---

## Acknowledgements

Special thanks to [ToS;DR](https://github.com/tosdr) and [Open Terms Archive](https://github.com/OpenTermsArchive) for maintaining the document repositories that power this tool.
