Metadata-Version: 2.4
Name: glimo-hsd
Version: 0.1.0
Summary: Glimo HSD privacy-preserving CSV pipeline for harmful-speech datasets.
Author: Glimo
License-Expression: Apache-2.0
Project-URL: Homepage, https://huggingface.co/batinium/glimo-dehatebert-hsd
Project-URL: Repository, https://huggingface.co/batinium/glimo-dehatebert-hsd
Keywords: privacy,hate-speech-detection,nlp,text-anonymization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: hf
Requires-Dist: huggingface-hub>=0.24; extra == "hf"
Requires-Dist: safetensors>=0.4; extra == "hf"
Requires-Dist: torch>=2.2; extra == "hf"
Requires-Dist: transformers>=4.43; extra == "hf"
Provides-Extra: llm
Provides-Extra: api
Requires-Dist: fastapi>=0.115; extra == "api"
Requires-Dist: uvicorn>=0.30; extra == "api"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.8; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Provides-Extra: all
Requires-Dist: fastapi>=0.115; extra == "all"
Requires-Dist: huggingface-hub>=0.24; extra == "all"
Requires-Dist: safetensors>=0.4; extra == "all"
Requires-Dist: torch>=2.2; extra == "all"
Requires-Dist: transformers>=4.43; extra == "all"
Requires-Dist: uvicorn>=0.30; extra == "all"
Dynamic: license-file

# glimo-hsd

Reusable Python package for the Glimo / PrivHSD backend pipeline. The package
processes CSV files into privacy-scrubbed, classifier-ready, optionally
restated outputs while keeping model weights outside the PyPI wheel.

## Install

```bash
pip install glimo-hsd
pip install "glimo-hsd[hf]"  # Transformers / torch classifier support
```

## Python API

```python
from glimo_hsd import PipelineConfig, process_csv

result = process_csv(
    "input.csv",
    config=PipelineConfig(
        text_col="text",
        label_col="hs",
        model_id="batinium/glimo-dehatebert-hsd",
        restatement_backend="none",
        final_scrub=True,
    ),
)

print(result.restated_csv)
print(result.audit_csv)
```

Use `classifier_backend="keyword"` for offline smoke tests. Production runs
should use `classifier_backend="hf"` with the Hugging Face model repo or a local
model directory.

## LLM Restatement Backend

The package does not insert instruct tokens, chat-template markers, or
model-specific control text. Those are handled by the LLM provider or local
runtime.

When `restatement_backend` is `qwen` or `local-http`, Glimo sends an
OpenAI-compatible chat-completions request and requires a structured tool call
with `tool_choice="required"`. That tool contract is intentional: the
restatement step needs exactly one ordered restatement per input row. Providers
or runtimes used for restatement must support OpenAI-style tool calling.

## CLI

```bash
glimo-hsd process input.csv \
  --text-col text \
  --label-col hs \
  --out outputs/run_001 \
  --model-id batinium/glimo-dehatebert-hsd \
  --classifier-backend hf \
  --restatement-backend none \
  --final-scrub
```

The process command writes deterministic artifacts:

```text
source.csv
scrubbed.csv
dehatebert_predictions.csv
token_importances.csv
restatement_input.csv
restated.csv
final_scrubbed.csv
deviation_audit.csv
manifest.json
```

## Hugging Face Model

The default model ID is:

```text
batinium/glimo-dehatebert-hsd
```

Export a local checkpoint with:

```bash
python scripts/export_dehatebert_for_hf.py \
  --checkpoint path/to/final_model \
  --out dist/hf/glimo-dehatebert-hsd
```

Then upload with the current Hugging Face CLI:

```bash
hf auth whoami
hf repos create batinium/glimo-dehatebert-hsd --type model --exist-ok
hf upload batinium/glimo-dehatebert-hsd dist/hf/glimo-dehatebert-hsd --type model
```

## Safety

Classifier scores and restatement audits are decision-support signals. They are
not appropriate for fully automated enforcement without human review. Do not
publish raw challenge data, admin uploads, or generated outputs containing
private source text.
