Metadata-Version: 2.4
Name: schift-ko-pii
Version: 0.5.0
Summary: Korean PII detection — 34.1M dual-path LoRA model (person + address + organization)
Project-URL: Homepage, https://github.com/schift-io/schift-ko-pii
Project-URL: Documentation, https://schift.io/docs/pii
Project-URL: Model Card, https://huggingface.co/schift-io/schift-ko-pii-v6
Project-URL: Bug Tracker, https://github.com/schift-io/schift-ko-pii/issues
Author-email: "Schift Inc." <oss@schift.io>
License: Schift License v2.0
        
        Copyright (c) 2026 Schift, Inc. (Room821 Co., Ltd.)
        
        Based on the Apache License, Version 2.0, with the following additional terms.
        
        ------------------------------------------------------------------------
        
        ADDITIONAL TERMS — Revenue Threshold
        
        The rights granted under this License are subject to the following condition:
        
          If the Legal Entity exercising rights under this License (or any entity
          that controls, is controlled by, or is under common control with such
          entity) has annual gross revenue exceeding ten million United States
          dollars (USD $10,000,000) in its most recently completed fiscal year,
          the rights granted under Sections 2 and 3 below do not apply to
          commercial use of the Work or Derivative Works. Such entities must
          obtain a separate commercial license from the Licensor.
        
          "Commercial use" means use of the Work or Derivative Works, directly or
          indirectly, for or in connection with revenue-generating activities,
          including but not limited to: offering the Work as part of a paid
          product or service, using the Work to process data in a commercial
          pipeline, or embedding the Work in commercial software.
        
          Research, education, evaluation, personal projects, and use by
          non-profit organizations are always permitted regardless of revenue.
        
        ------------------------------------------------------------------------
        
        Apache License
                                   Version 2.0, January 2004
                                http://www.apache.org/licenses/
        
           TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
        
           1. Definitions.
        
              "License" shall mean the terms and conditions for use, reproduction,
              and distribution as defined by Sections 1 through 9 of this document,
              together with the Additional Terms above.
        
              "Licensor" shall mean Schift Inc.
        
              "Legal Entity" shall mean the union of the acting entity and all
              other entities that control, are controlled by, or are under common
              control with that entity. For the purposes of this definition,
              "control" means (i) the power, direct or indirect, to cause the
              direction or management of such entity, whether by contract or
              otherwise, or (ii) ownership of fifty percent (50%) or more of the
              outstanding shares, or (iii) beneficial ownership of such entity.
        
              "You" (or "Your") shall mean an individual or Legal Entity
              exercising permissions granted by this License.
        
              "Source" form shall mean the preferred form for making modifications,
              including but not limited to software source code, documentation
              source, and configuration files.
        
              "Object" form shall mean any form resulting from mechanical
              transformation or translation of a Source form, including but
              not limited to compiled object code, generated documentation,
              and conversions to other media types.
        
              "Work" shall mean the work of authorship, whether in Source or
              Object form, made available under the License, as indicated by a
              copyright notice that is included in or attached to the work.
        
              "Derivative Works" shall mean any work, whether in Source or Object
              form, that is based on (or derived from) the Work and for which the
              editorial revisions, annotations, elaborations, or other modifications
              represent, as a whole, an original work of authorship.
        
              "Contribution" shall mean any work of authorship, including
              the original version of the Work and any modifications or additions
              to that Work or Derivative Works thereof, that is intentionally
              submitted to Licensor for inclusion in the Work by the copyright owner
              or by an individual or Legal Entity authorized to submit on behalf of
              the copyright owner.
        
              "Contributor" shall mean Licensor and any individual or Legal Entity
              on behalf of whom a Contribution has been received by Licensor and
              subsequently incorporated within the Work.
        
           2. Grant of Copyright License. Subject to the terms and conditions of
              this License (including the Additional Terms), each Contributor hereby
              grants to You a perpetual, worldwide, non-exclusive, no-charge,
              royalty-free, irrevocable copyright license to reproduce, prepare
              Derivative Works of, publicly display, publicly perform, sublicense,
              and distribute the Work and such Derivative Works in Source or Object
              form.
        
           3. Grant of Patent License. Subject to the terms and conditions of
              this License (including the Additional Terms), each Contributor hereby
              grants to You a perpetual, worldwide, non-exclusive, no-charge,
              royalty-free, irrevocable (except as stated in this section) patent
              license to make, have made, use, offer to sell, sell, import, and
              otherwise transfer the Work, where such license applies only to those
              patent claims licensable by such Contributor that are necessarily
              infringed by their Contribution(s) alone or by combination of their
              Contribution(s) with the Work to which such Contribution(s) was
              submitted. If You institute patent litigation against any entity
              alleging that the Work or a Contribution incorporated within the Work
              constitutes direct or contributory patent infringement, then any
              patent licenses granted to You under this License for that Work shall
              terminate as of the date such litigation is filed.
        
           4. Redistribution. You may reproduce and distribute copies of the
              Work or Derivative Works thereof in any medium, with or without
              modifications, and in Source or Object form, provided that You
              meet the following conditions:
        
              (a) You must give any other recipients of the Work or
                  Derivative Works a copy of this License; and
        
              (b) You must cause any modified files to carry prominent notices
                  stating that You changed the files; and
        
              (c) You must retain, in the Source form of any Derivative Works
                  that You distribute, all copyright, patent, trademark, and
                  attribution notices from the Source form of the Work; and
        
              (d) If the Work includes a "NOTICE" text file as part of its
                  distribution, then any Derivative Works that You distribute must
                  include a readable copy of the attribution notices contained
                  within such NOTICE file.
        
           5. Submission of Contributions. Unless You explicitly state otherwise,
              any Contribution intentionally submitted for inclusion in the Work
              by You to the Licensor shall be under the terms and conditions of
              this License, without any additional terms or conditions.
        
           6. Trademarks. This License does not grant permission to use the trade
              names, trademarks, service marks, or product names of the Licensor,
              except as required for reasonable and customary use in describing the
              origin of the Work and reproducing the content of the NOTICE file.
        
           7. Disclaimer of Warranty. Unless required by applicable law or
              agreed to in writing, Licensor provides the Work (and each
              Contributor provides its Contributions) on an "AS IS" BASIS,
              WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
              implied, including, without limitation, any warranties or conditions
              of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
              PARTICULAR PURPOSE. You are solely responsible for determining the
              appropriateness of using or redistributing the Work and assume any
              risks associated with Your exercise of permissions under this License.
        
           8. Limitation of Liability. In no event and under no legal theory,
              whether in tort (including negligence), contract, or otherwise,
              unless required by applicable law or agreed to in writing, shall
              any Contributor be liable to You for damages, including any direct,
              indirect, special, incidental, or consequential damages of any
              character arising as a result of this License or out of the use or
              inability to use the Work.
        
           9. Accepting Warranty or Additional Liability. While redistributing
              the Work or Derivative Works thereof, You may choose to offer,
              and charge a fee for, acceptance of support, warranty, indemnity,
              or other liability obligations and/or rights consistent with this
              License. However, in accepting such obligations, You may act only
              on Your own behalf and on Your sole responsibility, not on behalf
              of any other Contributor.
        
           END OF TERMS AND CONDITIONS
License-File: LICENSE
Keywords: korean,ner,onnx,pii,privacy,token-classification
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=0.20.0
Requires-Dist: ko-pii==1.15.2
Requires-Dist: safetensors>=0.4.0
Requires-Dist: torch>=2.0.0
Requires-Dist: transformers>=4.40.0
Provides-Extra: extended
Requires-Dist: ko-pii==1.15.2; extra == 'extended'
Description-Content-Type: text/markdown

---
language: ko
license: other
license_name: schift-2.0
license_link: LICENSE
library_name: transformers
pipeline_tag: token-classification
tags:
  - pii
  - korean
  - ner
  - privacy
  - token-classification
datasets:
  - custom
---

# schift-ko-pii-v6

**34.1M parameter Korean PII detector built on a dual-path LoRA encoder.**

The `0.4.0` release keeps the v6 detector contract and adds an optional,
typed selective-adoption flow for structured and contextual categories.
The base install does not install `ko-pii`; install the `extended` extra only
when those additional deterministic categories are needed.

## Release status

This source tree prepares `schift-ko-pii` `0.4.0`. The previously published
baseline was `0.3.3`. The Cloud Run ONNX service under `services/pii` is a
separate deployment lane; this package does not bundle its ONNX artifacts.

## Quick start

```bash
pip install schift-ko-pii
```

The original detector API remains available:

```python
from schift_ko_pii import detect

spans = detect("피고 김민수의 전화번호는 010-1234-5678이다.")
# [
#   {"start": 3, "end": 6, "label": "private_person", ...},
#   {"start": 14, "end": 27, "label": "private_phone", ...},
# ]
```

For a typed operational result, use the selective-adoption flow:

```python
from schift_ko_pii import AnalysisConfig, ProcessingMode, analyze_text

result = analyze_text(
    "피고 김민수의 전화번호는 010-1234-5678이다.",
    config=AnalysisConfig(mode=ProcessingMode.PERMISSIVE),
)

# The result contains typed detections, policy actions, BLOCK-only masking,
# counts, and a metadata-only review queue. It has no separate source-text
# field; its policy text can still preserve REVIEW/ALLOW spans for operators.
print(result.summary)
print(result.masking.masked_text)
print(result.review_items())
```

## Selective-adoption flow

The public workflow is deliberately ordered:

**detect -> assess -> BLOCK-only masking -> review queue**

`analyze_text()` or `analyze()` runs detection, sends non-sensitive detection
metadata to `assess()`, and masks only spans whose action is `Action.BLOCK`.
`Action.REVIEW` detections remain unmasked and are represented by
`ReviewItem` values in `PiiResult.review_queue`; callers can create a typed
`FeedbackPatch` with `propose_feedback_patch()` without persisting raw PII.
`Action.ALLOW` detections are retained in the typed result but are not masked.
The resulting `masking.masked_text` is therefore a policy output, not a safe
untrusted-egress string: use a caller-owned all-span redaction step before
sending it to logs, external APIs, or other untrusted surfaces.

Detector confidence and operational risk are separate. `ProcessingMode` sets
the action thresholds:

| Mode | BLOCK threshold | REVIEW threshold |
|---|---|---|
| `AUDIT` | never blocks | all detections are allowed for audit output |
| `PERMISSIVE` | `CRITICAL` risk at score `>= 0.95` | `HIGH` risk at score `>= 0.70` |
| `STRICT` | `MEDIUM` risk at score `>= 0.70` | `LOW` risk at score `>= 0.50` |
| `BALANCED` | `HIGH` risk at score `>= 0.80` | `MEDIUM` risk at score `>= 0.60` |
| `PARANOID` | `LOW` risk at score `>= 0.50` | all lower-risk detections |

`PERMISSIVE` is an action-policy choice, not a change to the v6 detector's
`score_threshold`. Use `AnalysisConfig.score_threshold` separately when
configuring detection.

## Extended profiles (opt-in)

```bash
pip install "schift-ko-pii[extended]"
```

The optional adapter is enabled per request:

```python
from schift_ko_pii import AnalysisConfig, analyze_text

result = analyze_text(
    "사업자등록번호 104-81-49532, 직책 팀장",
    config=AnalysisConfig(extended=True, extended_profile="contextual"),
)
```

`extended_profile="structured"` adopts deterministic identifier and anchor
categories such as business/corporate registration numbers, medical
insurance and prescription identifiers, PNU, postal code, fax, employee,
document, petition, and drug IDs. `extended_profile="contextual"` includes
that structured set plus contextual attributes such as nationality, birth
date, education, major, position, age, height, and weight.

The taxonomy registry is exposed through `LABELS`,
`STRUCTURED_UPSTREAM_LABELS`, `CONTEXTUAL_UPSTREAM_LABELS`,
`EXCLUDED_UPSTREAM_LABELS`, `lookup_label()`, `label_for_upstream()`, and
`upstream_labels_for_profile()`. The adapter excludes categories already
owned by the v6 detector or current Schift postprocessing, including person,
address, phone, email, existing structured identifiers, URLs, IPs, and legal
case references. Existing v6 spans win overlaps, except a generic
`account_number` span may be refined by a more specific extended label.

## Masking boundaries

Masking is request-local and occurs only after policy assessment. Select a
`MaskingStrategy` in `AnalysisConfig` or call `mask_text()` directly with
typed `MaskSpan` values:

- `TOKEN`: replace with a stable label token such as `[PII_PHONE_1]`.
- `REDACT`: replace with `[REDACTED]`.
- `PARTIAL`: retain a small leading/trailing portion for recognition.
- `HASHED`: replace with a deterministic local SHA-256 digest.

These strategies are output transformations, not custody. No strategy stores
a reverse map, restores source values, or talks to Vault. Central Vault
custody, retention, tenant isolation, KMS, and audit requirements remain a
separate caller-side project.

## Documents and the document-helper boundary

Document APIs accept already extracted text and provenance, not files:

```python
from schift_ko_pii import (
    ExtractedPageInput,
    analyze,
    from_pages,
)

document = from_pages(
    (
        ExtractedPageInput(page_num=1, text="첫 페이지", source="helper"),
        ExtractedPageInput(page_num=2, text="둘째 페이지", source="helper"),
    ),
    source_id="doc-123",
)
result = analyze(document)
```

`from_text()`, `from_pages()`, and `from_document_helper()` build the typed
text-only envelope. `DocumentInput` preserves the concatenated text and page
boundaries; `SourceSpan` and `PageSpan` preserve character offsets and page
provenance. `scan_document()` is the convenience scan over that envelope.

The package does not parse HWP/HWPX, DOCX, XLSX, PDF, or other file formats.
Use the document-helper service (or another caller-owned parser) to produce a
text-only envelope, then pass it to this package. Do not treat the envelope as
file storage or a Vault integration.

## Postprocessing and legacy API

Postprocessing is enabled by default for `detect()`. It applies Korean-specific
structured-ID validation, checksum checks where applicable, context-aware span
merging, and false-positive suppression for legal case numbers and statute
references. The legacy `detect()` path preserves original input text by
default; pass `normalize=True` when you want NFKC-normalized model input and
source-offset remapping. The typed `analyze()`/`analyze_text()` flow enables
that normalization by default. Pass `postprocess=False` for encoder heads
only (person, address, and organization).

Existing root exports remain available: `detect`, `mask`, `apply`, `assess`,
`detect_extended`, `detect_extended_entities`, `Action`, `ProcessingMode`,
`RiskLevel`, and `ExtendedDependencyError`.

For compatibility, `AnonymizationResult` is an alias of `PiiResult`, and both
`anonymize_text` and `anonymize` are aliases of `analyze_text`. They do not
introduce a second execution path.

## API (free)

For production use without managing model files:

```python
from schift import Schift

client = Schift(api_key="...")  # free at schift.io
result = client.pii.redact("김민수의 전화번호는 010-1234-5678입니다.")
```

## Labels

The stable local taxonomy is available as immutable `TaxonomyEntry` values in
`LABELS`. Common labels include:

| Label | Description | Examples |
|---|---|---|
| `private_person` | Person names | 김민수, 황보영희, Lee Jenny |
| `private_phone` | Phone numbers | 010-1234-5678, 02-1234-5678 |
| `private_email` | Email addresses | user@example.com |
| `private_address` | Street/postal addresses | 서울특별시 강남구 테헤란로 521 |
| `private_date` | Dates | 2024년 3월 15일, 2024-03-15 |
| `private_url` | URLs and IP addresses | instagram.com/user, 192.168.1.1 |
| `account_number` | Structured account/identity surfaces | 850205-1234567, M12345678 |
| `secret` | Secrets, API keys, passwords | |

## Benchmark

The benchmark suite is included under `benchmark/`.

```bash
python benchmark/run_benchmark.py
python benchmark/run_benchmark.py --postprocess
python benchmark/run_benchmark.py --hf-model LiquidAI/LFM2.5-Encoder-350M-PII-Detector
```

## Model details

- **Checkpoint**: `schift-io/schift-ko-pii-v6`
- **Architecture**: dual-path LoRA encoder with person, address, and organization heads
- **Training**: LoRA adapter on Korean legal/financial/admin examples
- **Format**: safetensors release source
- **Inference**: custom `transformers`/PyTorch dual-path loader
- **Max length**: 512 tokens
- **Tagging scheme**: `O/B/I/E/S`

## License

[Schift License v2.0](LICENSE) — Apache 2.0 base with a revenue threshold.
Free for everyone under $10M annual revenue. Research, education, and
non-profit use always permitted. Companies above the threshold: contact
hello@schift.io.

## Citation

```bibtex
@software{schift_ko_pii_2026,
  author = {Schift Inc.},
  title = {schift-ko-pii: Korean PII Detection Model},
  year = {2026},
  url = {https://huggingface.co/schift-io/schift-ko-pii-v6},
}
```
