Metadata-Version: 2.4
Name: sieve-lens
Version: 0.12.0
Summary: Invisible Prompt Observation Engine (Zero-Dependency, Deterministic)
Author: Kai IWASAKI
License: MIT
Project-URL: Homepage, https://github.com/neguseatama/sieve-lens
Project-URL: Repository, https://github.com/neguseatama/sieve-lens.git
Project-URL: Bug Tracker, https://github.com/neguseatama/sieve-lens/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Text Processing :: Filters
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: pdf
Requires-Dist: pypdf>=4.0; extra == "pdf"
Provides-Extra: pdf-images
Requires-Dist: pypdf>=4.0; extra == "pdf-images"
Requires-Dist: Pillow>=10.0; extra == "pdf-images"
Requires-Dist: pytesseract>=0.3.10; extra == "pdf-images"
Provides-Extra: ocr
Requires-Dist: Pillow>=10.0; extra == "ocr"
Requires-Dist: pytesseract>=0.3.10; extra == "ocr"
Provides-Extra: html-css
Requires-Dist: tinycss2>=1.1; extra == "html-css"
Provides-Extra: html-css-full
Requires-Dist: tinycss2>=1.1; extra == "html-css-full"
Requires-Dist: cssselect2>=0.7; extra == "html-css-full"
Provides-Extra: dashboard
Requires-Dist: Jinja2>=3.0; extra == "dashboard"
Requires-Dist: Plotly>=5.0; extra == "dashboard"
Provides-Extra: pdf-report
Requires-Dist: weasyprint>=60.0; extra == "pdf-report"
Requires-Dist: Jinja2>=3.0; extra == "pdf-report"
Requires-Dist: Plotly>=5.0; extra == "pdf-report"
Requires-Dist: kaleido>=0.2; extra == "pdf-report"
Provides-Extra: all
Requires-Dist: pypdf>=4.0; extra == "all"
Requires-Dist: Pillow>=10.0; extra == "all"
Requires-Dist: pytesseract>=0.3.10; extra == "all"
Requires-Dist: tinycss2>=1.1; extra == "all"
Requires-Dist: cssselect2>=0.7; extra == "all"
Requires-Dist: Jinja2>=3.0; extra == "all"
Requires-Dist: Plotly>=5.0; extra == "all"
Requires-Dist: weasyprint>=60.0; extra == "all"
Requires-Dist: kaleido>=0.2; extra == "all"
Dynamic: license-file

**English** | [日本語](README.ja.md)

# Sieve Lens v0.13.0

## 💡 Concept

Modern AI evaluation systems parse documents as text.
Humans read documents as rendered output.
These two views are not the same, and the gap between them can be exploited:

- Zero-width characters (`U+200B`...) that a model reads but a human cannot see.
- CSS rules (`display:none`) that hide text from a human but not from a parser.
- Out-of-band channels (comments, metadata, headers, footnotes) that a parser
  reads but a human never notices.

**Sieve Lens observes this divergence.**  
It does not judge intent. It surfaces what is invisible, with location and
decoded content, so that a human can decide what to do.

> The name "Sieve Lens" comes from the design philosophy of seeing through a
> document the way a lens reveals what the naked eye cannot.

---

## 📐 7-bit Observation Space

| Bit | Name | What it observes |
|-----|------|------------------|
| H1 | Parseability | The file can be read and parsed |
| H2 | Zero-Width Density | Density of zero-width chars exceeds threshold |
| H3 | Bidi Controls | Bidi control characters present |
| H4 | Format Concealment | CSS hiding rules (`display:none`, etc.) present |
| H5 | Out-of-Band Channel | Meaningful content in comments / metadata / headers / footers / footnotes |
| H6 | Script Mixing | Latin+Cyrillic or Latin+Greek mixed within a token |
| H7 | Contiguous Payload | A run of ≥ N contiguous invisible chars exists |

Mask format: `H1H2H3H4-H5H6H7` (e.g., `1000-100`).

---

## 📦 Supported Formats

| Format | Coverage | Dependencies |
|--------|----------|--------------|
| `.txt`, `.md` | Plain text, full character-level scanning | none (core) |
| `.html`, `.htm` | HTML with inline-style and `<style>` block concealment detection; optional external CSS resolution | none (core); `tinycss2` (optional) |
| `.docx` | Body, comments, core properties, headers, footers, footnotes, endnotes | none (core) |
| `.pdf` | Body text, metadata, annotations, invisible text (Tr 3) | `pypdf` (optional) |
| `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.tif`, `.tiff`, `.webp` | Metadata (EXIF / PNG text / XMP) + OCR text | `Pillow`, `pytesseract` + tesseract binary (optional) |

**Not supported**: audio, video.

---

## 🔥 Key Features

- **7-bit observation space (H1–H7)**  
  Independent, deterministic detection of each class of invisible content.
- **Zero dependencies**  
  Python standard library only. Runs offline, no telemetry, no network.
- **Deterministic**  
  Same input always yields the same observation, on any platform.
- **Explainable**  
  Every observation carries location and decoded content, so a human can verify.
- **Observation, not judgment**  
  The engine reports evidence. It never declares "this is an attack".
- **Self-contained HTML report** (since v0.1)  
- **PDF support** via `pypdf` (since v0.2)  
- **Image OCR support** via `Pillow` + `pytesseract` (since v0.3)
- **Interactive HTML dashboard** (since v0.5)  
  Batch-analyze a directory and produce a single self-contained dashboard
  with summary cards, mask distribution, hypothesis activation charts,
  and a sortable file table. Requires `Jinja2` + `Plotly`.
- **Printable PDF report** (since v0.6)  
  Generate an A4 PDF report from a directory of documents. Includes
  summary, hypothesis table, mask distribution, and a color-coded file
  list. Requires `weasyprint` (and optionally `kaleido` for charts).
- **Image layer analysis** (since v0.7)  
  Detects pixel-level concealment in images: transparent RGBA layers
  carrying non-zero RGB values, and low-contrast text with a WCAG ratio
  below 2:1. Requires `Pillow`.
- **Full CSS selector resolution** (since v0.8)  
  Extends the HTML extractor with full CSS selector matching via
  `cssselect2`: descendant, child, sibling, attribute, and pseudo-class
  selectors. Hidden rules are only reported when they match an element.
- **Report internationalization (i18n)** (since v0.9)  
  Dashboard and PDF reports are available in English and Japanese.
  Auto-detection selects the language based on input document content.
- **PDF embedded image analysis** (since v0.10)  
  Extracts images embedded in PDF pages and applies OCR, transparent
  layer detection, low-contrast detection, and metadata analysis to
  each image. Requires `pypdf` + `Pillow`.

---

## 📦 Installation

```bash
# Core (zero-dependency)
pip install git+https://github.com/neguseatama/sieve-lens.git

# With PDF support
pip install "git+https://github.com/neguseatama/sieve-lens.git#egg=sieve-lens[pdf]"

---

## 💻 Quick Start

```python
from sieve_lens import SieveLensEngine, format_report, format_report_html

engine = SieveLensEngine()
obs = engine.observe("resume.docx")

print(obs.mask)                    # e.g. "1000-100"
print(obs.h_states)                # {'H1': 1, 'H2': 0, ..., 'H7': 0}
print(format_report(obs))          # human-readable text report

html = format_report_html(obs)     # self-contained HTML report
```

To write the HTML report to a file:

```python
from sieve_lens import write_report_html
write_report_html(obs, "report.html")
```

### PDF (requires the `pdf` extra)

```python
from sieve_lens import SieveLensEngine
from sieve_lens_ext.pdf import install

engine = SieveLensEngine()
install(engine)                    # attach .pdf support
obs = engine.observe("resume.pdf")
print(obs.mask)

---

## 🔬 Testing

# v0 test suite (26 tests)
python -m unittest discover -s tests -p "test_sieve_lens.py" -v

# v0.1 test suite (14 tests)
python -m unittest discover -s tests -p "test_sieve_lens_v0_1.py" -v

# v0.2 PDF extension (7 tests, requires pypdf)
python -m unittest discover -s tests -p "test_sieve_lens_pdf.py" -v

---

## 🔗 Redact Bridge (v0.12.0)

`sieve_lens_ext.redact_bridge` exports an observation catalog that
[Sieve Redact](https://github.com/neguseatama/sieve-redact) consumes directly:

    python -m sieve_lens_ext.redact_bridge input.txt catalog.json
    python -m sieve_redact input.txt -o clean.txt --from-lens catalog.json

- Catalog format (version 1): `observations[]` with `kind`, `codepoint`,
  `count`, `positions` (first-seen order).
- Look-alike characters (Cyrillic/Greek -> Latin, full-width digits
  and letters -> ASCII, 102 entries) carry an optional
  `twin` field, which Redact converts into a replacement rule; invisible
  characters become deletions.
- Deterministic, stdlib-only. The bridge imports the core character tables,
  so the two engines can never disagree on what counts as invisible.
- End-to-end check: `python3 probe_redact_roundtrip.py`
  (synthetic input -> catalog -> redact -> re-observe clean).

## v0.11.0 Highlights

- **Fixed critical regressions**: PDF embedded-image analysis and HTML body
  text extraction (which had silently disabled H2/H3/H6/H7 detection), and
  EXIF UserComment/SubIFD reading.
- **Security**: report output is now HTML-escaped (untrusted specimens can no
  longer inject markup into dashboards/PDF reports), and local stylesheet
  resolution is confined to the specimen's directory tree.
- **Deterministic PDF reports**: byte-identical output across builds,
  verified in CI (fixed dcterms dates + pinned XMP DocumentID).
- **Chained extensions**: `sieve_lens_ext.image_layers` now composes with
  `sieve_lens_ext.ocr` automatically; new `SieveLensEngine.get_extractor()`.

**Invisible Prompt Observation Engine**

*Sieve Lens is not a tool that distrusts AI.*  
*It is a tool that helps AI receive trustworthy input.*

> Zero-dependency, deterministic, explainable observation of invisible content
> in documents (resumes, submissions, reports) that may be intended to
> influence AI-based evaluation systems.

Extends the [Sieve](https://github.com/neguseatama/sieve-core) series to the
domain of **machine-readable vs. human-visible divergence**.

[![CI](https://github.com/neguseatama/sieve-lens/actions/workflows/test.yml/badge.svg)](https://github.com/neguseatama/sieve-lens/actions/workflows/test.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python Version](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)](https://www.python.org/)

---

## ⚠️ Known Limitations (v0.1)

1. **PDF and images are not supported.** This is a deliberate scope decision.
2. **Semantic intent is not evaluated.** A visible but manipulative prompt is
   outside the scope of this tool.
3. **Only inline CSS is inspected.** External stylesheets and CSS classes are
   not resolved.
4. **Character-table coverage is fixed.** New invisible techniques require
   updating the tables.
5. **This is an observation engine, not a detector.** Findings should always
   be confirmed by human review.
6. **Homoglyph twins are non-ASCII -> ASCII only** — lookalike pairs
   inside ASCII (O/0, l/1) and CJK lookalikes are out of scope

---

## 📄 License

MIT License. See [LICENSE](LICENSE) for details.

---

## 👤 Author

* **Kai IWASAKI**
