Metadata-Version: 2.5
Name: semantic-json-transport
Version: 0.2.0a3
Summary: Query-conditioned EvidenceRegion transport with source-grounded retrieval and research-preview evidence construction architecture.
Author: Semantic JSON Contributors
License: Apache-2.0
License-File: LICENSE
Keywords: dynamic-chunking,evidence-retrieval,json,llm,provenance,rag,retrieval,semantic-search
Requires-Python: >=3.10
Requires-Dist: numpy>=1.26
Requires-Dist: sentence-transformers>=3.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: lite
Provides-Extra: training
Requires-Dist: torch>=2.1; extra == 'training'
Description-Content-Type: text/markdown

# Semantic JSON Transport

> **Find locations cheaply. Compose query-conditioned evidence. Transport the original source.**

Semantic JSON Transport is an experimental RAG/evidence-construction project for long documents. Its core premise is that **the indexed unit does not have to be the final context unit**.

Documents are stored as fine-grained, source-grounded `SemanticUnit`s. A query first locates promising positions, then evidence is composed from the original source while preserving provenance.

## Install

```bash
pip install semantic-json-transport
```

Current package version: **0.2.0a3**.

The PyPI runtime remains the stable v0.2 `SemanticRepository` / Region Compatibility API. The newer scalar-potential Local Composer and neural Global Evidence-Set Value/Search stack described below is a **research-preview architecture under benchmark validation**, not yet claimed as the default packaged inference runtime.

## Runtime quick start

```python
from semantic_json import SemanticRepository

text = """
B기업은 주요 거래처와 공급계약을 체결하고 있다.
해당 계약은 내년 말 만료될 예정이다.
해당 거래처 매출 의존도는 높은 수준이다.
중장기 상환능력을 낙관하기 어렵다.
"""

repo = SemanticRepository()
repo.add_text(
    text,
    document_id="company_b",
    source_uri="documents/company_b.txt",
)

result = repo.search("B기업의 중장기 상환능력", top_k=5)

print(result.to_json())
print(result.to_text())
```

Lite fallback:

```python
from semantic_json import LiteEmbedder, SemanticRepository

repo = SemanticRepository(
    embedder=LiteEmbedder(),
    region_model="lite",
)
```

## Current research architecture

The research line has converged on a three-stage semantic architecture plus deployment indexing:

```text
Offline indexing / hierarchy / reusable embeddings
                    ↓
              Navigation
       where should we look?
                    ↓
       Local EvidenceRegion Composer
          Φθ(Q, R, ∂R)
                    ↓
          source-contiguous basins
             R1*, R2*, ... Rm*
                    ↓
        Global Evidence-Set Value
              Vω(Q, E)
                    +
           Structured Search
                    ↓
             Evidence Set E*
                    ↓
              downstream LLM
```

### Local composition

A Local Region is a source-contiguous interval `R=[i,j]`. The research Local Composer learns a boundary-aware scalar potential:

```text
Φθ(Q, R, ∂R)
```

Neighboring boundary edits are compared through potential differences rather than independent edit heads. This replaced an earlier action-field formulation that produced cyclic trajectories.

### Global composition

Local potential values are intentionally **not** treated as globally calibrated relevance scores. Distant Local maxima are handled by a separate Evidence-Set Value:

```text
Vω(Q, E)
```

The current research implementation uses a unary relevance foundation plus a learned set-interaction residual and combines the learned value with structured beam search.

### Scaling principle

Large-document deployment is designed as coarse-to-fine reduction:

```text
N primitive units
      ↓
approximate / hierarchical navigation
      ↓
k candidate anchors
      ↓
Local potential trajectories
      ↓
m unique Region basins
      ↓
Global value-guided search
```

Reusable representation work belongs at ingestion time. Expensive neural evaluation should operate only on the reduced query-time frontier.

## Benchmark status

The research code is evaluated on QASPER with two deliberately separated metric families:

1. **Internal diagnostics** — source-unit Evidence F1, Boundary IoU, Oracle candidate-space F1, and selection efficiency.
2. **QASPER-compatible text-evidence evaluation** — paragraph Evidence F1 for the text evidence represented by the current source-unit pipeline.

We do **not** combine MSE, correlation, pair accuracy, or other diagnostics into a synthetic checkpoint score. Global checkpoints are selected by the actual deployed validation search Evidence F1.

### Fixed-RAG comparison

The phase-1 benchmark is designed around the practical baseline the project aims to improve:

```text
Fixed token chunks
      ↓
vector retrieval
      ↓
Top-k context
```

Fixed chunk length, overlap, and retrieval depth are tuned on validation. The v0.16 held-out experiment showed higher project text-evidence paragraph F1 for query-conditioned Evidence Construction than for the validation-tuned fixed-chunk + exact-cosine vector-retrieval baseline under the same **maximum** context-token budget.

The v0.17 closing benchmark strengthens this test with:

- a validation-tuned Fixed RAG + neural reranker baseline;
- actual-token quality Pareto analysis;
- paired bootstrap 95% confidence intervals;
- no arbitrary composite performance score.

Until the complete official QASPER task protocol, including non-text evidence conditions, is reproduced, published QASPER leaderboard scores are treated as orientation points rather than apples-to-apples claims.

See [`docs/research_architecture_v0.17.md`](docs/research_architecture_v0.17.md) for the current research formulation and benchmark rules.

## SemanticUnit and source provenance

`SemanticUnit` is a fine-grained, contiguous source span. It is a retrieval/composition primitive, not a rewritten summary.

Every returned EvidenceRegion retains source coordinates. The source remains authoritative:

```python
region = result[0]
repo.locate(region)
repo.get_source(region)
repo.verify_source(region)
```

Core invariant:

```python
document.text[region.start_char:region.end_char] == region.text
```

## Structured transport

`SearchResult` is canonically serializable as versioned JSON while plain text remains available for inspection.

```python
result.to_dict()
result.to_json()
result.to_text()
```

The structured output retains document identity, Region identity, anchors, source coordinates, original text, and boundary traces.

## Training utilities

The package includes dependency-light utilities for Region Compatibility training-data generation:

```python
from semantic_json import LLMRegionTeacher, RegionDatasetBuilder, RegionQuery

teacher = LLMRegionTeacher(my_judge_callable, name="internal-llm")
builder = RegionDatasetBuilder(teacher, random_seed=13)
```

Fine-tuning tools remain optional:

```bash
pip install "semantic-json-transport[training]"
```

LLM teachers are offline labeling tools and are not mandatory production dependencies.

## Research principles

The project currently follows these rules:

- keep original-source provenance intact;
- separate Navigation, Local Composition, and Global Selection;
- use diagnostics to understand models, but select checkpoints using deployed-task performance;
- compare against validation-tuned conventional RAG rather than weak straw-man baselines;
- report evidence quality and compute/context cost as separate axes instead of inventing a weighted aggregate score;
- treat negative benchmark results as evidence against the hypothesis rather than adding complexity solely to rescue it.

## Status

`semantic-json-transport` is alpha software. APIs and research formulations may change. The v0.2 runtime remains usable for source-grounded retrieval experiments; the v0.17 research architecture should be treated as experimental until it is converted into a stable package API and reproduced across additional datasets.

Apache-2.0 license.
