Quantify field-level extraction risk in PDFs, images, and structured JSON outputs using Gemini and self-hosted VLMs. Isolate value-only logprobs & compute semantic entropy to automate HITL routing.
Strips fixed structural JSON characters ({}, [], ":") that artificially inflate confidence, measuring pure field extraction uncertainty.
Groups stochastic outputs into semantic equivalence classes (Kuhn et al., 2023) to eliminate string-formatting false alarms.
Seamlessly integrates Google Gemini (google-genai), vLLM (GuidedDecodingParams), Ollama, MLX-VLM, and local OpenAI endpoints.
Evaluates bottleneck perplexity and semantic entropy to route payloads into AUTO_ACCEPT, NEEDS_REVIEW, or REJECT workflows.
from pydantic import BaseModel
from multimodal_uncertainty import GeminiUncertaintyEngine, PipelineDecisionEngine
class InvoiceSchema(BaseModel):
invoice_number: str
total_amount: float
vendor: str
# 1. Single-pass logprobs + multi-sample semantic entropy
engine = GeminiUncertaintyEngine()
node_result = engine.extract_and_evaluate(
image_or_pdf_bytes=pdf_bytes,
mime_type="application/pdf",
prompt="Extract invoice fields.",
schema_cls=InvoiceSchema
)
# 2. Decision Matrix Routing
decision_engine = PipelineDecisionEngine(ppl_threshold=2.2, entropy_threshold=0.5)
result = decision_engine.evaluate([node_result])
print(f"Action: {result.action} | Bottleneck PPL: {result.bottleneck_perplexity}")Comparing model perplexity on valid vs out-of-domain targets on a synthetic sofa scene.