Metadata-Version: 2.4
Name: lexoid
Version: 0.1.24
Summary: Turn documents and live websites into agent-ready context with parsing and evidence-grounded browser automation.
License-File: LICENSE
Requires-Python: >=3.11,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Provides-Extra: browse
Requires-Dist: accelerate (>=1.10.1,<2.0.0)
Requires-Dist: agent-framework-core (>=1.16.0,<2.0.0) ; extra == "browse"
Requires-Dist: agent-framework-openai (>=1.14.1,<2.0.0) ; extra == "browse"
Requires-Dist: anthropic (>=0.96.0,<1.0.0)
Requires-Dist: bs4 (>=0.0.2,<0.0.3)
Requires-Dist: click (>=8.0.0,<9.0.0)
Requires-Dist: docx2pdf (>=0.1.8,<0.2.0)
Requires-Dist: google-genai (>=1.56.0,<2.0.0)
Requires-Dist: huggingface-hub (>=0.31.2,<0.32.0)
Requires-Dist: levenshtein (>=0.27.1,<0.28.0)
Requires-Dist: loguru (>=0.7.2,<0.8.0)
Requires-Dist: markdown (>=3.7,<4.0)
Requires-Dist: markdownify (>=1.2.3,<2.0.0)
Requires-Dist: matplotlib (>=3.10.6,<4.0.0)
Requires-Dist: mistralai (>=1.8.2,<2.0.0)
Requires-Dist: nest-asyncio (>=1.6.0,<2.0.0)
Requires-Dist: openai (>=2.18.0,<3.0.0)
Requires-Dist: opencv-python (>=4.10.0.84,<5.0.0.0)
Requires-Dist: openpyxl (>=3.1.5,<4.0.0)
Requires-Dist: paddleocr[doc-parser] (>=3.3.0,<4.0.0)
Requires-Dist: paddlepaddle (==3.2.2)
Requires-Dist: pandas (>=2.2.3,<3.0.0)
Requires-Dist: patchright (>=1.49.0,<2.0.0) ; extra == "browse"
Requires-Dist: pdfplumber (>=0.11.4,<0.12.0)
Requires-Dist: pikepdf (>=9.3.0,<10.0.0)
Requires-Dist: playwright (>=1.49.0,<2.0.0)
Requires-Dist: pptx2md (>=2.0.6,<3.0.0)
Requires-Dist: pypdfium2 (>=4.30.0,<5.0.0)
Requires-Dist: pyqt5 (>=5.15.11,<6.0.0) ; platform_system != "Linux"
Requires-Dist: pyqtwebengine (>=5.15.7,<6.0.0) ; platform_system != "Linux"
Requires-Dist: python-docx (>=1.1.2,<2.0.0)
Requires-Dist: python-dotenv (>=1.0.0,<2.0.0)
Requires-Dist: scikit-image (>=0.25.2,<0.26.0)
Requires-Dist: scikit-learn (>=1.7.1,<2.0.0)
Requires-Dist: tabulate (>=0.9.0,<0.10.0)
Requires-Dist: tenacity (>=9.1.4,<10.0.0)
Requires-Dist: together (>=1.5.34,<2.0.0)
Requires-Dist: torch (>=2.7.0,<3.0.0)
Requires-Dist: transformers (>=4.51.3,<5.0.0)
Description-Content-Type: text/markdown

<div align="center">
  <img src="assets/logo.png">
</div>

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/oidlabs-com/Lexoid/blob/main/examples/example_notebook_colab.ipynb) [![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-yellow)](https://huggingface.co/spaces/oidlabs/Lexoid) [![GitHub license](https://img.shields.io/badge/License-Apache_2.0-turquoise.svg)](https://github.com/oidlabs-com/Lexoid/blob/main/LICENSE) [![PyPI](https://img.shields.io/pypi/v/lexoid)](https://pypi.org/project/lexoid/) [![Docs](https://github.com/oidlabs-com/Lexoid/actions/workflows/deploy_docs.yml/badge.svg)](https://oidlabs-com.github.io/Lexoid/)

**Lexoid** turns messy documents and live websites into **agent-ready context**.

- **`parse()`:** [High-accuracy](#benchmark) document and webpage parsing across static, local LLM, and frontier LLM pipelines.
- **`browse()` (preview):** Goal-driven, evidence-grounded browser automation for live web workflows.

[Documentation](https://oidlabs-com.github.io/Lexoid/)

## Why Lexoid?

Useful information is buried in documents and scattered across websites. Getting it out shouldn't require expensive a custom pipeline for every task.

- **Extract what matters:** Turn documents and web pages into usable text and structured data for downstream workflows.
- **Automate the search, retain the evidence:** Automate browser tasks with evidence.
- **Choose your approach:** Use static parsing, hosted LLMs, or supported local models.
- **Build openly:** Collaborate under the Apache 2.0 license.

# Installation

## Installing with pip

```bash
pip install lexoid
```

To use LLM-based parsing, define the following environment variables or create a `.env` file with the following definitions

```bash
OPENAI_API_KEY=""
GOOGLE_API_KEY=""
```

For local inference with Ollama, no API key is required. Install Ollama, pull the target model, and keep the local server running:

```bash
ollama pull gemma4
export OLLAMA_BASE_URL=127.0.0.1:11434
ollama list
ollama serve

# docker
Reference: https://docs.ollama.com/docker#run-model-locally
CPU example (will most likely be slower; remember to adjust `OLLAMA_TIMEOUT` as needed)
- docker run -d -v ollama:/root/.ollama -p 11434:11434 -e OLLAMA_BASE_URL=0.0.0.0 -e OLLAMA_TIMEOUT=240 --name ollama ollama/ollama
- docker exec -it ollama ollama pull gemma4:latest
```

For webpage retrieval and URL-to-PDF conversion with `parse()`, install Playwright's Chromium browser binaries. The Playwright Python package is installed with Lexoid:

```bash
playwright install --with-deps --only-shell chromium
```

To use the autonomous browser tasks shown below with `browse()`, install the optional agent dependencies and Patchright's Chromium binaries:

```bash
pip install "lexoid[browse]"
patchright install --with-deps chromium
```

## Building `.whl` from source

> [!NOTE] Installing the package from within the virtual environment could cause unexpected behavior, as Lexoid creates and activates its own environment in order to build the wheel.

```
make build
```

## Creating a local installation

To install dependencies:

```bash
make install
```

or, to install with dev-dependencies:

```bash
make dev
```

To activate virtual environment:

```bash
source .venv/bin/activate
```

# Usage

[Example Notebook](https://github.com/oidlabs-com/Lexoid/blob/main/examples/example_notebook.ipynb)

[Example Colab Notebook](https://colab.research.google.com/github/oidlabs-com/Lexoid/blob/main/examples/example_notebook_colab.ipynb)

Here's a quick example to parse documents using Lexoid:

```python
from lexoid.api import parse

parsed_md = parse("https://www.justice.gov/eoir/immigration-law-advisor", parser_type="AUTO")["raw"]
# or
pdf_path = "path/to/immigration-law-advisor.pdf"
parsed_md = parse(pdf_path, parser_type="LLM_PARSE")["raw"]
# or
pdf_path = "path/to/immigration-law-advisor.pdf"
parsed_md = parse(pdf_path, parser_type="STATIC_PARSE")["raw"]

print(parsed_md)
```

## Parameters

- path (str): The file path or URL.
- parser_type (str, optional): The type of parser to use ("LLM_PARSE" or "STATIC_PARSE"). Defaults to "AUTO".
- pages_per_split (int, optional): Number of pages per split for chunking. Defaults to 4.
- max_threads (int, optional): Maximum number of threads for parallel processing. Defaults to 4.
- \*\*kwargs: Additional arguments for the parser.

# Web Pages and Browser Tasks

Use `parse()` when you have a URL to retrieve and parse. Set `as_pdf=True` to render the page to PDF before parsing; pass `save_dir` to keep the generated PDF after the call returns:

```python
from lexoid.api import parse

# Retrieve and parse a webpage
page = parse("https://example.com")

# Render the webpage to PDF, then parse the PDF
page_pdf = parse(
    "https://example.com",
    as_pdf=True,
    save_dir="output/",
    save_filename="example.pdf",
)
print(page_pdf["pdf_path"])
```

## Web Operator (Preview): for browser tasks

Use `browse()` when the task requires goal-driven interaction with a website, including searching, applying filters, and completing multi-step non-binding quote, estimate, eligibility, or lookup forms with values supplied in the request. It returns an evidence-backed answer; it does not convert the source URL to PDF:

```python
import asyncio
from lexoid.api import browse

async def main():
    result = await browse(
        "Visit https://example.com and find the current enterprise plan price."
    )
    print(result.answer)

asyncio.run(main())
```

**Note:**
The navigator is instructed not to log in, create accounts, accept legal
agreements, purchase, bind coverage, pay, or book, and to stop when required
information, authentication, or consequential consent is needed. These are
navigator instructions, not semantic enforcement for every site control;
ambiguous actions should be reviewed. Popups are handled by purpose:
promotional overlays can be dismissed, and cookie banners use reject or
necessary-only options when available. CAPTCHA solving is planned; until it is
implemented, a CAPTCHA that prevents progress is reported as a blocker.

# Command Line Usage

Lexoid provides a command-line interface for document parsing without writing Python code.

## Installation

The CLI is automatically available after installing Lexoid:

```bash
pip install lexoid
lexoid --help
```

Alternatively, use with Python module syntax:

```bash
python -m lexoid --help
```

## Parse Documents

Convert documents to markdown or JSON:

```bash
# Parse to stdout (default markdown)
lexoid parse --input document.pdf

# Save to file
lexoid parse --input document.pdf --output output.md

# Output as JSON (includes metadata, segments, token usage)
lexoid parse --input document.pdf --format json --output result.json

# Use specific parser (STATIC_PARSE, LLM_PARSE, or AUTO)
lexoid parse --input document.pdf --parser-type STATIC_PARSE

# Use specific LLM model
lexoid parse --input document.pdf --model gpt-4o

# Enable verbose logging
lexoid parse --input document.pdf --verbose
```

## Extract Structured Data with JSON Schema

Extract data conforming to a JSON schema:

```bash
# Inline schema
lexoid schema \
  --input document.pdf \
  --schema '{"type": "object", "properties": {"title": {"type": "string"}}}' \
  --output result.json

# Schema from file
lexoid schema \
  --input document.pdf \
  --schema schema.json \
  --output result.json

# Specify LLM provider
lexoid schema \
  --input document.pdf \
  --schema schema.json \
  --api openai \
  --model gpt-4o
```

## Convert to LaTeX

Convert documents to LaTeX format:

```bash
# Convert to stdout
lexoid latex --input document.pdf

# Save to file
lexoid latex --input document.pdf --output output.tex

# Use specific model
lexoid latex --input document.pdf --model gpt-4o
```

## Command-line Options

### Common Options

- `--input, -i`: Input file path (required) - Supports PDF, images, HTML, DOCX, XLSX, PPTX, or URLs
- `--output, -o`: Output file path (optional) - If not specified, output is printed to stdout
- `--verbose, -v`: Enable detailed logging

### Parse Command

```
lexoid parse --help
```

- `--parser-type, -p`: Parser type - `AUTO` (default), `LLM_PARSE`, or `STATIC_PARSE`
- `--model, -m`: LLM model name (default: gemini-3.5-flash-lite, thinking: medium)
- `--pages-per-split`: Pages per chunk (default: 4)
- `--max-processes`: Parallel processes (default: 4)
- `--framework`: Static parser framework - `pdfplumber` or `paddleocr`
- `--format`: Output format - `markdown` (default, plain markdown text) or `json` (full result with metadata, segments, token usage)

### Schema Command

```
lexoid schema --help
```

- `--schema, -s`: JSON schema (file path or inline JSON, required)
- `--model, -m`: LLM model (default: gpt-4o-mini)
- `--api`: API provider - openai, gemini, anthropic, ollama, etc. (auto-detected if not specified)
- `--example-schema`: Provide example data for the schema
- `--fill-single-schema`: Auto-fill single schemas

### LaTeX Command

```
lexoid latex --help
```

- `--model, -m`: LLM model (default: gpt-4o-mini)
- `--api`: API provider - openai, gemini, anthropic, ollama, etc. (auto-detected if not specified)

# Supported API Providers

- Google
- OpenAI
- Hugging Face
- Together AI
- OpenRouter
- Fireworks
- Ollama

# Ollama Local Parsing

Lexoid supports local `LLM_PARSE` inference through Ollama. The initial recommended model is `gemma4:latest`.

```python
from lexoid.api import parse

result = parse(
    "path/to/document.pdf",
    parser_type="LLM_PARSE",
    api_provider="ollama",
    model="gemma4:latest",
    max_processes=1,
)

print(result["raw"])
```

Notes:

- Ollama uses the default local endpoint `http://localhost:11434` unless `OLLAMA_BASE_URL` is set.
- Lexoid forces `max_processes=1` for Ollama-backed parsing to avoid local multiprocess contention.
- `AUTO` routing does not select Ollama in this first version; choose it explicitly with `api_provider="ollama"`.

# Benchmark

Results aggregated across 14 documents.

_Note:_ Benchmarks are currently done in the zero-shot setting.

| Rank | Model                                                                                                     | SequenceMatcher Similarity | Cosine- Similarity | Time (s) | Cost ($) |
| ---- | --------------------------------------------------------------------------------------------------------- | -------------------------- | ------------------ | -------- | -------- |
| 1    | gemini-3.5-flash-lite (thinking: medium)                                                                  | 0.959 (±0.071)             | 0.986 (±0.018)     | 9.84     | 0.00659  |
| 2    | claude-sonnet-5-5 (thinking: low)                                                                         | 0.956 (±0.084)             | 0.991 (±0.010)     | 6.85     | 0.01478  |
| 3    | claude-opus-5-5 (thinking: low)                                                                           | 0.952 (±0.088)             | 0.983 (±0.022)     | 9.13     | 0.02805  |
| 4    | claude-haiku-5-5 (thinking: medium)                                                                       | 0.949 (±0.074)             | 0.980 (±0.025)     | 7.96     | 0.00121  |
| 5    | gpt-6.1-sol (thinking: low)                                                                               | 0.947 (±0.100)             | 0.986 (±0.014)     | 31.41    | 0.02349  |
| 6    | gemini-3.8-flash (thinking: low)                                                                          | 0.932 (±0.117)             | 0.975 (±0.032)     | 20.73    |          |
| 7    | gpt-6-astra (thinking: medium)                                                                            | 0.931 (±0.125)             | 0.986 (±0.014)     | 30.83    | 0.14173  |
| 8    | ![Default](https://img.shields.io/badge/Default-2E8B57) AUTO + gemini-3.5-flash-lite (thinking: medium)   | 0.928 (±0.112)             | 0.989 (±0.015)     | 8.93     | 0.005315 |
| 9    | ![Lower-cost](https://img.shields.io/badge/Lower--cost-2E8B57) AUTO + claude-haiku-5-5 (thinking: medium) | 0.925 (±0.086)             | 0.982 (±0.023)     | 8.05     | 0.001215 |
| 10   | gemini-4-argon (thinking: high)                                                                           | 0.890 (±0.229)             | 0.969 (±0.071)     | 69.63    |          |
| 11   | gpt-5.2                                                                                                   | 0.890 (±0.193)             | 0.975 (±0.036)     | 33.32    | 0.03959  |
| 12   | mistral-ocr-latest                                                                                        | 0.882 (±0.106)             | 0.932 (±0.091)     | 5.75     | 0.00121  |
| 13   | gpt-5.5                                                                                                   | 0.874 (±0.209)             | 0.939 (±0.138)     | 72.11    | 0.14495  |
| 14   | gpt-6-luna (thinking: low)                                                                                | 0.869 (±0.146)             | 0.953 (±0.053)     | 13.35    | 0.00096  |
| 15   | claude-3-5-sonnet-20241022                                                                                | 0.858 (±0.184)             | 0.930 (±0.098)     | 17.32    | 0.01804  |
| 16   | gpt-5.4-mini                                                                                              | 0.835 (±0.210)             | 0.948 (±0.066)     | 13.14    | 0.00902  |
| 17   | gpt-5-mini                                                                                                | 0.819 (±0.201)             | 0.917 (±0.104)     | 52.84    | 0.00811  |
| 18   | gpt-5                                                                                                     | 0.807 (±0.215)             | 0.919 (±0.088)     | 98.12    | 0.05505  |
| 19   | gpt-5.4                                                                                                   | 0.803 (±0.238)             | 0.936 (±0.150)     | 31.98    | 0.03887  |
| 20   | claude-sonnet-4-20250514                                                                                  | 0.801 (±0.188)             | 0.905 (±0.136)     | 22.02    | 0.02056  |
| 21   | claude-opus-4-20250514                                                                                    | 0.789 (±0.184)             | 0.886 (±0.148)     | 29.55    | 0.09513  |
| 22   | accounts/fireworks/models/llama4-maverick-instruct-basic                                                  | 0.772 (±0.203)             | 0.930 (±0.117)     | 16.02    | 0.00147  |
| 23   | gemini-3-flash-preview                                                                                    | 0.766 (±0.293)             | 0.858 (±0.210)     | 39.38    | 0.00969  |
| 24   | claude-opus-4-8                                                                                           | 0.764 (±0.254)             | 0.863 (±0.154)     | 11.10    | 0.03195  |
| 25   | claude-sonnet-4-6                                                                                         | 0.757 (±0.302)             | 0.843 (±0.206)     | 16.50    | 0.01804  |
| 26   | gpt-4.1-mini                                                                                              | 0.754 (±0.249)             | 0.803 (±0.193)     | 23.28    | 0.00347  |
| 27   | accounts/fireworks/models/llama4-scout-instruct-basic                                                     | 0.754 (±0.243)             | 0.942 (±0.063)     | 13.36    | 0.00087  |
| 28   | gpt-4o                                                                                                    | 0.752 (±0.269)             | 0.896 (±0.123)     | 28.87    | 0.01469  |
| 29   | gpt-4o-mini                                                                                               | 0.728 (±0.241)             | 0.850 (±0.128)     | 18.96    | 0.00609  |
| 30   | claude-haiku-4-5-20251001                                                                                 | 0.683 (±0.300)             | 0.841 (±0.187)     | 7.86     | 0.00504  |
| 31   | claude-3-7-sonnet-20250219                                                                                | 0.646 (±0.397)             | 0.758 (±0.297)     | 57.96    | 0.01730  |
| 32   | gpt-4.1                                                                                                   | 0.637 (±0.301)             | 0.787 (±0.185)     | 35.37    | 0.01498  |
| 33   | google/gemma-3-27b-it                                                                                     | 0.604 (±0.342)             | 0.788 (±0.297)     | 23.16    | 0.00020  |
| 34   | ds4sd/SmolDocling-256M-preview                                                                            | 0.603 (±0.292)             | 0.705 (±0.262)     | 507.74   | 0.00000  |
| 35   | gpt-5.4-nano                                                                                              | 0.600 (±0.309)             | 0.856 (±0.119)     | 22.51    | 0.00321  |
| 36   | microsoft/phi-4-multimodal-instruct                                                                       | 0.589 (±0.273)             | 0.820 (±0.197)     | 14.00    | 0.00045  |
| 37   | qwen/qwen-2.5-vl-7b-instruct                                                                              | 0.498 (±0.378)             | 0.630 (±0.445)     | 14.73    | 0.00056  |

# Citation

If you use Lexoid in production or publications, please cite accordingly and acknowledge usage. We appreciate the support 🙏

