Metadata-Version: 2.4
Name: toa
Version: 0.1.1
Summary: Convert documents (txt, markdown, pdf, etc.) to HTML
Project-URL: Homepage, https://github.com/sunshmo/toa
Project-URL: Repository, https://github.com/sunshmo/toa
Author-email: sunshmo <sunsilently@outlook.com>
License-File: LICENSE
Keywords: audio,document-parser,excel,file-parser,pdf,video
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.12
Requires-Dist: charset-normalizer>=3.4.7
Requires-Dist: httpx[socks]>=0.28.1
Requires-Dist: magika>=0.5.1
Requires-Dist: markdownify>=1.2.2
Requires-Dist: mcp>=1.27.2
Requires-Dist: mistune>=3.2.1
Requires-Dist: onnxruntime>=1.20.1
Requires-Dist: pygments>=2.20.0
Requires-Dist: python-socks>=2.8.1
Requires-Dist: websockets>=16.0
Provides-Extra: all
Requires-Dist: feedparser; extra == 'all'
Requires-Dist: mammoth; extra == 'all'
Requires-Dist: olefile; extra == 'all'
Requires-Dist: opencv-python; extra == 'all'
Requires-Dist: openpyxl; extra == 'all'
Requires-Dist: pandas; extra == 'all'
Requires-Dist: pdfplumber; extra == 'all'
Requires-Dist: python-pptx; extra == 'all'
Requires-Dist: selectolax; extra == 'all'
Requires-Dist: vosk; extra == 'all'
Requires-Dist: xlrd; extra == 'all'
Provides-Extra: audio
Requires-Dist: vosk; extra == 'audio'
Provides-Extra: csv
Requires-Dist: pandas; extra == 'csv'
Provides-Extra: docx
Requires-Dist: mammoth; extra == 'docx'
Provides-Extra: epub
Requires-Dist: selectolax; extra == 'epub'
Provides-Extra: excel
Requires-Dist: openpyxl; extra == 'excel'
Requires-Dist: xlrd; extra == 'excel'
Provides-Extra: lite
Requires-Dist: feedparser; extra == 'lite'
Requires-Dist: mammoth; extra == 'lite'
Requires-Dist: olefile; extra == 'lite'
Requires-Dist: openpyxl; extra == 'lite'
Requires-Dist: pandas; extra == 'lite'
Requires-Dist: pdfplumber; extra == 'lite'
Requires-Dist: python-pptx; extra == 'lite'
Requires-Dist: selectolax; extra == 'lite'
Requires-Dist: vosk; extra == 'lite'
Requires-Dist: xlrd; extra == 'lite'
Provides-Extra: outlook
Requires-Dist: olefile; extra == 'outlook'
Provides-Extra: pdf
Requires-Dist: pdfplumber; extra == 'pdf'
Provides-Extra: pptx
Requires-Dist: python-pptx; extra == 'pptx'
Provides-Extra: rss
Requires-Dist: feedparser; extra == 'rss'
Provides-Extra: video
Requires-Dist: opencv-python; extra == 'video'
Description-Content-Type: text/markdown

Toa is a library for converting documents to HTML.

> [!tip]
> By default, it returns a string-type HTML fragment. You can decide how to render it yourself,
> such as beautifying the page display.
> 
> The built-in HTML templates can return fully structured HTML content, supporting Mermaid stylesheets,
> formulas, and syntax highlighting, which can be enabled selectively.


Toa supports the following conversions:

- Ebooks: EPUB
- Emails: EML, MSG(outlook)
- Documents: DOCX, PDF, TXT, MD(Markdown)
- Spreadsheets: XLSX, XLS, CSV
- Presentations: PPTX
- Data Exchange: JSON, JSONL
- Markup Languages: XML, HTML
- Multimedia: audio files, video files, supports audio-to-text conversion and video subtitles extraction.
- Resources from the Internet
- ...

Toa provides convenience for online previewing of documents, emails, data reports, e-books, etc.

Of course, Toa can also convert HTML to Markdown.


## Install

```bash
pip install toa
```

Can be used after installation
```bash
toa -h
toa-mcp -help
```

It may only support following content can be converted to HTML.
- such as `json`, `jsonl`, `txt`, `md`, etc.
- file streams
- external links (such as web crawlers, FTP, and WebSockets)

Extras: `[mcp]`, `[csv]`, `[docx]`, `[epub]`, `[excel]`, `[outlook]`, `[pdf]`, `[pptx]`, `[rss]`, `[audio]`, `[video]`

```bash
uv add 'toa[mcp, csv, excel]'
```

- `[lite]` supports capabilities other than `[video]`.
- `[all]` will install all dependencies. If you don't want to make a choice, this might be the best option, and an additional OCR engine needs to be installed.


**with OCR** If OCR image recognition is required, you can choose an OCR engine. Such as `paddleocr`, `easyocr`, `pytesseract`, for example:
```bash
uv add paddlepaddle paddleocr
```


## Usage

Use in command line
```bash
toa md.md

toa md.md -o md.html --wrapper default --mermaid --highlight --formula
```

> [!note]
> If the file does not exist, it may be treated as a string and ultimately returned as an HTML fragment.

- `--wrapper`: HTML fragment wrapper
- `-o`: output filename
- `--mermaid`: mermaid rendering is required.
- `--highlight`: highlight rendering is required.
- `--formula`: Requires support for rendering mathematical formulas.


standard input
```bash
cat pdf.pdf | toa

cat pdf.pdf | toa --wrapper default -o pdf.html --source-url https://example.com/pdf.pdf --engine default
```
- `--source-url`: An accessible file address ensures the page displays correctly.
- `--engine`: rendering engine name

> [!important]
> Some content can only be displayed correctly with server support.


fetch from websocket endpoint
```bash
toa ws://examples.com/ws
toa wss://examples.com/wss
```

fetch from ftp server
```bash
toa ftp://ftp.example.com/readme.txt
```


Use it in Python coding

```py
from toa import Toa

toa = Toa()
result = toa.convert('md.md')
print(result.content)  # <p>md.md</p>
```


## toa-mcp

Toa has built-in MCP capabilities. The corresponding separate command is `toa-mcp`. `toa-mcp` is a server based on the **MCP (Model Context Protocol)** protocol, providing a standardized API interface for the `toa` command-line tool.

Multi-mode Operation:
- **HTTP**: Provides a RESTful API interface, suitable for web integration.
- **SSE**: Server sends events, supports real-time push notifications.
- **WebSocket**: Bidirectional persistent connection for real-time interaction.
- **Stdio**: Standard input/output communication, suitable for integration with AI assistants.


endpoint  | 	method          |	description
--- |------------------| ---
`/sse`        | Get/POST         |	Message data is transmitted in a streaming manner (Server-to-Client).
`/messages`   | GET              |	Standard message retrieval endpoint (Polling/REST).
`/mcp`        | GET              |	Streamable HTTP (Model Context Protocol / Long-polling transport).
`/ws`         | GET (Upgrade)    |	Full-duplex bidirectional streaming (WebSocket connection for real-time interaction).
`/ping`       | GET              |	Check service availability (Heartbeat).



MCP protocol supports
```json
{
  "jsonrpc": "2.0",
  "method": "tools/list",
  "params": {},
  "id": 1
}
```


## Integrations

There are multiple ways to integrate.

**Microservice architecture**

prepare `docker-compose.yml`
```yaml
# docker-compose.yml
services:
  toa-cli:
    image: sunshmo/toa:latest
    entrypoint: ["toa"]
    command: ["--help"]
    stdin_open: true
    tty: true
  toa-mcp:
    image: sunshmo/toa:latest
    entrypoint: ["toa-mcp", "--transport", "http"]
    ports:
      - "56156:56156"

  # other services
```

Build and start service
```bash
docker-compose up toa-cli
docker-compose up -d toa-mcp
```


**AI assistant integration** Such as, Cursor, openclaw, claude, etc.
```json
{
  "mcpServers": {
    "toa": {
      "command": "toa-mcp",
      "args": ["--transport", "stdio"]
    }
  }
}
```

You can also use the docker command.
```json
{
  "mcpServers": {
    "toa": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "-v",
        "./path/data:/data", // resource directory mapping
        "sunshmo/toa:latest",
        "toa-mcp",
        "--transport",
        "stdio"
      ]
    }
  }
}
```


**Integration with coding** If `toa-mcp` is already running, then you can call it in the program. No programming language restrictions. For example, below uses HTTP transport.

```py
async def with_http():
    import httpx

    async with httpx.AsyncClient(follow_redirects=True) as client:
        response = await client.post(
            "http://localhost:56156/mcp",
            headers={"Content-Type": "application/json", "Accept": "application/json"},
            json={"jsonrpc": "2.0", "method": "tools/list", "id": 1},
        )
        print("tools:", response.json())
```

<details><summary>Below is the capability of Java to call toa-mcp</summary>

```java
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class McpHttpClient {
    private static final ObjectMapper mapper = new ObjectMapper();
    
    public static void withHttp() throws Exception {
        HttpClient client = HttpClient.newBuilder()
            .followRedirects(HttpClient.Redirect.NORMAL)
            .connectTimeout(Duration.ofSeconds(10))
            .build();
        
        // build JSON-RPC request
        ObjectNode request = mapper.createObjectNode();
        request.put("jsonrpc", "2.0");
        request.put("method", "tools/list");
        request.put("id", 1);
        
        HttpRequest httpRequest = HttpRequest.newBuilder()
            .uri(URI.create("http://localhost:56156/mcp"))
            .header("Content-Type", "application/json")
            .header("Accept", "application/json")
            .timeout(Duration.ofSeconds(30))
            .POST(HttpRequest.BodyPublishers.ofString(mapper.writeValueAsString(request)))
            .build();
        
        HttpResponse<String> response = client.send(httpRequest, HttpResponse.BodyHandlers.ofString());
        
        if (response.statusCode() == 200) {
            JsonNode result = mapper.readTree(response.body());
            System.out.println("tools: " + result.toPrettyString());
        } else {
            System.err.println("HTTP Error: " + response.statusCode());
        }
    }
    
    public static void main(String[] args) throws Exception {
        withHttp();
    }
}
```
</details>


**with web MCP Inspector debug**

```bash
npx @modelcontextprotocol/inspector
```
The accessible UI address after startup is `http://localhost:6274`. You can switch the `Transport Type` to `STDIO`, `SSE`, or `Streamable HTTP`.

**STDIO**: Enter `toa-mcp` in the `command` field and click `Connect`.

**SSE**:
- Start service: `toa-mcp --transport http`, or `toa-mcp --transport sse`, or `toa-mcp --transport ws` can all start the web service.
- Enter `URL`: `http://localhost:56156/sse`
- click `Connect`

**Streamable HTTP**:
- Enter `URL`: `http://localhost:56156/mcp`
- click `Connect`


## Docker


**Running MCP Web Server** Using Docker to build eliminates the hassle of OCR installation. Running `toa-mcp` service in Docker.
```bash
docker run -d \
  -p 56156:56156 \
  --name toa \
  -v /the/host/dir:/data \
  sunshmo/toa:latest \
  toa-mcp --transport http
```

Verify if the service is usable, visit http://localhost:56156/ping, Seeing "pong" indicates a normal startup.

> [!tip]
> `-v` option maps a host directory to a container directory, allowing you to use a directory path within the container as the target file path when calling the function.


**STDIO**
```bash
docker run --rm -i sunshmo/toa:latest toa < your.eml
```


## Plugin

In reality, there may be files that Toa currently does not support for conversion, or you may want to override Toa's default capabilities. In such cases, you can achieve this by using an extension plugin.

You can do like this:
1. Implement a class that inherits from BaseConverter
1. then register the converter with Toa


Below is one full example scenario.

<details><summary>plugin example</summary>

```py
from typing import BinaryIO

from toa import Toa, BaseConverter, InputInfo, OutputInfo, ConversionResult


class MyToaConverter(BaseConverter):
    def __init__(self) -> None:
        super().__init__()

    @property
    def check_dependencies(self) -> bool:
        """
        Returns a boolean value. Returns True if there are no external dependencies;
        otherwise, the conversion will throw an error.
        """
        return True

    @property
    def suffixes(self) -> list[str]:
        """Define extension (Format: ['.toa'])"""
        return [".toa"]

    @property
    def mime_types(self) -> list[str]:
        return []

    def convert(
        self,
        input_stream: BinaryIO,
        input_info: InputInfo,
        output_info: OutputInfo | None = None,
        **kwargs,
    ) -> ConversionResult:
        return ConversionResult(content="Result from plugin conversion")


toa = Toa()

# register
toa.add_converter(MyToaConverter())

result = toa.convert("abc.toa")
print(result.content)
```
</details>

> [!note]
> Either `suffixes` or `mime_types` must be a value that is not of type `NoneType`.
