Metadata-Version: 2.5
Name: zleap_parser
Version: 0.3.1
Summary: zleap 文档解析 SDK：多格式文件/URL 解析为 knowledge-only OCTX 归档
Author: zleap
License-Expression: MIT
License-File: LICENSE
License-File: LICENSES/Apache-2.0.txt
Keywords: document,knowledge,markdown,octx,parser
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing :: Markup
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: camoufox==0.5.8
Requires-Dist: octx>=0.1.6
Requires-Dist: openai<2,>=1.68
Requires-Dist: playwright>=1.40
Requires-Dist: pycryptodome>=3.19
Requires-Dist: pydantic<3,>=2.13.4
Requires-Dist: tzdata>=2024.1; sys_platform == 'win32'
Requires-Dist: uuid6>=2025.0.1
Provides-Extra: all-databases
Requires-Dist: elasticsearch<10,>=8.0; extra == 'all-databases'
Requires-Dist: psycopg2-binary>=2.9; extra == 'all-databases'
Requires-Dist: pymongo>=4.6; extra == 'all-databases'
Requires-Dist: pymysql>=1.1; extra == 'all-databases'
Requires-Dist: sqlalchemy>=2.0; extra == 'all-databases'
Provides-Extra: bigdb
Requires-Dist: clickhouse-connect>=0.8; extra == 'bigdb'
Requires-Dist: psycopg2-binary>=2.9; extra == 'bigdb'
Requires-Dist: pymysql>=1.1; extra == 'bigdb'
Provides-Extra: bigdb-clickhouse
Requires-Dist: clickhouse-connect>=0.8; extra == 'bigdb-clickhouse'
Provides-Extra: bigdb-mysql
Requires-Dist: pymysql>=1.1; extra == 'bigdb-mysql'
Provides-Extra: bigdb-postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'bigdb-postgres'
Provides-Extra: db
Requires-Dist: oracledb>=2.0; extra == 'db'
Requires-Dist: psycopg2-binary>=2.9; extra == 'db'
Requires-Dist: pymssql>=2.2; extra == 'db'
Requires-Dist: pymysql>=1.1; extra == 'db'
Provides-Extra: db-all
Requires-Dist: clickhouse-connect>=0.8; extra == 'db-all'
Requires-Dist: oracledb>=2.0; extra == 'db-all'
Requires-Dist: psycopg2-binary>=2.9; extra == 'db-all'
Requires-Dist: pymssql>=2.2; extra == 'db-all'
Requires-Dist: pymysql>=1.1; extra == 'db-all'
Provides-Extra: db-mssql
Requires-Dist: pymssql>=2.2; extra == 'db-mssql'
Provides-Extra: db-mysql
Requires-Dist: pymysql>=1.1; extra == 'db-mysql'
Provides-Extra: db-oracle
Requires-Dist: oracledb>=2.0; extra == 'db-oracle'
Provides-Extra: db-postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'db-postgres'
Provides-Extra: dev
Requires-Dist: build>=1.5.0; extra == 'dev'
Requires-Dist: eventlet>=0.37; extra == 'dev'
Requires-Dist: gevent>=24.11; extra == 'dev'
Requires-Dist: jsonschema>=4.26; extra == 'dev'
Requires-Dist: mypy>=2.3.0; extra == 'dev'
Requires-Dist: pytest-mock>=3.15.1; extra == 'dev'
Requires-Dist: pytest>=9.1.1; extra == 'dev'
Requires-Dist: ruff>=0.16.2; extra == 'dev'
Requires-Dist: twine>=7.0.0; extra == 'dev'
Requires-Dist: uvloop>=0.21; (sys_platform != 'win32') and extra == 'dev'
Provides-Extra: dingtalk-oa
Requires-Dist: mcp<3,>=2; extra == 'dingtalk-oa'
Provides-Extra: dingtalk-profile
Requires-Dist: keyring>=25.0; extra == 'dingtalk-profile'
Provides-Extra: docx
Requires-Dist: docling>=2.73.1; extra == 'docx'
Requires-Dist: markitdown[docx]>=0.1.7; extra == 'docx'
Provides-Extra: elasticsearch
Requires-Dist: elasticsearch<10,>=8.0; extra == 'elasticsearch'
Provides-Extra: email-all
Requires-Dist: google-api-python-client>=2.170; extra == 'email-all'
Requires-Dist: httpx>=0.28; extra == 'email-all'
Requires-Dist: msal>=1.31; extra == 'email-all'
Provides-Extra: email-gmail
Requires-Dist: google-api-python-client>=2.170; extra == 'email-gmail'
Provides-Extra: email-graph
Requires-Dist: httpx>=0.28; extra == 'email-graph'
Requires-Dist: msal>=1.31; extra == 'email-graph'
Provides-Extra: kafka
Requires-Dist: kafka-python>=2.0; extra == 'kafka'
Provides-Extra: media
Requires-Dist: docling-slim[format-video]<3,>=2.119.0; extra == 'media'
Requires-Dist: docling[asr]<3,>=2.119.0; extra == 'media'
Requires-Dist: setuptools<81,>=80; extra == 'media'
Provides-Extra: mongodb
Requires-Dist: pymongo>=4.6; extra == 'mongodb'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.1; extra == 'mysql'
Provides-Extra: pdf
Requires-Dist: docling>=2.73.1; extra == 'pdf'
Requires-Dist: markitdown[pdf]>=0.1.7; extra == 'pdf'
Requires-Dist: onnxruntime<2,>=1.17; extra == 'pdf'
Requires-Dist: pypdfium2>=5.12.1; extra == 'pdf'
Requires-Dist: rapidocr>=3.9.2; extra == 'pdf'
Provides-Extra: postgresql
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgresql'
Provides-Extra: ppt
Requires-Dist: docling>=2.73.1; extra == 'ppt'
Requires-Dist: markitdown[pptx]>=0.1.7; extra == 'ppt'
Provides-Extra: rabbitmq
Requires-Dist: kombu>=5.4; extra == 'rabbitmq'
Provides-Extra: redis
Requires-Dist: redis>=5.0; extra == 'redis'
Provides-Extra: service
Requires-Dist: alembic>=1.13; extra == 'service'
Requires-Dist: cryptography>=42.0; extra == 'service'
Requires-Dist: fastapi>=0.115; extra == 'service'
Requires-Dist: httpx>=0.27; extra == 'service'
Requires-Dist: minio>=7.2; extra == 'service'
Requires-Dist: python-multipart>=0.0.20; extra == 'service'
Requires-Dist: sqlalchemy>=2.0; extra == 'service'
Requires-Dist: uvicorn[standard]>=0.34; extra == 'service'
Provides-Extra: sql
Requires-Dist: sqlalchemy>=2.0; extra == 'sql'
Provides-Extra: tiktok
Requires-Dist: curl-cffi<0.17,>=0.14; extra == 'tiktok'
Provides-Extra: xlsx
Requires-Dist: docling>=2.73.1; extra == 'xlsx'
Requires-Dist: markitdown[xlsx]<0.2,>=0.1.7; extra == 'xlsx'
Requires-Dist: matplotlib>=3.11.1; extra == 'xlsx'
Requires-Dist: openpyxl>=3.1.5; extra == 'xlsx'
Description-Content-Type: text/markdown

# zleap_parser

`zleap_parser` 是 zleap 的**数据接入与归档 SDK**：把本地文件、公开 URL 以及企业内的
办公套件、代码仓库、邮件、数据库、RSS 与通用 API 数据，统一解析并归档为知识型
`.octx` 文件，供下游 AI / RAG 直接消费。

PyPI 分发包只包含 SDK 与连接器适配器。当前版本 **0.3.1**，支持 Python 3.11+。

## 安装

```bash
python -m pip install zleap_parser

# URL 采集首次使用时准备浏览器二进制
python -m camoufox fetch
```

按需安装可选能力：

```bash
python -m pip install "zleap_parser[pdf]"
python -m pip install "zleap_parser[xlsx]"
python -m pip install "zleap_parser[media]"        # 另需系统 ffmpeg
python -m pip install "zleap_parser[db-all]"
python -m pip install "zleap_parser[tiktok]"
python -m pip install "zleap_parser[dingtalk-oa]"
python -m pip install "zleap_parser[dingtalk-profile]"
```

SQLite、Markdown 归档和普通文本解析可直接运行。RTF/XLSX 预览需要系统
`soffice`/`libreoffice`；URL 采集需要 Camoufox 浏览器二进制。

## 快速开始

### 解析本地文件或 URL

```python
import zleap_parser
from zleap_parser import CacheConfig, Parser

zleap_parser.config(cache=CacheConfig(cache_namespace="sdk-default"))
parser = Parser()

parsed = parser.parse(file="/path/to/document.pdf")

# 默认 json：{root, files, tree} 文件目录树
tree = parsed["result"]
print(tree["root"])            # 解包后的本地目录
print(tree["files"])           # [{path, local_path, size}, ...]
print(parsed["usage"])         # 用量账本

# 历史契约：返回 *.octx 路径
octx_path = parser.parse(
    file="/path/to/document.pdf", output_type="octx"
)["result"]
```

`url` 与 `file` 必须二选一。归档产物包含 `knowledge/main.md`、metadata 和原始附件；
不会生成事项、关系或向量文件。

### 连接器取数

```python
from zleap_parser.connector import Connector, RSSAdapter

adapter = RSSAdapter(url="https://example.com/feed.xml", limit=20)
result = Connector().fetch(adapter=adapter)

# 默认 json：每个成功文档一个目录树对象
for tree in result["results"]:
    print(tree["root"])

# 历史契约：返回 *.octx 路径列表
paths = Connector().fetch(adapter=adapter, output_type="octx")["results"]
```

## 输出类型

`Parser.parse()`、`Connector.fetch()` 以及连接器的 `email_results` / `sync_cursor`
共用 `output_type` 参数，大小写不敏感：

| `output_type` | 输出 |
| --- | --- |
| `json`（默认） | `{root, files, tree}`：解包目录、文件清单与嵌套目录树；`knowledge/` 固定排在最前 |
| `octx` | 已存在的 knowledge-only `.octx` 路径（单文档为字符串，连接器为列表） |

`json` 模式的 `files[]` 提供：

```json
{
  "root": "/abs/path/document",
  "files": [
    {
      "path": "knowledge/main.md",
      "local_path": "/abs/path/document/knowledge/main.md",
      "size": 1234
    }
  ],
  "tree": [
    {
      "name": "knowledge",
      "type": "dir",
      "path": "knowledge",
      "children": [
        {
          "name": "main.md",
          "type": "file",
          "path": "knowledge/main.md",
          "local_path": "/abs/path/document/knowledge/main.md",
          "size": 1234
        }
      ]
    }
  ]
}
```

已有归档可单独解包：

```python
from zleap_parser import materialize_octx_tree

tree = materialize_octx_tree("/path/to/document.octx")
```

## 内置数据源

共 29 个，均从 `zleap_parser.connector` 直接导入：

- **通用 API / RSS**：`ApiRequestAdapter`、`RSSAdapter`
- **搜索与网页**：`WebSearchAdapter`、`DeepCrawlAdapter`、`TrendingNewsAdapter`
- **企业微信 / 飞书**：`WxWorkChatAdapter`、`FeishuBotChatAdapter`、
  `FeishuChatApiAdapter`、`FeishuGroupListAdapter`、`FeishuDriveInventoryAdapter`、
  `FeishuWikiInventoryAdapter`、`SalesmartlyChatAdapter`
- **钉钉**：`DingTalkChatAdapter`、`DingTalkGroupListAdapter`、
  `DingTalkWikiInventoryAdapter`、`DingTalkDriveFilesAdapter`、
  `DingTalkDocumentAdapter`、`DingTalkOAAdapter`
- **邮件**：`EmailAdapter`（IMAP / POP3 / Gmail / Microsoft Graph）
- **代码仓库**：`GithubRepositoryAdapter`、`GitlabRepositoryAdapter`、
  `GiteeRepositoryAdapter`、`AtomgitRepositoryAdapter`
- **数据库**：`DbAdapter`、`BigDbAdapter`
- **内容平台**：`WechatPublicAccountAdapter`、`TwitterAdapter`、`TikTokAdapter`、
  `ZhipinResumeAdapter`

数据源统一通过 `Connector.fetch()` 取数；邮件额外返回邮件产物与 sync-cursor，
Boss 直聘额外写出结构化记录 `records.jsonl`。

## 配置

```python
import zleap_parser
from zleap_parser import LLMConfig

zleap_parser.config(
    llm=LLMConfig(
        base_url="https://api.example.com/v1",
        api_key="sk-...",
        model="gpt-4o-mini",
    )
)
```

- 普通解析、非 Git 连接器和 `pack_*` 不要求 `llm`，返回完整零用量。
- Git 仓库分析需要统一 `LLMConfig`；usage 明细统一为 `kind="llm"`。
- `CacheConfig.cache_namespace` 默认 `"sdk-default"`，参与所有缓存 key/signature。
  多租户场景应为每个任务创建独立配置快照，例如
  `tenant-<sha256(tenant_id) 前 32 位>`，不要在并发任务中修改全局配置。
- `supports_multimodal=False` 为默认值，只有显式设为 `True` 才声明模型接受多模态消息。

## 支持格式

文本、源码、PDF、RTF、DOCX、PPTX、XLSX、HTML、XML、图片和音视频。
格式识别优先依据 magic bytes/内容特征，不依赖扩展名；新增格式通过
`BaseAdapter` 插件接入。

## 版本变更

### 0.3.1

- 修复 Google / Bing 搜索结果获取：解析 `/goto` 跳转链接、轮询等待结果节点、
  规范化 Google 结果 URL。
- PyPI 描述改为 SDK 专用文档，移除与分发包无关的内容。
- 优化 `json` 输出渲染：同一 bundle 归档只解包一次，文件大小只读取一次。

### 0.3.0

- `Parser.parse()` / `Connector.fetch()` 新增 `output_type`（默认 `json`）。
- `json` 返回 `{root, files, tree}` 目录树；`octx` 保持历史路径契约。
- 补齐 `ZhipinResumeAdapter` 等全部内置数据源的 SDK 导出。
- 新增 `materialize_octx_tree()` 公共入口。

## 许可证

MIT
