Metadata-Version: 2.4
Name: docforge-rag
Version: 0.1.0
Summary: Automatically discovers, crawls, versions, chunks, and indexes official software documentation into a RAG-ready knowledge base.
Project-URL: Homepage, https://github.com/aritra0309/DocForge
Project-URL: Repository, https://github.com/aritra0309/DocForge
Project-URL: Issues, https://github.com/aritra0309/DocForge/issues
Project-URL: Documentation, https://docforge.example.com
Project-URL: Changelog, https://github.com/aritra0309/DocForge/blob/main/CHANGELOG.md
Author-email: DocForge Contributors <docforge@example.com>
Maintainer-email: DocForge Maintainers <maintainers@docforge.example.com>
License: Apache-2.0
License-File: LICENSE
Keywords: documentation,embeddings,knowledge-base,llm,rag,vector-search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Requires-Dist: alembic<2,>=1.13
Requires-Dist: chromadb<1,>=0.5
Requires-Dist: cssselect<2,>=1.2
Requires-Dist: faiss-cpu<2,>=1.8
Requires-Dist: httpx<1,>=0.27
Requires-Dist: jsonschema<5,>=4.20
Requires-Dist: lxml<6,>=5.2
Requires-Dist: markdown-it-py<4,>=3.0
Requires-Dist: markdownify<1,>=0.12
Requires-Dist: orjson<4,>=3.10
Requires-Dist: pydantic-settings<3,>=2.3
Requires-Dist: pydantic<3,>=2.8
Requires-Dist: python-dotenv<2,>=1.0
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: readability-lxml<1,>=0.8
Requires-Dist: rich<14,>=13.7
Requires-Dist: sentence-transformers<4,>=3.0
Requires-Dist: sqlalchemy<3,>=2.0
Requires-Dist: structlog<25,>=24.1
Requires-Dist: tenacity<9,>=8.2
Requires-Dist: tiktoken<1,>=0.7
Requires-Dist: trafilatura<2,>=1.6
Requires-Dist: typer<1,>=0.12
Requires-Dist: xxhash<4,>=3.4
Provides-Extra: dev
Requires-Dist: mypy<2,>=1.10; extra == 'dev'
Requires-Dist: pre-commit<4,>=3.7; extra == 'dev'
Requires-Dist: pytest-asyncio<1,>=0.23; extra == 'dev'
Requires-Dist: pytest-cov<6,>=5.0; extra == 'dev'
Requires-Dist: pytest-mock<4,>=3.14; extra == 'dev'
Requires-Dist: pytest<9,>=8.2; extra == 'dev'
Requires-Dist: respx<1,>=0.21; extra == 'dev'
Requires-Dist: ruff<1,>=0.5; extra == 'dev'
Requires-Dist: types-jsonschema<5,>=4.20; extra == 'dev'
Requires-Dist: types-pyyaml<7,>=6.0; extra == 'dev'
Requires-Dist: types-requests<3,>=2.31; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-autorefs<2,>=1.1; extra == 'docs'
Requires-Dist: mkdocs-material<10,>=9.5; extra == 'docs'
Requires-Dist: mkdocs<2,>=1.5; extra == 'docs'
Requires-Dist: mkdocstrings[python]<1,>=0.25; extra == 'docs'
Provides-Extra: jina
Requires-Dist: jina<4,>=3.28; extra == 'jina'
Provides-Extra: js
Requires-Dist: playwright<2,>=1.40; extra == 'js'
Provides-Extra: lancedb
Requires-Dist: lancedb<1,>=0.12; extra == 'lancedb'
Provides-Extra: ml
Requires-Dist: accelerate<1,>=0.30; extra == 'ml'
Requires-Dist: torch<3,>=2.3; extra == 'ml'
Requires-Dist: transformers<5,>=4.40; extra == 'ml'
Provides-Extra: openai
Requires-Dist: openai<2,>=1.30; extra == 'openai'
Provides-Extra: qdrant
Requires-Dist: qdrant-client<2,>=1.10; extra == 'qdrant'
Provides-Extra: weaviate
Requires-Dist: weaviate-client<5,>=4.6; extra == 'weaviate'
Description-Content-Type: text/markdown

# DocForge

> Build a versioned, RAG-ready knowledge base from official software documentation.

[![CI](https://github.com/aritra0309/DocForge/actions/workflows/ci.yml/badge.svg)](https://github.com/aritra0309/DocForge/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/)
[![License](https://img.shields.io/github/license/aritra0309/DocForge.svg)](LICENSE)

DocForge discovers documentation, crawls pages, extracts clean Markdown, classifies and chunks content, creates embeddings, and stores vectors for semantic search. It supports versioned indexes, incremental updates, and replaceable pipeline providers.

## Features

- Zero-config discovery for PostgreSQL, MySQL, MongoDB, FastAPI, React, Kubernetes, and Redis
- Version-aware indexing and filtered semantic search
- Type-aware chunking for guides, tutorials, API references, code, and tables
- Incremental updates and re-embedding without a fresh crawl
- Embedding providers: Sentence Transformers, OpenAI, Voyage, BGE, and Jina
- Vector backends: ChromaDB, FAISS, Qdrant, LanceDB, and Weaviate
- CLI, async Python API, custom plugin contracts, and MkDocs documentation

## Quickstart

Requires Python 3.11+ and internet access. First index downloads default local embedding model and fetches documentation pages.

```bash
git clone https://github.com/aritra0309/DocForge.git
cd DocForge

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

docforge index postgresql --version 17
docforge search "how to create an index" --software postgresql --version 17 --top-k 5
```

DocForge stores local data in `~/.docforge` by default.

## Python API

```python
import asyncio

from docforge import DocForge


async def main() -> None:
    async with DocForge() as forge:
        await forge.index("postgresql", version="17")
        results = await forge.search(
            "how to create an index",
            software="postgresql",
            version="17",
            k=5,
        )

    for result in results:
        print(f"{result.metadata.title}: {result.metadata.url}")


asyncio.run(main())
```

## Configuration

Create `docforge.toml` in project directory. Environment variables with `DOCFORGE_` prefix override file values.

```toml
[crawler]
max_pages_per_version = 500
rate_limit_rps = 2

[storage]
backend = "chromadb"
path = "~/.docforge/vectordb"
```

```bash
export DOCFORGE_CRAWLER__MAX_PAGES_PER_VERSION=100
export DOCFORGE_STORAGE__PATH="$PWD/.docforge/vectordb"
docforge config --source
```

## Common commands

```bash
docforge index redis
docforge update postgresql
docforge reembed postgresql --model BAAI/bge-small-en-v1.5
docforge list
docforge stats postgresql
docforge delete postgresql --version 17
```

Run `docforge --help` or `docforge COMMAND --help` for all options.

## Documentation and examples

- [Getting started](docs/getting-started.md)
- [CLI reference](docs/cli-reference.md)
- [Architecture](docs/architecture.md)
- [Plugin contracts](docs/plugins.md)
- [Examples](examples/)
- [Changelog](CHANGELOG.md)

## Development

```bash
pip install -e ".[dev,docs]"

make lint
make typecheck
make test
make docs-build
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for pull-request process and registry contributions.

## License

Apache-2.0. See [LICENSE](LICENSE).
