Metadata-Version: 2.5
Name: ershu-worker
Version: 1.0.0.post20260812
Summary: Analysis worker service for Ershu.
Project-URL: Documentation, https://ershu.org
Project-URL: Repository, https://github.com/ershu-lab/ershu-worker
Project-URL: Issues, https://github.com/ershu-lab/ershu-worker/issues
Author: Ershu contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: code-analysis,developer-tools,ershu,language-server
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.14
Requires-Dist: ershu-core<2.0.0,>=1.0.1.post20260812
Requires-Dist: langgraph<2.0.0,>=1.2.10
Requires-Dist: langsmith<1.0.0,>=0.10.15
Requires-Dist: pydantic<3.0.0,>=2.10.0
Requires-Dist: pyyaml<7.0.0,>=6.0.3
Requires-Dist: sqlalchemy[asyncio]<3.0.0,>=2.0.36
Requires-Dist: tree-sitter-language-pack==1.8.1
Requires-Dist: tree-sitter==0.25.2
Provides-Extra: dev
Requires-Dist: basedpyright>=1.35.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.25.0; extra == 'dev'
Requires-Dist: pytest>=8.3.0; extra == 'dev'
Requires-Dist: ruff>=0.15.17; extra == 'dev'
Description-Content-Type: text/markdown

# Ershu Worker

[简体中文](README-CN.md)

Ershu Worker is the background analysis service for Ershu. It is for operators running an Ershu installation and developers working on repository analysis. The worker claims jobs from the shared queue, inspects source repositories through read-only tools, and writes code indexes, project relationships, documents, and search data back to Ershu.

This package is one part of Ershu, not a standalone repository scanner. It does not expose an HTTP or MCP endpoint and it does not create analysis jobs. Use the Ershu deployment repository when you need the complete system.

## What the worker does

A run can include the following work, depending on its type and the state of the repository:

- prepare a remote repository cache or an approved local workspace;
- identify source files, manifests, languages, symbols, calls, imports, and public entry points;
- build repository and project knowledge relationships;
- extract OpenAPI, GraphQL, Protocol Buffers, Thrift, AsyncAPI, generated client, database schema, message channel, and framework route facts;
- generate, review, and publish project documents with source references;
- update keyword and vector search data;
- record progress, warnings, model usage, and a final run snapshot.

A deterministic refresh only updates repository facts, code indexes, search
indexes, and knowledge relationships. It does not rewrite AI-generated
documents as a side effect.

## Place in Ershu

```text
Frontend -> API -> PostgreSQL queue -> Worker -> PostgreSQL
                                      |       -> Milvus
                                      |
                                      -> read-only repository workspace

MCP reads the project facts and documents stored by the Worker.
```

Each Ershu component has a separate job:

- `ershu-core` owns shared models, database services, settings contracts, and provider clients.
- The API owns HTTP, authentication, and job creation.
- The Worker owns queue consumption, repository analysis, document workflows, code intelligence, and LSP orchestration.
- MCP exposes stored results and refresh operations to clients.
- The deployment repository selects compatible component, package, and container versions.

The Worker imports `ershu-core`. It does not import the API or MCP packages.

## Safety boundary

Repository analysis inspects project files without running the project:

- analysis tools expose bounded repository, symbol, index, graph, and document operations instead of a general shell;
- LSP containers mount the repository read-only, run without a network, drop Linux capabilities, and have CPU, memory, process, session, and time limits;
- strict LSP mode disables or limits automatic dependency loading and project behavior where the language server supports it;
- optional dependency preparation runs separately, uses supported dependency manifests, and does not share its network access with the LSP container;
- repository credentials, package source credentials, model keys, and `ENCRYPTION_KEY` belong in deployment configuration, not in source control or generated documents.

Semantic LSP mode loads more project context than strict mode. Keep it disabled for repositories you do not trust.

## Requirements

- Python 3.14 or later
- [uv](https://docs.astral.sh/uv/)
- a compatible `ershu-core` package and database schema
- PostgreSQL for the queue, run state, documents, and graph data
- Milvus for vector search
- configured model and embedding channels for runs that need them
- Docker only when LSP analysis or managed dependency preparation is enabled

## Install

Install the Worker as part of a tested component set through the
[`ershu` installer package](https://github.com/ershu-lab/ershu):

```bash
pip install "ershu[worker]"
```

To keep the Worker command in its own UV-managed environment, install the
component directly:

```bash
uv tool install "ershu-worker==1.0.0.post20260812"
```

For development from a checkout:

```bash
uv sync --extra dev --locked
```

The locked development environment must be able to resolve the `ershu-core` version recorded in `uv.lock`.

## Configure and start

Copy `.env.example` to `.env`, then replace its development values. The example file is the reference for environment settings used by the Worker process.

The main groups are:

| Area | Settings | Purpose |
| --- | --- | --- |
| Data | `DATABASE_URL`, `MILVUS_URI`, `MILVUS_TOKEN`, `MILVUS_DB_NAME` | Queue, run data, documents, graphs, and search |
| Secrets | `ENCRYPTION_KEY` | Decrypt credentials stored by Ershu |
| Queue | `RUN_QUEUE_POLL_INTERVAL_MS`, `RUN_QUEUE_LEASE_TIMEOUT_SECONDS`, `ANALYSIS_RUN_RETENTION_COUNT` | Polling, recovery, and retained run history |
| Analysis | `MAX_PARALLEL_REPOSITORY_RUNS`, `DOCUMENT_WORKFLOW_ENGINE`, relationship scan limits | Repository concurrency and workflow limits |
| Repository access | `REPOSITORY_STORAGE_PATH`, trusted local paths, Git host and URL policy, Git provider tokens | Read-only workspace preparation |
| Search | `SEARCH_INDEX_BACKGROUND_ENABLED`, `SEARCH_INDEX_BATCH_SIZE` | Search refresh behavior |
| Questions | `ENABLE_ASK_USER` and the `ASK_USER_*` limits | Optional pauses for user input |
| LSP | `REPOSITORY_LSP_*`, `LSP_IMAGES`, and the `LSP_*` resource limits | Optional language-server analysis |
| Dependency preparation | `REPOSITORY_DEPENDENCY_*` | Optional, policy-controlled dependency downloads for LSP |

At minimum, use a current database schema, a reachable database and Milvus instance, a strong `ENCRYPTION_KEY`, and a writable persistent `REPOSITORY_STORAGE_PATH`. Configure repository credentials and model channels through the API or frontend before queuing a run. Do not commit `.env`.

Start from the source checkout with:

```bash
uv run ershu-worker
```

If installed with `uv tool install`, run:

```bash
ershu-worker
```

The process validates its settings and analysis registries before it claims a job. It also initializes the structured parsers under `REPOSITORY_STORAGE_PATH` and runs a parser self-check. A healthy startup includes these log messages:

```text
analysis worker started
structured parser self-check passed for configured languages
```

The Worker does not listen on a port. Monitor the process, startup self-check, queue claims, run heartbeats, warnings, and final run states instead of probing an HTTP health route.

## LSP images

LSP is off by default. This repository contains six runtime images because their commands and safety policy are coupled to the Worker:

| Image source | Published image | Languages | Server |
| --- | --- | --- | --- |
| `docker/lsp/python` | `ghcr.io/ershu-lab/ershu-lsp-python` | Python | basedpyright |
| `docker/lsp/typescript` | `ghcr.io/ershu-lab/ershu-lsp-typescript` | TypeScript, TSX, JavaScript | typescript-language-server |
| `docker/lsp/go` | `ghcr.io/ershu-lab/ershu-lsp-go` | Go | gopls |
| `docker/lsp/rust` | `ghcr.io/ershu-lab/ershu-lsp-rust` | Rust | rust-analyzer |
| `docker/lsp/clangd` | `ghcr.io/ershu-lab/ershu-lsp-clangd` | C and C++ | clangd |
| `docker/lsp/java` | `ghcr.io/ershu-lab/ershu-lsp-java` | Java | Eclipse JDT LS |

Build the default local tags from the repository root:

```bash
docker build -t ershu-lsp-python:basedpyright docker/lsp/python
docker build -t ershu-lsp-typescript:typescript-language-server docker/lsp/typescript
docker build -t ershu-lsp-go:gopls docker/lsp/go
docker build -t ershu-lsp-rust:rust-analyzer docker/lsp/rust
docker build -t ershu-lsp-clangd:clangd docker/lsp/clangd
docker build -t ershu-lsp-java:jdtls docker/lsp/java
```

Enable only the languages you need:

```dotenv
REPOSITORY_LSP_ENABLED=true
REPOSITORY_LSP_LANGUAGES=["python","typescript"]
REPOSITORY_LSP_DEFAULT_MODE=strict
REPOSITORY_LSP_ALLOW_SEMANTIC_MODE=false
```

`LSP_IMAGES` maps each adapter to the image tag used by the deployment. Image Dockerfiles and image release jobs belong with this component; the deployment repository should pin published image tags and supply the mapping.

For `v1.0.0-20260812`, the release workflow publishes versioned GHCR images
such as `ghcr.io/ershu-lab/ershu-lsp-python:v1.0.0-20260812`. The deployment
repository should use the exact tag instead of `latest`. Published images
currently target `linux/amd64`.

### Managed dependency preparation

LSP stays offline. If a language server needs installed dependency metadata, the Worker can run a separate short-lived download container before starting it:

```dotenv
REPOSITORY_DEPENDENCY_RESOLUTION_MODE=managed
REPOSITORY_DEPENDENCY_SOURCE_POLICY=configured_or_public
```

Managed preparation currently recognizes:

| Ecosystem | Required input |
| --- | --- |
| Go | `go.mod` |
| npm | `package-lock.json` |
| Python | `uv.lock` or `requirements.txt` |
| Rust | `Cargo.lock` with `Cargo.toml` |

pnpm and Yarn lock files are not prepared. Java dependency preparation is disabled because Maven and Gradle project loading can execute repository-controlled extensions or build scripts. C and C++ have no managed package source in this Worker.

The source policy accepts configured sources, public sources, or configured sources with public fallback. HTTP package sources remain blocked unless the deployment explicitly permits them. A host-specific CA bundle and download cache can be supplied through the matching settings in `.env.example`.

## Shutdown and recovery

On `SIGTERM` or an interactive stop, the Worker stops claiming jobs, drains background search work, shuts down LSP sessions, and returns its active runs to the queue. If the process exits before that handoff completes, another Worker can recover the run after its queue lease expires.

Transient database and infrastructure failures are retried. Programming and configuration errors stop the process so an operator can see the failure instead of leaving a broken Worker in the pool.

## Develop and test

The main source areas are:

| Path | Responsibility |
| --- | --- |
| `src/ershu/workers/analysis.py` | Process entry point, polling, startup checks, recovery, and shutdown |
| `src/ershu/workers/runtime_settings.py` | Worker settings and runtime overrides |
| `src/ershu/workers/readonly_run/` | Queue job lifecycle and repository run stages |
| `src/ershu/workers/tools/` | Read-only repository and symbol tools |
| `src/ershu/workers/services/` | Indexing, fact extraction, graph, prompt, and document services |
| `src/ershu/workers/orchestrator/` | Agent, tool, project graph, and document workflows |
| `src/ershu/workers/lsp/` | LSP sessions, Docker runtime, and dependency preparation |
| `docker/lsp/` | Language-server image sources |
| `tests/` | Unit, contract, workflow, queue, graph, document, and LSP tests |

Run the repository checks with uv:

```bash
uv run pytest -q
uv run ruff check src tests scripts
uv run ruff format --check src tests scripts
uv build --no-sources --out-dir dist
uv run --no-project scripts/check_package_contents.py dist
```

Tests marked `integration` need the external service named by the test, such as PostgreSQL, Milvus, or Docker. The ordinary test suite must not require API or MCP source code.

## License

Ershu Worker is licensed under the [Apache License 2.0](LICENSE).
