Metadata-Version: 2.4
Name: pumpwood-deploy-crawler-escavador
Version: 0.0.1
Summary: Package to assist deploy Pumpwood Crawler Escavador on K8s
License: BSD-3-Clause License
License-File: LICENSE
Author: André Andrade Baceti
Author-email: a.baceti@murabei.com
Requires-Python: >=3.6
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Dist: pumpwood-deploy
Project-URL: Homepage, https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-crawler-escavador
Description-Content-Type: text/markdown

# pumpwood-deploy-crawler-escavador

Satellite deploy package for the **Pumpwood Crawler Escavador**
microservice on Kubernetes. It generates manifests for the API
application, the legal proceeding worker, and crawler secrets — then
hands them to
[`pumpwood-deploy`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy)
for apply.

Developed by [Murabei Data Science](https://murabei.com). BSD-3-Clause.

<p align="center" width="60%">
  <img src="static_doc/sitelogo-horizontal.png" /> <br>

  <a href="https://en.wikipedia.org/wiki/Cecropia">
    Pumpwood is a native Brazilian tree
  </a> with a symbiotic relation to ants (Murabei)
</p>

---

## Objective and motivation

This package deploys the Crawler Escavador app and its legal
proceeding RabbitMQ worker onto a Pumpwood Kubernetes cluster.

### Why this exists
Queue CRUD lives in the API image; Escavador fetch and ETL live in
the worker image. The cluster needs one app Service and one worker
Deployment that consume
`CrawlerEscavadorLegalProceedingQueue__on_queue`.

### How it is used
A cluster deploy script imports
`PumpWoodCrawlerEscavadorMicroservice` and calls
`DeployPumpWood.add_microservice`. PgBouncer is added in that same
script via `PGBouncerDatabase` from `pumpwood-deploy`.

### Scope
**In scope:** Secret, app Deployment + Service, legal proceeding
worker.

**Out of scope:** Postgres, PgBouncer, Kong, Auth, Datalake,
Complex Datalake, Graph Datalake, Models LLM, and the database
itself.

---

## What it deploys

| Manifest | Kubernetes resources |
|----------|----------------------|
| `pumpwood_crawler_escavador__secrets` | Secret `pumpwood-crawler-escavador` |
| `pumpwood_crawler_escavador__deploy` | Deployment + Service `pumpwood-crawler-escavador-app` |
| `pumpwood_crawler_escavador__worker` | Deployment `pumpwood-crawler-escavador-worker` |

```mermaid
flowchart LR
    subgraph pkg [pumpwood-deploy-crawler-escavador]
        A[PumpWoodCrawlerEscavadorMicroservice]
    end
    subgraph core [pumpwood-deploy]
        B[DeployPumpWood]
    end
    subgraph cluster [Cluster]
        S[crawler Secret]
        APP[crawler-escavador-app]
        W[legal proceeding worker]
        RMQ[rabbitmq-main]
    end
    A --> B
    B --> S
    B --> APP
    B --> W
    APP --> RMQ
    RMQ --> W
```

---

## Prerequisites

This package does **not** stand alone. Before crawler pods can start,
the cluster must already provide:

| Resource | Provided by |
|----------|-------------|
| `storage` ConfigMap | `StandardMicroservices` in `pumpwood-deploy` |
| `general-secrets` | `StandardMicroservices` |
| `rabbitmq-main-secrets` | `StandardMicroservices` |
| Storage keys (GCP / Azure / AWS) | `DeployPumpWood` storage config |
| Postgres for crawler | `PGBouncerDatabase` in the cluster script |
| Auth | [`pumpwood-deploy-auth`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-auth) |
| Datalake | [`pumpwood-deploy-datalake`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-datalake) |
| Complex Datalake | complex-datalake satellite |
| Graph Datalake | graph-datalake satellite |
| Models LLM | models-llm satellite (document extraction) |

Storage bucket name and type are read from the cluster `storage`
ConfigMap — they are **not** passed to
`PumpWoodCrawlerEscavadorMicroservice`.

The database named in `db_database` must already exist on the
external Postgres. `PGBouncerDatabase` only pools connections.

The worker queue name is fixed in the worker image as
`CrawlerEscavadorLegalProceedingQueue__on_queue`.

---

## Installation

```bash
pip install pumpwood-deploy-crawler-escavador
```

Requires `pumpwood-deploy`.

---

## Quick start

```python
import os
import simplejson as json
from dotenv import load_dotenv
from pumpwood_deploy.deploy import DeployPumpWood
from pumpwood_deploy.microservices.postgres.deploy import (
    PGBouncerDatabase)
from pumpwood_deploy_crawler_escavador import (
    PumpWoodCrawlerEscavadorMicroservice)

with open("secrets/production.json", "r") as file:
    secrets = json.loads(file.read())
load_dotenv()

deploy.add_microservice(
    PGBouncerDatabase(
        name="pgbouncer-pumpwood-crawler-escavador",
        postgres_database="pumpwood_crawler_escavador",
        postgres_secret="postgres-general",
        postgres_host=POSTGRES_HOST,
    ))

deploy.add_microservice(
    PumpWoodCrawlerEscavadorMicroservice(
        app_version=os.getenv("PUMPWOOD_ESCAVADOR_CRAWLER_APP"),
        worker_version=os.getenv(
            "PUMPWOOD_ESCAVADOR_CRAWLER_WORKER"),
        repository="my-registry.example.com",
        db_host="pgbouncer-pumpwood-crawler-escavador",
        db_database="pumpwood_crawler_escavador",
        db_password=secrets["postgres_password"],
        microservice_password=secrets[
            "microservice--crawler-escavador"],
        escavador_auth_token=secrets["escavador_auth_token"],
        openai_api_key=secrets["openai_api_key"],
        app_replicas=1,
        app_debug="FALSE",
        worker_replicas=1,
    ))

deploy.create_deploy_files()
deploy.deploy_microservices()
```

### Environment variables

```bash
PUMPWOOD_ESCAVADOR_CRAWLER_APP=0.0.3
PUMPWOOD_ESCAVADOR_CRAWLER_WORKER=0.0.10
```

If the rendered manifest matches what is already on the cluster,
`kubectl apply` produces no changes — safe for rolling image updates.

---

## Configuration reference

### Required

| Parameter | Description |
|-----------|-------------|
| `app_version` | Image tag for `pumpwood-crawler-escavador-app` |
| `worker_version` | Image tag for `pumpwood-crawler-escavador-worker` |

Workers need a non-empty `escavador_auth_token` to call Escavador.

### Database

| Parameter | Default | Description |
|-----------|---------|-------------|
| `db_host` | `postgres-pumpwood-crawler-escavador` | Postgres host (PgBouncer in prod) |
| `db_port` | `5432` | Postgres port |
| `db_database` | `pumpwood` | Database name |
| `db_username` | `pumpwood` | Database user |
| `db_password` | `pumpwood` | Database password |
| `microservice_password` | `microservice--crawler-escavador` | Service user password |
| `repository` | GCR default | Docker registry for app and worker |

### Application

| Parameter | Default | Description |
|-----------|---------|-------------|
| `app_replicas` | `1` | Number of app pods |
| `app_debug` | `FALSE` | Debug flag |
| `app_workers` | `10` | Granian workers (`GRANIAN_WORKERS`) |
| `app_timeout` | `300` | Request timeout (seconds) |
| `app_limits_memory` | `4Gi` | Memory limit |
| `app_limits_cpu` | `3000m` | CPU limit |
| `app_requests_memory` | `1Gi` | Memory request |
| `app_requests_cpu` | `500m` | CPU request |

### Worker

Queue: `CrawlerEscavadorLegalProceedingQueue__on_queue`
(hardcoded in the worker image).

| Parameter | Default | Description |
|-----------|---------|-------------|
| `worker_replicas` | `1` | Worker pod count |
| `worker_debug` | `FALSE` | Worker debug flag |
| `worker_query_limit` | `100000` | ETL batch size (`QUERY_LIMIT`) |
| `escavador_auth_token` | `""` | Escavador API token |
| `openai_api_key` | `""` | OpenAI key for MarkItDown |
| `worker_limits_memory` | `4Gi` | Worker memory limit |
| `worker_limits_cpu` | `2000m` | Worker CPU limit |
| `worker_requests_memory` | `20Mi` | Worker memory request |
| `worker_requests_cpu` | `1m` | Worker CPU request |

---

## Health check

The app Deployment exposes a readiness probe at:

```
GET /health-check/pumpwood-crawler-escavador-app/  (port 5000)
```

Use this path for ingress and load balancer health checks.

---

## Related packages

| Package | Role |
|---------|------|
| [`pumpwood-deploy`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy) | Orchestrator, Kong, RabbitMQ, Postgres |
| [`pumpwood-deploy-auth`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-auth) | Authorization microservice |
| [`pumpwood-deploy-datalake`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-datalake) | Datalake app and dataloader |
| [`pumpwood-deploy-crawler-bigdata`](https://github.com/Murabei-OpenSource-Codes/pumpwood-deploy-crawler-bigdata) | Sister crawler satellite |

---

## Development

```bash
pip install -e ../pumpwood-deploy
pip install -e .

ruff check src/
```

---

## License

BSD-3-Clause — see [LICENSE](LICENSE).

