Metadata-Version: 2.4
Name: fleet-framework
Version: 0.2.0
Summary: generic distributed-automation framework — master/worker, browser pool, anti-bot helpers, and abstract automation contracts (SERP, content, news, place, marketplace, jobs, social)
Author: Sarper Avci
License: MIT
Project-URL: Homepage, https://github.com/sarperavci/fleet
Project-URL: Repository, https://github.com/sarperavci/fleet
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.115.0
Requires-Dist: uvicorn[standard]>=0.30.0
Requires-Dist: pydantic>=2.7.0
Requires-Dist: websockets>=12.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: redis>=5.0.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: psutil>=5.9.0
Requires-Dist: click>=8.1.0
Provides-Extra: browser
Requires-Dist: DrissionPage>=4.1.0; extra == "browser"
Requires-Dist: cryptography>=42.0.0; extra == "browser"
Requires-Dist: mitmproxy>=12.0; extra == "browser"
Provides-Extra: cloudflare
Requires-Dist: fleet-framework[browser]; extra == "cloudflare"
Provides-Extra: cloak
Requires-Dist: fleet-framework[browser]; extra == "cloak"
Requires-Dist: cloakbrowser>=0.3; extra == "cloak"
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.27.0; extra == "otel"
Requires-Dist: opentelemetry-sdk>=1.27.0; extra == "otel"
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27.0; extra == "otel"
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23; extra == "test"
Provides-Extra: all
Requires-Dist: fleet-framework[browser,cloak,cloudflare,otel]; extra == "all"
Dynamic: license-file

# Fleet

A framework for distributed automation fleets. One master, many workers, typed output streams between them. Born out of a production CAPTCHA-solving farm, generalized to fit any "N machines each doing M parallel things" workload.

## What it is

Two pip packages. `fleet-core` is the framework: master, worker shell, reconcile loop, Redis-backed store, output streams, event bus, a small dashboard. `fleet-browser` is an optional add-on that ships a battle-tested Chromium pool with fingerprint rotation, proxy auth, and orphan-process cleanup.

You write a Python class subclassing `ContinuousAutomation` or `BatchAutomation` and a Pydantic config. Ship it as a pip package with one entry-point line. Install it on the master and on each worker host and the framework wires up everything else: config push, validation, slot lifecycle, recycle, heal, stats, dashboard, stream-based inter-automation comms.

## What's already in the box

- Master process: REST + WebSocket API, auto-discovered plugins, dashboard, Redis-backed state.
- Worker process: pulls config, runs N concurrent slots, reconciles to desired state, heals on failure.
- Reconcile loop: pure-functional diff, smallest-converging-plan semantics, slot recycle, self-heal.
- Typed output streams: ZSET-backed, bounded by length and TTL, one per automation type, pop or peek over REST or in-process.
- Event bus: Redis pubsub for lightweight signals.
- Auth: bearer tokens, admin vs worker, constant-time comparison. WebSocket handshake validated before accept.
- Hardening from the original farm: tini in the worker container reaps zombies, `psutil`-based process-tree cleanup catches orphans, recycle's heal step closes the "zero-slot lockout" failure mode.

## What you write

Roughly 30 lines per automation. Here's a fully working example:

```python
import asyncio
import httpx
from fleet.core import BaseConfig, ContinuousAutomation, register

class PingerConfig(BaseConfig):
    url: str
    interval_seconds: float = 5.0

@register("pinger")
class Pinger(ContinuousAutomation[PingerConfig]):
    Config = PingerConfig

    async def run_slot(self, ctx):
        async with httpx.AsyncClient(proxy=ctx.proxy) as client:
            while not ctx.shutdown.is_set():
                r = await client.get(ctx.config.url)
                await ctx.emit({"status": r.status_code})
                await asyncio.sleep(ctx.config.interval_seconds)
```

Plus one line in your `pyproject.toml`:

```toml
[project.entry-points."fleet.automations"]
pinger = "my_package:Pinger"
```

## Quickstart

```bash
git clone <repo-url> fleet && cd fleet
python3 -m venv .venv && source .venv/bin/activate
pip install -e packages/fleet-core
pip install -e examples/hello-world

# in one terminal:
redis-server --port 6379 --appendonly yes
# in another:
FLEET_ADMIN_TOKEN=admin FLEET_WORKER_TOKEN=worker fleet master
# in another:
MASTER_URL=http://localhost:8000 FLEET_WORKER_TOKEN=worker fleet worker --type hello-world --id w1
# in another:
curl -X PATCH http://localhost:8000/api/v1/automations/hello-world/workers/w1/config \
  -H "Authorization: Bearer admin" -H "Content-Type: application/json" \
  -d '{"enabled": true, "slots": 2}'
curl -X POST http://localhost:8000/api/v1/automations/hello-world/output/pop \
  -H "Authorization: Bearer admin" -H "Content-Type: application/json" \
  -d '{"n": 5}'
```

Walkthrough with explanations: [docs/getting-started/quickstart.md](docs/getting-started/quickstart.md).

## Repo layout

```
fleet/
├── packages/
│   ├── fleet-core/      master, worker, reconcile, store, streams, event bus, dashboard, CLI
│   └── fleet-browser/   optional Chromium pool, fingerprint, humanizer, proxy auth, cert
├── examples/
│   ├── hello-world/     minimal ContinuousAutomation (emit greetings on a timer)
│   └── echo-consumer/   consumes another automation's output stream
└── docs/                GitBook-format documentation (also lives on the `docs` branch)
```

## Documentation

Full docs in [`docs/`](docs/), rendered as a GitBook from the `docs` branch.

- [Introduction](docs/README.md)
- [Installation](docs/getting-started/installation.md)
- [Quickstart](docs/getting-started/quickstart.md)
- [Your first automation](docs/getting-started/first-automation.md)
- [Architecture](docs/concepts/architecture.md)
- [Writing a continuous automation](docs/guides/continuous-automation.md)
- [Writing a batch automation](docs/guides/batch-automation.md)
- [Making two automations talk](docs/guides/inter-automation.md)
- [Browser-based automations](docs/guides/browser-automations.md)
- [Deployment](docs/guides/deployment.md)
- [REST API](docs/reference/rest-api.md)
- [WebSocket protocol](docs/reference/ws-protocol.md)
- [Authentication](docs/operations/auth.md)
- [Troubleshooting](docs/operations/troubleshooting.md)

## Origins

Fleet started as the control plane for a Cloudflare Turnstile-solving farm. That farm hit every interesting failure mode in distributed automation: zombie subprocess accumulation, master state wipe on Redis restart, gen-counter regression, slot-recycle drops with no heal, and config drift across heterogeneous workers. Each of those has a fix baked into the framework.

The framework is what the original farm should have been from day one. It's deliberately small. If you want a scheduler, an orchestrator, or a service mesh, this isn't it. If you want a thin layer that turns "N hosts doing repetitive work" into a manageable system, it is.

## Status

Pre-alpha. The public API will change. No PyPI release yet — install from source. Production deployments are running on it but with a lot of manual operational knowledge to back them up. Watch the changelog before relying on anything sensitive.

See [ROADMAP.md](ROADMAP.md) for the path to v0.1.

## License

MIT. See [LICENSE](LICENSE).
