Metadata-Version: 2.4
Name: governor-bq
Version: 0.4.29
Summary: BigQuery cost optimization platform — detect opportunities, generate solutions, create PRs or apply local fixes
Author: Simple Machines
License-Expression: MIT
Project-URL: Homepage, https://github.com/simple-machines/governor
Project-URL: Repository, https://github.com/simple-machines/governor
Project-URL: Issues, https://github.com/simple-machines/governor/issues
Keywords: bigquery,cost-optimization,dbt,data-engineering
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Office/Business :: Financial
Requires-Python: >=3.14
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.115.0
Requires-Dist: jinja2>=3.1.4
Requires-Dist: sqlalchemy>=2.0.36
Requires-Dist: alembic>=1.13.3
Requires-Dist: bcrypt>=5.0.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: opentelemetry-api>=1.27.0
Requires-Dist: opentelemetry-sdk>=1.27.0
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.48b0
Requires-Dist: opentelemetry-instrumentation-sqlalchemy>=0.48b0
Requires-Dist: datastar-py>=0.1.0
Requires-Dist: psycopg[binary]>=3.2.3
Requires-Dist: uvicorn>=0.30.6
Requires-Dist: pydantic>=2.10.0
Requires-Dist: google-cloud-resource-manager>=1.13.0
Requires-Dist: google-auth>=2.38.0
Requires-Dist: google-cloud-storage>=3.8.0
Requires-Dist: google-cloud-bigquery>=3.28.0
Requires-Dist: PyGithub>=2.1.0
Requires-Dist: google-genai>=1.0.0
Requires-Dist: google-adk>=0.6.0
Requires-Dist: tenacity>=8.3.0
Requires-Dist: sqlglot>=28.0.0
Requires-Dist: PyYAML>=6.0.0
Requires-Dist: APScheduler[sqlalchemy]<4.0,>=3.11.0
Requires-Dist: cron-descriptor<3.0,>=2.0.0
Requires-Dist: typer[all]>=0.12.0
Dynamic: license-file

# Governor

BigQuery cost optimization platform for dbt projects. Governor syncs your warehouse metadata, detects cost and performance issues, generates solutions (deterministic or AI-assisted), and then either opens pull requests or applies fixes locally depending on how you run it.

## Install

```sh
pip install governor-bq
```

Requires Python 3.14+ and Docker (for automatic PostgreSQL).

## Quick start

```sh
governor init
```

That's it. The wizard walks you through:

1. **Database** — starts PostgreSQL via Docker (or uses yours)
2. **Google Cloud** — verifies local credentials or reads a service account file
3. **Project** — auto-detects your local dbt project and opens the app
4. **Launch** — starts the server and opens `http://localhost:8000`

## What Governor detects

| Category | Issues |
|----------|--------|
| Performance | shuffle spill, slot contention, join explosion, partition pruning |
| Dead code | dead CTEs, dead columns, dead window expressions, unused aggregations |
| SQL cleanup | redundant ORDER BY, unused JOINs |
| Storage | billing model optimization |

## What Governor does about it

- **Deterministic fixes** (no LLM) — add partition/cluster, remove dead code, switch billing model
- **AI-assisted fixes** (Gemini) — complex rewrites with safety validation
- **Code changes** — opens PRs in cloud mode or applies changes to your local working tree in local CLI mode
- **Verified savings** — compares BigQuery costs before and after merge

## Day-to-day commands

```sh
governor start          # start server + database
governor stop           # stop server
governor stop --all     # stop server + database container
governor status         # check what's running
governor sync           # trigger a manual sync
governor auth refresh   # re-authenticate Google or optional GitHub fallback
```

## Main surfaces

| Page | What it shows |
|------|---------------|
| `/dashboard` | Global spend, opportunities, savings |
| `/opportunities` | Cross-project opportunity explorer |
| `/projects/{id}` | Project-level dashboard |
| `/projects/{id}/opportunities/{id}` | Opportunity detail, diff, verification SQL |
| `/admin/configurations` | Project config, rules, guardrails |
| `/prs` | Pull request tracking and cost verification for cloud / GitHub workflows |

## Install from source (contributors)

```sh
git clone https://github.com/simple-machines/governor.git
cd governor
uv pip install -e .
governor init
```

The `governor init` command handles everything interactively:

1. **Database** — starts a PostgreSQL container via Docker (or uses yours)
2. **Google Cloud** — verifies local credentials or reads a service account file
3. **Project** — auto-detects your first local dbt project
4. **Launch** — starts the server and opens `http://localhost:8000`

After the first run, use these to manage your instance:

```sh
governor start          # start server + database
governor stop           # stop server
governor stop --all     # stop server + database container
governor status         # check what's running
governor sync           # trigger a manual sync
governor auth refresh   # re-authenticate Google or optional GitHub fallback
```

See [SETUP.md](SETUP.md) for the full setup guide including Cloud Run deployment.

For a fast architecture view of how local and cloud execution differ, see [docs/execution-modes.md](docs/execution-modes.md).

## Local development (Devbox)

For contributors working on Governor itself, local development is managed with Devbox.

### Prerequisites

- Devbox: https://www.jetify.com/devbox
- Docker (for database, or bring your own PostgreSQL)

### Enter the environment

```sh
devbox shell
```

The Devbox shell runs `uv sync` automatically.

### Database initialization

Initialize PostgreSQL from your host shell:

```sh
devbox run -- initdb
devbox services start postgresql
devbox run -- createdb -h localhost -p 5432 governor
```

Then, inside `devbox shell`, run migrations:

```sh
make migrate
```

### Start the app

Build CSS and run the server:

```sh
make run
```

If you want the Tailwind watcher in a separate process:

```sh
bun run tailwind:watch
```

If you prefer to start background services together:

```sh
devbox services up
process-compose up
```

If you need to override the database connection:

```sh
DATABASE_URL="postgresql+psycopg://localhost:5432/governor" make run
```

## Common commands

```sh
make migrate
make run
make test
make test-serial
uv run ruff format
uv run ruff check
```

## Seed an admin user

```sh
uv run python scripts/seed_admin.py --email admin@example.com
```

You will be prompted for a password unless you pass `--password`.

## GitHub integration setup

Governor integrates with GitHub to create pull requests and manage repository content. This requires creating a GitHub App and configuring Governor with its credentials.

### Prerequisites

- GitHub organization administrator access
- Governor deployment with environment variable access

### Step 1: Create GitHub App

1. Navigate to your organization's GitHub App settings:
   `https://github.com/organizations/{your-org}/settings/apps`

2. Click **New GitHub App**

3. Fill in the required fields:
   - **GitHub App name**: `Governor - {YourOrgName}` (must be globally unique)
   - **Homepage URL**: Your Governor instance URL (e.g., `https://governor.example.com`)

4. Under **Webhook**:
   - **Uncheck** "Active" (webhooks are not required)

5. Under **Permissions** → **Repository permissions**:
   - **Contents**: Read and write
   - **Pull requests**: Read and write
   - **Metadata**: Read-only (auto-selected)

6. Under **Where can this GitHub App be installed?**:
   - Select **Only on this account**

7. Click **Create GitHub App**

### Step 2: Generate Private Key

1. After creation, you'll be on the app settings page
2. Note the **App ID** displayed near the top (e.g., `123456`)
3. Scroll down to **Private keys** section
4. Click **Generate a private key**
5. A `.pem` file will download automatically
6. Store this file securely - it cannot be retrieved again

### Step 3: Install App to Organization

1. In the left sidebar, click **Install App**
2. Click **Install** next to your organization
3. Choose repository access:
   - **All repositories** or
   - **Only select repositories** (choose repos Governor will manage)
4. Click **Install**
5. Note the **Installation ID** from the URL after installation:
   - URL format: `https://github.com/organizations/{org}/settings/installations/{installation_id}`

### Step 4: Configure Governor

Set the following environment variables in your Governor deployment:

**Shell export:**
```bash
# GitHub App ID from Step 2
export GITHUB_APP_ID="123456"

# Contents of the .pem file from Step 2
export GITHUB_APP_PRIVATE_KEY="$(cat /path/to/private-key.pem)"

# Installation ID from Step 3
export GITHUB_APP_INSTALLATION_ID="12345678"
```

**`.env` file format:**
```
GITHUB_APP_ID=123456
GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA...
...
-----END RSA PRIVATE KEY-----"
GITHUB_APP_INSTALLATION_ID=12345678
```

### Required Permissions Reference

| Permission | Level | Purpose |
|------------|-------|---------|
| Contents | Read & Write | Create branches, commit changes |
| Pull requests | Read & Write | Create and manage pull requests |
| Metadata | Read-only | Access basic repository information |

### Verifying Configuration

1. Log in to Governor as an admin
2. Navigate to **Admin** → **System Verification**
3. Click **Run Verification**
4. Confirm the **GitHub** check shows **Pass** status

### Troubleshooting

#### Check shows "Configuration incomplete"

**Cause**: One or more environment variables not set.

**Solution**: Verify all three variables are set:
```bash
echo $GITHUB_APP_ID
echo $GITHUB_APP_INSTALLATION_ID
# Don't echo the private key for security
test -n "$GITHUB_APP_PRIVATE_KEY" && echo "Private key is set"
```

#### Check shows "Private key is malformed"

**Cause**: Private key format is incorrect.

**Solution**:
- Ensure key starts with `-----BEGIN RSA PRIVATE KEY-----` or `-----BEGIN PRIVATE KEY-----`
- Check for proper newlines (not literal `\n` strings in shell)
- Verify no extra whitespace or truncation

#### Check shows "Authentication failed"

**Cause**: App ID doesn't match private key.

**Solution**:
- Verify GITHUB_APP_ID matches the app settings page
- Generate a new private key if needed
- Ensure you're using the correct app's key

#### Check shows "App is suspended"

**Cause**: GitHub App has been suspended.

**Solution**:
1. Go to GitHub organization settings → GitHub Apps
2. Find your app and check its status
3. Go to Advanced and click "Unsuspend" if available

#### Check shows "Rate limit exceeded"

**Cause**: Too many API requests.

**Solution**:
- Wait for the rate limit to reset (typically 1 hour)
- Check for other processes using the same App ID

#### Check shows "Connection timeout"

**Cause**: Network connectivity issues.

**Solution**:
- Verify Governor can reach `api.github.com` port 443
- Check firewall rules and proxy settings
- Test with: `curl -I https://api.github.com`
