Metadata-Version: 2.4
Name: pkgdb
Version: 0.2.0
Summary: Track, store and query your PyPI package download statistics
Keywords: pypi,statistics,downloads,analytics,cli
Author: Shakeeb Alireza
Author-email: Shakeeb Alireza <shakfu@users.noreply.github.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Software Distribution
Requires-Dist: pypistats>=1.12.0
Requires-Dist: tabulate>=0.9.0
Requires-Python: >=3.10
Project-URL: Homepage, https://github.com/shakfu/pkgdb
Project-URL: Repository, https://github.com/shakfu/pkgdb
Project-URL: Issues, https://github.com/shakfu/pkgdb/issues
Description-Content-Type: text/markdown

# pkgdb

A tool designed to monitor, collect, and analyze download statistics for PyPI packages, along with GitHub metrics for Python projects.

It retrieves data through the pypistats and GitHub APIs, storing it as historical records in a SQLite database. On the first fetch it backfills roughly 180 days of true daily download data per package, so trends are useful immediately rather than only after weeks of collection. The data can then be analyzed and presented as static HTML reports with visual charts or through an interactive local web interface, compared across time periods, grouped with tags for portfolio rollups, and monitored for download anomalies and milestones.

[Documentation](https://shakfu.github.io/pkgdb/)

## Features

- **True daily download time series** - the first fetch backfills roughly 180 days of real per-day downloads (overall, by Python version, and by OS), so trends are useful immediately. The local store keeps history past the ~180-day pypistats window as it accumulates.
- **Self-owned historical database** - all data lives in a single SQLite file you control: offline, scriptable, no rate-limited web UI.
- **Terminal views** - `show` (trend sparklines and growth %), `diff` (previous fetch / exact week-over-week / month-over-month), `stats` (Python version and OS breakdown).
- **HTML reports** - download charts with a release-timeline overlay (PyPI and GitHub), environment distribution, and per-package project views.
- **Interactive web dashboard** (`serve`) - live from SQLite, with zoomable download and GitHub-stars charts, release markers, and multi-package comparison. No Flask/FastAPI dependency.
- **Anomaly and milestone alerts** (`check`) - weekly spike/drop detection and download-milestone crossings, exiting non-zero for shell/CI notifiers.
- **Package tags / portfolio rollups** - group related packages and view aggregate download stats per group.
- **GitHub integration** - repository stats (stars, forks, activity, language), a daily metrics snapshot with star-growth tracking, and release history.
- **Export and badges** - CSV / JSON / Markdown export, shields.io-style SVG badges, and machine-readable `--json` on most commands.
- **Configuration file** - `~/.pkgdb/config.toml` for persistent defaults, including anomaly thresholds and milestones.

## Installation

```sh
pip install pkgdb
```

To build:

Requires Python 3.10+. Uses [uv](https://github.com/astral-sh/uv) for dependency management.

```bash
uv sync
```

## Usage

### Quick start

The fastest way to get started is `pkgdb init`, which walks you through setup:

```bash
pkgdb init
```

This prompts for your PyPI username (optional), syncs your packages, fetches current stats, and generates an HTML report -- all in one step. For non-interactive use:

```bash
pkgdb init --user <pypi-username> --no-browser
```

### Configure packages

Create `~/.pkgdb/packages.json` to list packages to track:

```json
["my-package", "another-package"]
```

Or use an object with a `published` key:

```json
{"published": ["my-package", "another-package"]}
```

Alternatively, use `pkgdb add <package>` to add packages individually. By default, packages are verified to exist on PyPI before adding. Use `--no-verify` to skip this check.

### Configuration file

Create `~/.pkgdb/config.toml` to set persistent defaults:

```toml
[defaults]
github = true           # always include GitHub stats
environment = true      # always include environment summary
no_browser = false      # don't auto-open reports
sort_by = "total"       # default sort order (total, month, week, day, growth, name)
# database = "~/.pkgdb/pkg.db"  # custom database path

[report]
# output = "~/.pkgdb/report.html"  # custom report path

[init]
# pypi_user = "myusername"  # default PyPI username for init command

[check]
# milestones = [1000, 10000, 100000, 1000000]  # download targets to watch
# baseline_weeks = 8    # weeks of history used as the anomaly baseline
# z_threshold = 2.5     # std-devs from baseline to flag a spike/drop
# min_weekly = 10       # ignore packages averaging fewer weekly downloads
```

CLI flags always override config values. The config file is optional -- all settings have sensible defaults.

### Commands

```bash
# Guided first-run setup (sync packages, fetch stats, generate report)
pkgdb init

# Non-interactive init with PyPI username
pkgdb init --user <pypi-username>

# Add a package to tracking (verifies it exists on PyPI)
pkgdb add <package-name>

# Add without verification (offline/bulk use)
pkgdb add <package-name> --no-verify

# Remove a package from tracking
pkgdb remove <package-name>

# Show tracked packages (with their tags)
pkgdb packages

# Group packages with tags, then view portfolio rollups
pkgdb tag <package-name> web api    # add one or more tags
pkgdb untag <package-name> api      # remove a tag (or --all)
pkgdb tags                          # per-tag aggregate downloads
pkgdb show --tag web                # filter show to a group, with a group total

# Import packages from a file (JSON or plain text)
pkgdb import packages.json

# Fetch latest stats from PyPI and store in database
# (skips packages already fetched in the last 24 hours)
pkgdb fetch

# Display stats in terminal (includes trend sparklines and growth %)
pkgdb show

# Show package history (HTML report with the daily download chart + releases, opens in browser)
pkgdb history <package-name>

# Show the daily download series as a text table in terminal
pkgdb history <package-name> --text

# Filter history by date (works with both HTML and text output)
pkgdb history <package-name> --since 7d   # last 7 days
pkgdb history <package-name> --since 2w   # last 2 weeks
pkgdb history <package-name> --since 1m   # last month (30 days)
pkgdb history <package-name> --since 2024-01-01

# Compare stats between time periods
pkgdb diff                   # compare to previous fetch
pkgdb diff --period week     # this week vs last week (exact, from daily data)
pkgdb diff --period month    # this month vs last month

# Detect download anomalies (weekly spikes/drops) and milestone crossings
pkgdb check                       # exits non-zero if any event is found
pkgdb check --json                # machine-readable events
pkgdb check --milestone 100000    # watch a download target (repeatable)
pkgdb check --exit-zero           # report but always exit 0

# Show release history for a package (PyPI and GitHub)
pkgdb releases <package-name>

# Show only the most recent 10 releases
pkgdb releases <package-name> --limit 10

# Generate HTML report with charts (opens in browser)
pkgdb report

# Generate detailed HTML report for a single package
pkgdb report <package-name>

# Generate project view with release timeline overlay
pkgdb report <package-name> --project

# Include environment summary (Python versions, OS) in report
pkgdb report -e

# Export stats in various formats
pkgdb export -f csv      # CSV format (default)
pkgdb export -f json     # JSON format
pkgdb export -f markdown # Markdown table

# Show detailed stats for a package (Python versions, OS breakdown)
pkgdb stats <package-name>

# Show database info (size, record counts, date range)
pkgdb show --info

# Generate SVG badge for a package
pkgdb badge <package-name>

# Badge for monthly downloads
pkgdb badge <package-name> --period month

# Save badge to file
pkgdb badge <package-name> -o badge.svg

# Fetch GitHub repository stats (stars, forks, activity, language)
# Records a daily snapshot; shows star growth (Stars Δ) once history accumulates
pkgdb github

# Sort GitHub stats by name or activity instead of stars
pkgdb github fetch --sort name

# Bypass GitHub cache and fetch fresh data
pkgdb github fetch --no-cache

# Show GitHub cache statistics
pkgdb github cache

# Clear expired GitHub cache entries (or --all for everything)
pkgdb github clear

# Launch interactive web dashboard (opens browser)
pkgdb serve

# Serve on a custom port without opening browser
pkgdb serve --port 3000 --no-browser

# Fetch stats and generate report in one step
# (skips packages already fetched in the last 24 hours)
pkgdb update

# Fetch stats and generate report with environment summary
pkgdb update -e

# Include GitHub stats in fetch, report, or update
pkgdb fetch --github
pkgdb report --github
pkgdb update --github

# Clean up orphaned stats (for packages no longer tracked)
pkgdb cleanup

# Prune stats older than N days
pkgdb cleanup --days 365

# Sync packages from PyPI user account (initial or refresh)
pkgdb sync --user <pypi-username>

# Sync and remove packages no longer in user's PyPI account
pkgdb sync --user <pypi-username> --prune

# Show version
pkgdb version
```

### Options

```bash
# Use custom database file
pkgdb -d custom.db fetch

# Verbose output (show debug messages)
pkgdb -v fetch

# Quiet mode (only show warnings/errors)
pkgdb -q fetch

# Specify output file for report
pkgdb report -o custom-report.html

# Generate report without opening browser (useful for automation)
pkgdb report --no-browser

# Limit history to N days
pkgdb history my-package -n 14

# History report to file without opening browser
pkgdb history my-package -o history.html --no-browser

# Show history since a specific date (or relative: 7d, 2w, 1m)
pkgdb history my-package --since 2024-01-01
pkgdb history my-package --since 7d

# Skip package verification on import
pkgdb import packages.json --no-verify

# Limit show output to top N packages
pkgdb show --limit 10

# Sort show output by field (total, month, week, day, growth, name)
pkgdb show --sort-by month

# JSON output (available on show, packages, history, stats, cleanup, github)
pkgdb show --json
pkgdb packages --json
pkgdb history my-package --json
pkgdb stats my-package --json
pkgdb cleanup --json
pkgdb github --json

# Export to file instead of stdout
pkgdb export -f json -o stats.json
```

## Architecture

Modular CLI application with the following commands:

**Setup:**
- **init**: Guided first-run setup (sync packages, fetch stats, generate report)

**Package management:**
- **add**: Add a package to tracking
- **remove**: Remove a package from tracking
- **packages**: Show tracked packages with their added dates and tags
- **import**: Import packages from file (JSON or text)
- **sync**: Sync packages from a PyPI user account (with optional `--prune`)
- **tag** / **untag** / **tags**: Group packages with tags and view per-group (portfolio) rollups

**Data operations:**
- **fetch**: Fetch download stats from PyPI and store in SQLite (with `-g` for GitHub stats). Also backfills roughly 180 days of true daily downloads per package on the first fetch
- **show**: Display stats in terminal with trend sparklines and growth %; `--tag` filters to a group with an aggregate total
- **diff**: Compare download stats between time periods (previous fetch, week-over-week, month-over-month). Week/month comparisons use exact daily sums
- **check**: Detect download anomalies (weekly spikes/drops) and milestone crossings; exits non-zero on any event for CI/notifier use
- **history**: Show package history as HTML report (default) or text table (`--text`), using the true daily download series
- **stats**: Show detailed breakdown (Python versions, OS) for a package
- **releases**: Show release history for a package (PyPI and GitHub)
- **github**: Fetch and display GitHub repository stats (stars, forks, activity, language). Records a daily snapshot and shows star growth
- **export**: Export stats in CSV, JSON, or Markdown format

**Reporting:**
- **report**: Generate HTML report with SVG charts. With `-e` flag, includes Python/OS summary. With `-g` flag, includes GitHub stats (stars, forks, language, activity) in the table. With package argument, generates detailed single-package report. With `-p/--project` flag, generates project view with release timeline overlay
- **badge**: Generate shields.io-style SVG badge for a package
- **update**: Run fetch then report in one step (supports `-e` for environment summary, `-g` for GitHub stats)
- **serve**: Launch interactive web dashboard with live data from SQLite. Overview with sortable/filterable stats table, package detail with zoomable download and GitHub-stars charts plus release markers, comparison with multi-package overlay

**Maintenance:**
- **cleanup**: Remove orphaned stats and optionally prune old data
- **version**: Show pkgdb version

### Data flow

```
packages.json -> pypistats + GitHub APIs -> SQLite (pkg.db) -> terminal / HTML reports / web dashboard / anomaly checks
```

### Database schema

The `package_stats` table stores:
- `package_name`: Package identifier
- `fetch_date`: Date stats were fetched (YYYY-MM-DD)
- `last_day`, `last_week`, `last_month`: Recent download counts
- `total`: Total downloads (excluding mirrors)

The `daily_downloads` table stores the true per-day download time series:
- `package_name`: Package identifier
- `date`: Download date (YYYY-MM-DD)
- `dimension`: `overall` (mirror split), `python` (version), or `os`
- `category`: e.g. "without_mirrors", "3.12", or "Linux"
- `downloads`: Downloads on that date for that category

This series is backfilled (~180 days) from pypistats on the first fetch and drives the history/report charts, the exact `diff` period math, and growth %. Unlike the snapshot table it survives past the ~180-day pypistats window as it accumulates locally.

The `python_version_stats` and `os_stats` tables cache the latest environment breakdown:
- `package_name`: Package identifier
- `fetch_date`: Date stats were fetched (YYYY-MM-DD)
- `category`: Python version (e.g. "3.12") or OS name (e.g. "Linux")
- `downloads`: Download count for that category

The `fetch_attempts` table tracks API requests:
- `package_name`: Package identifier (primary key)
- `attempt_time`: ISO timestamp of last fetch attempt
- `success`: Whether the fetch succeeded (1) or failed (0)

The `github_cache` table caches GitHub API responses:
- `repo_key`: Lowercased `owner/repo` identifier (primary key)
- `data`: Full JSON response from the GitHub API
- `fetched_at`: When the response was cached
- `expires_at`: Cache expiry time (default: 24 hours)

The `pypi_releases` table caches PyPI release history:
- `package_name`: Package identifier
- `version`: Release version string
- `upload_date`: Date the version was uploaded (YYYY-MM-DD)

The `github_releases` table caches GitHub release history:
- `repo_key`: Lowercased `owner/repo` identifier
- `tag_name`: Release tag (e.g. "v0.1.0")
- `published_at`: Date the release was published (YYYY-MM-DD)

The `release_cache` table tracks freshness of release data:
- `cache_key`: Cache identifier (e.g. "pypi:my-package" or "github:owner/repo")
- `fetched_at`: When the data was last fetched
- `expires_at`: Cache expiry time (default: 24 hours)

The `github_stats_history` table stores a daily snapshot of GitHub metrics:
- `package_name`: Package identifier
- `repo_key`: Lowercased `owner/repo` identifier
- `date`: Snapshot date (YYYY-MM-DD)
- `stars`, `forks`, `open_issues`, `watchers`: Repository counts on that date

The `package_tags` table groups packages:
- `package_name`: Package identifier
- `tag`: Normalized (lowercased) tag name

Stats are upserted per package per day. Fetch attempts are tracked to avoid hitting PyPI rate limits - packages are only fetched once per 24-hour period. The daily download series is backfilled (~180 days) on first fetch and refreshed idempotently on each run. Environment stats are cached alongside download stats, so reports can be generated offline. GitHub API responses are cached for 24 hours to minimize API calls, while a daily GitHub metrics snapshot accumulates for star/fork history. Release data (PyPI and GitHub) is cached for 24 hours.

## Files

Source modules in `src/pkgdb/`:
- `__init__.py`: Public API and version
- `cli.py`: CLI argument parsing and commands
- `config.py`: Configuration file loading (`~/.pkgdb/config.toml`)
- `service.py`: High-level service layer
- `db.py`: Database operations and context manager
- `api.py`: pypistats API wrapper with parallel fetching (incl. daily series)
- `checks.py`: anomaly and milestone detection for `pkgdb check`
- `reports.py`: HTML/SVG report generation
- `server.py`: HTTP server for the interactive web dashboard
- `dashboard.py`: HTML page templates for the dashboard (overview, detail, comparison)
- `github.py`: GitHub API client with caching and rate limit handling
- `badges.py`: SVG badge generation
- `export.py`: CSV/JSON/Markdown export
- `utils.py`: Helper functions and validation
- `types.py`: TypedDict definitions for type safety
- `logging.py`: Logging configuration

Data files (all in `~/.pkgdb/`):
- `config.toml`: Configuration file for persistent defaults (optional)
- `packages.json`: Package list configuration (optional, can use `add` command instead)
- `pkg.db`: SQLite database (auto-created)
- `report.html`: Generated HTML report (default output)

## GitHub Actions

An example workflow is provided at `.github/workflows/fetch-stats.yml.example` for automated daily stats fetching. To use it:

1. Copy to `.github/workflows/fetch-stats.yml` (remove `.example`)
2. Configure your package list or PyPI username
3. The workflow will fetch stats daily and commit updates to your repo

The example also runs `pkgdb check` each day so download spikes, drops, and milestone crossings show up automatically. Because `pkgdb fetch` backfills roughly 180 days of daily history on the first run, anomaly detection is useful immediately rather than only after weeks of collection. Drop the step's `|| true` to turn any detected event into a failed run (a red X and GitHub's usual failure notification), or pipe the JSON output to Slack, email, or an issue.

## Documentation

API documentation is built with MkDocs:

```bash
# Build docs
make docs

# Serve locally with live reload
make docs-serve

# Deploy to GitHub Pages
make docs-deploy
```

Then open `http://127.0.0.1:8000` to browse the docs locally.

## Development

```bash
# Install dev dependencies
uv sync

# Run tests
pytest

# Run tests with verbose output
pytest -v

# Full QA (test + lint + typecheck + format)
make qa
```

## Dependencies

Runtime:
- `pypistats`: PyPI download statistics API client
- `tabulate`: Terminal table formatting

Development:
- `pytest`: Testing framework

Documentation:
- `mkdocs`: Static site generator
- `mkdocs-material`: Material theme
- `mkdocstrings[python]`: Auto-generated API docs from docstrings
