Metadata-Version: 2.4
Name: velocirepo
Version: 0.3.0
Summary: All your open-source and social-media metrics in one place
Project-URL: Homepage, https://github.com/jeroenjanssens/velocirepo
Project-URL: Repository, https://github.com/jeroenjanssens/velocirepo
Project-URL: Issues, https://github.com/jeroenjanssens/velocirepo/issues
Author-email: Jeroen Janssens <jeroen@jeroenjanssens.com>
License-Expression: MIT
Keywords: analytics,github,metrics,open-source,pypi
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Other
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# velocirepo

All your open-source and social-media metrics in one place.

## Table of contents

- [Overview](#overview)
- [Installation](#installation)
- [Configuration](#configuration)
- [Usage](#usage)
- [Data storage](#data-storage)
- [Migrating data](#migrating-data)
- [Querying the data](#querying-the-data)
- [Exporting data](#exporting-data)
- [Generating badges](#generating-badges)

## Overview

velocirepo collects metrics from multiple sources, stores them as JSONL files, and exposes them via SQL (powered by DuckDB). It's designed to run on a schedule (e.g., nightly via GitHub Actions) and commit the results to git, giving you a permanent record of your project's growth.

Supported sources:

| Source | What it tracks |
|--------|----------------|
| **GitHub Events** | Individual events (stars, forks, issues, PRs) with user and timestamp |
| **GitHub Traffic** | Daily page views and git clones (requires admin access) |
| **PyPI** | Daily download counts |
| **CRAN** | Daily download counts |
| **Homebrew** | Install counts (30-day, 90-day, 365-day, lifetime) |
| **Plausible** | Daily pageviews, visitors, visits |
| **OpenVSX** | Total downloads, reviews, ratings |
| **YouTube** | Views, likes, comments, subscribers (channel and per-video) |

## Installation

velocirepo can be used as a GitHub Action for automated nightly fetching, or installed locally for ad-hoc use.

### GitHub Actions (recommended)

```yaml
name: Fetch Metrics

on:
  schedule:
    - cron: '0 1 * * *'
  workflow_dispatch:

permissions:
  contents: write

jobs:
  fetch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: jeroenjanssens/velocirepo@v0
        with:
          github-token: ${{ secrets.GH_TOKEN }}
          # plausible-token: ${{ secrets.PLAUSIBLE_TOKEN }}
          # youtube-token: ${{ secrets.YOUTUBE_TOKEN }}

      - name: Commit and push
        run: |
          git config --local user.email "github-actions[bot]@users.noreply.github.com"
          git config --local user.name "github-actions[bot]"
          git add velocirepo/
          git diff --staged --quiet || git commit -m "Update metrics - $(date -u +'%Y-%m-%d')"
          git push
```

Or generate this workflow automatically (no installation required):

```bash
uvx velocirepo install-ci
```

### Local installation

#### Homebrew (macOS / Linux)

```bash
brew install jeroenjanssens/tap/velocirepo
```

#### Scoop (Windows)

```powershell
scoop bucket add jeroenjanssens https://github.com/jeroenjanssens/scoop-bucket
scoop install velocirepo
```

#### uv

```bash
uv tool install velocirepo
```

Or run without installing:

```bash
uvx velocirepo fetch
```

#### Go

```bash
go install github.com/jeroenjanssens/velocirepo/cmd/velocirepo@latest
```

#### From source

```bash
git clone https://github.com/jeroenjanssens/velocirepo.git
cd velocirepo
go build -o bin/velocirepo ./cmd/velocirepo
cp bin/velocirepo ~/.local/bin/
```

#### Shell installer (macOS / Linux)

```bash
curl -sSfL https://raw.githubusercontent.com/jeroenjanssens/velocirepo/main/install.sh | sh
```

This downloads the latest binary to `./bin/velocirepo`. Set `INSTALL_DIR` to install elsewhere:

```bash
curl -sSfL https://raw.githubusercontent.com/jeroenjanssens/velocirepo/main/install.sh | INSTALL_DIR=/usr/local/bin sh
```

#### Download binary

Pre-built binaries for Linux, macOS, and Windows are available on the [Releases](https://github.com/jeroenjanssens/velocirepo/releases) page.

## Configuration

Create a `velocirepo.toml` in your project root:

```toml
[projects.my-project]
name = "My Project"
github-events = "owner/repo"
github-traffic = "owner/repo"
pypi = "my-package"

[projects.other-project]
name = "Other Project"
github-events = ["owner/other", "owner/other-utils"]
cran = "other"
homebrew = "other"
youtube = "@ChannelHandle"
```

Each source field accepts either a single string or an array of strings, so you can track multiple repositories or packages under one project.

The `github-traffic` source fetches daily page views and clone counts. GitHub only retains this data for 14 days, so velocirepo preserves it before it's lost. It requires a token with **Administration:read** permission (or the `repo` scope for classic tokens).

Or initialize one interactively (auto-detects sources from your repository):

```bash
velocirepo init
```

velocirepo looks for `velocirepo.toml` by walking up from the current directory. Override with `--config` or the `VELOCIREPO_CONFIG` environment variable.

### Environment variables

| Variable | Description |
|----------|-------------|
| `GITHUB_TOKEN` | GitHub personal access token (increases rate limits) |
| `PLAUSIBLE_TOKEN` | Plausible API token |
| `YOUTUBE_TOKEN` | YouTube Data API key |
| `VELOCIREPO_CONFIG` | Path to config file |

These can also be set in a `.env` file in the current directory.

## Usage

```
velocirepo fetch                 Fetch from all configured sources
velocirepo fetch-github          Fetch GitHub events (stars, forks, issues, PRs)
velocirepo fetch-traffic         Fetch GitHub traffic (views and clones)
velocirepo fetch-pypi            Fetch PyPI download counts
velocirepo fetch-cran            Fetch CRAN download counts
velocirepo fetch-homebrew        Fetch Homebrew install counts
velocirepo fetch-plausible       Fetch Plausible analytics
velocirepo fetch-openvsx         Fetch Open VSX extension metrics
velocirepo fetch-youtube         Fetch YouTube metrics

velocirepo query <sql>           Run a SQL query against the metrics data
velocirepo schema                Show table schemas

velocirepo export <dir>          Export data to Parquet or CSV files

velocirepo badge stars           Generate a stars badge
velocirepo badge forks           Generate a forks badge
velocirepo badge downloads       Generate a downloads badge
velocirepo badge pageviews       Generate a pageviews badge
velocirepo badge custom          Generate a badge from a custom query

velocirepo init                  Create a new velocirepo.toml
velocirepo add-project           Add a project to the config
velocirepo update-project        Update a project's configuration
velocirepo remove-project        Remove a project from the config
velocirepo rename-project        Rename a project's ID
velocirepo list-projects         List configured projects
velocirepo show-project          Show project details
velocirepo import-projects       Bulk-import from GitHub org/user or file
velocirepo validate-projects     Validate source URLs

velocirepo migrate               Migrate data to the latest schema version

velocirepo install-ci            Generate a GitHub Actions workflow
velocirepo sync-secrets          Sync .env secrets to GitHub Actions

velocirepo version               Print version information
```

## Data storage

All data is stored as JSONL files at `velocirepo/data/<source>/<project-id>/<date>.jsonl`. This entire directory is meant to be committed to git — it's your permanent metric history.

### Metrics sources

Sources like PyPI, CRAN, Homebrew, Plausible, OpenVSX, and GitHub Traffic store one JSON object per metric per day:

```json
{"source":"pypi","metric":"daily_downloads","project_id":"plotnine","target":"plotnine","date":"2026-06-15","value":1523}
{"source":"openvsx","metric":"total_downloads","project_id":"quarto","target":"quarto/quarto","date":"2026-06-15","value":1250000}
```

Metric names are prefixed with `daily_` (for deltas — values that reset each day) or `total_` (for snapshots — cumulative totals at a point in time). Homebrew metrics use their own naming (`downloads_30d`, `downloads_365d`, etc.).

Fields:

| Field | Description |
|-------|-------------|
| `source` | Source name (pypi, cran, homebrew, plausible, openvsx, github-traffic) |
| `metric` | Metric name (e.g., `daily_downloads`, `total_views`, `daily_pageviews`) |
| `project_id` | Project ID from your config |
| `target` | Specific package, repo, site, or extension being tracked |
| `date` | Date of the measurement (YYYY-MM-DD) |
| `value` | Integer value |
| `tags` | Optional key-value metadata (e.g., `{"video_id": "..."}` for YouTube) |

### GitHub Events source

The GitHub Events source stores individual events rather than pre-computed counts, giving you full historical detail including who performed each action and when:

```json
{"source":"github","event_type":"star","project_id":"quarto","github_repo":"quarto-dev/quarto-cli","datetime":"2026-06-15T14:23:01Z","user":"alice"}
{"source":"github","event_type":"fork","project_id":"quarto","github_repo":"quarto-dev/quarto-cli","datetime":"2026-06-15T09:11:44Z","user":"bob"}
```

Fields:

| Field | Description |
|-------|-------------|
| `source` | Always `github` |
| `event_type` | One of: star, fork, issue_open, issue_close, pr_open, pr_merge |
| `project_id` | Project ID from your config |
| `github_repo` | The owner/repo being tracked |
| `datetime` | Full timestamp of the event (ISO 8601) |
| `user` | GitHub username who performed the action |

These events are automatically aggregated into daily counts in the `metrics` DuckDB view (as `daily_stars`, `daily_forks`, etc.) so you can query them alongside other sources.

### YouTube index

The YouTube source also writes an `index.jsonl` file at `velocirepo/data/youtube/<project-id>/index.jsonl` containing video metadata:

```json
{"video_id":"ML3q7Ok4hJg","title":"God-Tier Developer Roadmap","published_at":"2024-03-15T16:00:00Z","channel":"@Fireship","duration":423,"tags":["programming","roadmap"]}
```

This is exposed as the `youtube_index` DuckDB view, allowing you to join video titles and publish dates with metrics data.

### Concatenation

Daily JSONL files are automatically concatenated into monthly and yearly files once a period is complete. For example, once all days in January 2026 have been fetched, they're concatenated into `2026-01.jsonl`. This keeps the file count manageable for long-running histories. The original daily files are removed after concatenation.

### Repository layout

You can either keep metrics in the same repository as your code, or create a dedicated metrics repository. A separate repo is useful when you want to track multiple projects in one place or keep metric history out of your main codebase.

## Migrating data

When a new version of velocirepo changes the on-disk data format, it tracks this with a schema version number in `velocirepo/data/.schema-version`. Commands like `fetch`, `query`, and `export` will refuse to run against stale data:

```
Error: data schema is at version 0, but version 1 is required; run `velocirepo migrate` to update
```

To migrate your data to the latest schema:

```bash
velocirepo migrate
```

If you've copied in data from an older schema (e.g., merged data from another repository), some files may be at a different version than what `.schema-version` claims. Use `--force` to re-run all migrations from scratch:

```bash
velocirepo migrate --force
```

This is safe to run repeatedly — all migrations are idempotent.

## Querying the data

The `query` command reads JSONL files directly using DuckDB and exposes four views:

| View | Description |
|------|-------------|
| `metrics` | Unified time-series: all sources including aggregated GitHub events |
| `github_events` | Raw GitHub events with user and timestamp |
| `youtube_index` | Video metadata (title, publish date, channel, duration) |
| `projects` | Project metadata from your config |

### Daily star counts

```bash
velocirepo query "
  SELECT project, date, value AS stars
  FROM metrics
  WHERE source = 'github' AND metric = 'daily_stars'
  ORDER BY date DESC
  LIMIT 5
"
```

### Total stars per project

```bash
velocirepo query "
  SELECT p.name, SUM(value) AS stars
  FROM metrics m
  JOIN projects p ON m.project = p.id
  WHERE m.source = 'github' AND m.metric = 'daily_stars'
  GROUP BY p.name
  ORDER BY stars DESC
  LIMIT 5
"
```

```
┌─────────────┬───────┐
│    name     │ stars │
├─────────────┼───────┤
│ ggplot2     │ 6877  │
│ Shiny for R │ 5600  │
│ Quarto      │ 5274  │
│ dplyr       │ 4995  │
│ plotnine    │ 4500  │
└─────────────┴───────┘
```

### Monthly star activity using raw events

The `github_events` view gives you access to individual events when you need per-user or per-timestamp detail:

```bash
velocirepo query "
  SELECT date_trunc('month', datetime)::DATE AS month, COUNT(*) AS stars
  FROM github_events
  WHERE project = 'quarto' AND event_type = 'star'
  GROUP BY month
  ORDER BY month DESC
  LIMIT 5
"
```

```
┌────────────┬───────┐
│   month    │ stars │
├────────────┼───────┤
│ 2026-02-01 │ 40    │
│ 2026-01-01 │ 107   │
│ 2025-12-01 │ 91    │
│ 2025-11-01 │ 97    │
│ 2025-10-01 │ 85    │
└────────────┴───────┘
```

### Top YouTube videos by views

```bash
velocirepo query "
  SELECT yi.title, m.value AS views
  FROM metrics m
  JOIN youtube_index yi ON m.tags->>'video_id' = yi.video_id
  WHERE m.source = 'youtube' AND m.metric = 'total_views'
  ORDER BY m.value DESC
  LIMIT 5
"
```

### Latest metrics across sources

```bash
velocirepo query "
  SELECT project, source, metric, date, value
  FROM metrics
  WHERE source != 'github'
  ORDER BY date DESC
  LIMIT 5
"
```

```
┌──────────┬─────────┬─────────────────┬────────────┬─────────┐
│ project  │ source  │     metric      │    date    │  value  │
├──────────┼─────────┼─────────────────┼────────────┼─────────┤
│ quarto   │ openvsx │ total_downloads │ 2026-06-16 │ 3101234 │
│ quarto   │ openvsx │ total_ratings   │ 2026-06-16 │ 500     │
│ quarto   │ openvsx │ total_reviews   │ 2026-06-16 │ 2       │
│ quarto   │ plausible│ daily_pageviews │ 2026-06-16 │ 14322   │
│ quarto   │ plausible│ daily_visitors  │ 2026-06-16 │ 3324    │
└──────────┴─────────┴─────────────────┴────────────┴─────────┘
```

### Output formats

By default, results are printed as a table. Use `--json`, `--csv`, or `--parquet` for machine-readable output:

```bash
velocirepo query --csv "SELECT project, metric, value FROM metrics LIMIT 3"
velocirepo query --json "SELECT project, metric, value FROM metrics LIMIT 3"
velocirepo query --parquet "SELECT * FROM metrics" > metrics.parquet
```

The `schema` command shows all available columns:

```bash
velocirepo schema
```

## Exporting data

Export metrics, events, and project metadata to Parquet or CSV for use in other tools:

```bash
velocirepo export ./out/
```

This writes one file per table:

```
  out/metrics.parquet (257 KB)
  out/github_events.parquet (2.9 MB)
  out/youtube_index.parquet (15 KB)
  out/projects.parquet (2 KB)
```

Use `--format csv` for CSV output, and `--source` or `--project` to filter:

```bash
velocirepo export ./out/ --format csv
velocirepo export ./out/ --source github
velocirepo export ./out/ --project quarto
```

## Generating badges

Generate shields.io-style SVG badges from your metrics data:

```bash
velocirepo badge stars -o stars.svg
velocirepo badge forks --project quarto -o forks.svg
velocirepo badge downloads --project ggplot2 -o downloads.svg
```

For custom metrics, provide a SQL query that returns a single value:

```bash
velocirepo badge custom \
  --label "contributors" \
  --query "SELECT COUNT(DISTINCT \"user\") AS value FROM github_events" \
  --color "#ea7233" \
  -o contributors.svg
```

### Styles

Three styles are available via `--style`:

| Style | Description |
|-------|-------------|
| `flat` (default) | Subtle gradient, rounded corners |
| `flat-square` | No gradient, sharp corners |
| `plastic` | Heavy gradient, more rounded |

### Options

| Flag | Default | Description |
|------|---------|-------------|
| `--project` | all | Scope to a specific project |
| `--style` | `flat` | Badge style |
| `--color` | preset-dependent | Message background color (hex) |
| `--label-color` | `#555` | Label background color |
| `--label` | preset-dependent | Override label text |
| `--height` | style-dependent | Badge height in pixels |
| `--radius` | style-dependent | Corner radius (0 = square) |
| `-o` | stdout | Output file path |

Numbers are automatically formatted for readability (e.g., `5274` becomes `5.3k`, `1500000` becomes `1.5M`).

## License

MIT
