Metadata-Version: 2.4
Name: scourgify
Version: 1.0.0
Summary: Normalize and consolidate tags/fandoms/characters/genres in a FanFicFare-imported Calibre library
Project-URL: Homepage, https://github.com/elfensky/scourgify
Project-URL: Repository, https://github.com/elfensky/scourgify
Project-URL: Issues, https://github.com/elfensky/scourgify/issues
Author-email: Andrei Lavrenov <andrei@lav.ren>
License-Expression: MIT
License-File: LICENSE
Keywords: calibre,ebook,epub,fanficfare,fanfiction,metadata,normalization,tags
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: End Users/Desktop
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Requires-Dist: rich>=13
Description-Content-Type: text/markdown

# scourgify

The tag-wrangler / canonizer for your fanfiction library — normalize and consolidate **tags,
fandoms, characters, relationships and genres** in a
[FanFicFare](https://github.com/JimmXinu/FanFicFare)-imported [Calibre](https://calibre-ebook.com)
library. Data-driven from **~1,700 bundled generic defaults**, fully customizable, audit-first and
reversible.

```bash
pipx install scourgify                                # or: uv tool install scourgify  (one dependency: rich)

export CALIBRE_LIBRARY="$HOME/Calibre/fanfiction"     # folder containing metadata.db
scourgify                                             # ← the wizard: everything below, menu-driven
```

Requires **Calibre installed** — the tool reads via read-only sqlite and shells out to Calibre's own
`calibre-debug` for writes. From a checkout, `uv run scourgify` (or `uvx --from . scourgify`) runs it
without installing; [uv](https://docs.astral.sh/uv/) handles the environment (one dependency: rich).

**The wizard** (no arguments) is the intended way in — and it steers you: on a fresh library it
detects missing columns/config and offers to run **setup** immediately; afterwards the status header
shows book count, column health, how many books are **new/changed since the last run**, and any
pending proposal, and the menu **defaults to whatever the library needs next**. Menu item **0 —
maintenance** walks the whole loop in the right order (wrangle → staleness → classify → review),
each step explained, previewed, and skippable. Classify runs show a **live dashboard** (progress +
tagged/failed/rate + throughput sparkline + rising tag candidates). Every write previews first,
asks for confirmation, and auto-backs-up `metadata.db`.

Each step is also a plain scriptable subcommand:

```bash
scourgify setup                                      # interactive health check + setup (FanFicFare, columns, config)
scourgify audit                                      # read-only dry-run report of every pass
scourgify apply --apply                              # write changes (Calibre CLOSED for this step)
```

Everything runs under plain `python3`. The tool reads via read-only sqlite and, for the actual writes,
shells out **once** to `calibre-debug -e _writer.py` (Calibre's API is the only fast batch-write path) —
so any command that writes (`apply --apply`, `setup` creating columns, `classify.py --apply`) needs
Calibre **closed**, and refuses to run while it's open. You never invoke `calibre-debug` yourself.

**`setup` is the first-run wizard + re-runnable health check.** It verifies, with `✓/⚠/✗` status and
`Y/n` prompts (default-yes; `--yes` to auto-accept): the library; that the **FanFicFare plugin is
installed and configured**, flagging + offering to fix the known gotchas (fandom-vs-series mapping,
`include_in_series:category`, unprotected `#genres`); that every needed column exists (`#fandoms`,
`#characters`, `#relationships`, `#genres`, `#status`, plus `#updated` and `#wrangled` for staleness /
incremental classification), creating any that are missing; and writes `config.toml` (preserving your
behavior toggles). Safe to re-run anytime.

**rich** is required for the wizard and powers the live dashboards/tables everywhere else; the plain
subcommands still degrade to text without it (rich is `try/except`-imported in the core tools, so
scripting/CI without rich keeps working, and `_writer.py` under Calibre's bundled Python needs none).

The engine reads:
- **`defaults/`** — bundled, generic fanfic knowledge (FFN `Harry P.`→`Harry Potter`, JP→English
  fandom titles, `no beta we die like…` = junk, …). Ships with the tool; not specific to anyone.
- **`config.toml`** — your column mapping + opinionated behavior toggles (generated by `setup`).
- **`overrides/`** *(optional)* — your own files (same formats as `defaults/`) that **extend and win
  over** the defaults.

---

## FanFicFare → Calibre columns (how the linking works)

FanFicFare scrapes metadata fields from each story and writes them into Calibre columns. The mapping
lives in the FFF Calibre-plugin config (stored per-library in the `metadata.db` preference
`namespaced:FanFicFarePlugin:settings` → key `custom_cols`). scourgify's `setup` reads that
mapping, and creates any recommended columns you're missing.

**Recommended mapping** (FFF metadata field → Calibre column):

| FanFicFare field | Calibre column | Type | Holds |
|---|---|---|---|
| `category`   | `#fandoms`       | text, multiple | fandom(s) — **map this to `category`, not `series`** (see gotcha) |
| `characters` | `#characters`    | text, multiple | characters |
| `ships`      | `#relationships` | text, multiple | pairings |
| `genre`      | `#genres`        | text, multiple | genres |
| `status`     | `#status`        | text           | In-Progress / Completed / … |
| `series`     | **Series** (built-in) | series    | the real site/AO3 series |
| `numWords`   | `#words`         | int            | word count |
| `numChapters`| `#chapters`      | int            | chapter count |
| `dateUpdated`| `#updated`       | datetime       | last-updated date |
| `storyUrl`   | `#storyurl`      | text           | source URL (also stored as the `url` identifier) |
| *subject tags* | `tags` (built-in) | —            | freeform tags (what this tool normalizes most) |

### ⚠️ The fandom-vs-series gotcha
FanFicFare's `personal.ini` setting **`include_in_series:category`** stuffs the *fandom* into FFF's
`series` field. If your `custom_cols` then maps **`#fandoms ← series`**, two things break: your
**Series** column fills with fandom names (not real series), and **#fandoms** is fed from that
fandom-stuffed series field. Fix:
1. Remove `include_in_series:category` from `personal.ini` → `series` becomes the real (e.g. AO3) series.
2. Map `#fandoms ← category` (the true fandom field).

`wrangle.py setup` detects and offers to fix both, plus the protection below (the original
`attic/apply_fff_config.py` does the same standalone). After that, real series fills in going
forward and fandoms come from `category`.

**Why fandom-as-series is especially bad:** Calibre's **Series** is a *numbered* field — every book
gets a `series_index` (`A Fandom Name [1]`, `[2]`, …). So fandom-as-series doesn't just duplicate the
fandom, it invents a bogus **ordered hierarchy**: dozens of unrelated stories become "book 1, book 2…
of Harry Potter," a sequence that reflects nothing real. Clearing it (see `attic/apply_other.py`) and
mapping `#fandoms ← category` removes the fake ordering; real series (where the index is meaningful,
e.g. a genuine 3-part AO3 series) then populate correctly.

### Franchise unification (fandom granularity)
Related works in one universe (e.g. `Fate/stay night`, `Fate/Zero`, `Fate/Grand Order`) are distinct
titles but one fandom. The bundled `defaults/fandoms.csv` unifies the obvious franchises to a single
canonical — **the Fate/Nasuverse works all map to `Type-Moon`** (the studio/umbrella name the
Nasuverse fandom uses). Prefer an **English title** as canonical wherever one exists (e.g.
`The Saga of Tanya the Evil`, not `Youjo Senki`; `Puella Magi Madoka Magica`, not the romaji). This
is a granularity *preference*: if you'd rather keep `Fate/Zero` separate from `Fate/stay night`,
remove those rows from your `overrides/fandoms.csv` (or leave them unmapped). Note some franchises
should **stay split** — Disney works are mostly standalone worlds (keep `DuckTales`, don't fold to a
`Disney` mega-fandom), and `Overlord (Game)` vs `Overlord (Anime)` are unrelated. Curated
unifications live in `build_defaults.py`'s `CURATED_FAN`.

### Protecting your cleanup from re-pollution
FFF's **`custom_cols_newonly`** (`{column: bool}`) controls overwrite-on-update: when `true`, FFF
only writes that column **if it's empty**, so a metadata refresh won't clobber your normalized
values. Recommended: `newonly:true` for `#genres`; leave `#status` **writable** so FanFicFare refreshes it on fetch (`staleness.py` re-derives the activity inference). The **built-in `tags` column is
never protected** by this — so new downloads/updates re-add raw tag junk, and you re-run
`wrangle.py` to clean it (see *Maintenance*).

---

## Customizing

**`config.toml`** — column map + behavior toggles (all have sane, opinionated defaults):

| Toggle | Default | Effect |
|---|---|---|
| `fold_characters` | `true` | apply abbreviation→full-name defaults |
| `ascii_only_tags` | `true` | transliterate non-ASCII tags to plain ASCII |
| `au_as` / `crossover_as` / `reincarnation_as` / `time_travel_as` | `genre` | put these tropes in `#genres` (`tag` to keep in tags) |
| `fold_ratings` | `false` | fold `Erotica`→`Smut`, `Adult`→`Mature` |
| `keep_categories` | `true` | keep `Multi`/`Gen`/`F/M` tags (`false` drops them) |

**`overrides/`** — drop in `characters.csv`, `fandoms.csv`, `tropes.csv`, `junk.txt`,
`genres_allow.txt`, … (same formats as `defaults/`). Anything here is merged on top of the bundled
defaults and wins on conflicts. This is where *your* preferences live — the code stays generic.

**`defaults/` formats:**
- `characters.csv` — `variant,canonical,fandom` (blank fandom = global; set = homonym-scoped, e.g. `Luke C.` differs in Marvel vs PJO)
- `fandoms.csv` — `alias,canonical`
- `tropes.csv` — `variant,canonical,route` (route = `tag`|`genre`|`character`|`fandom`)
- `genres_split.csv` (`combined,atoms`), `genres_canon.csv` (`variant,canonical`), `genres_allow.txt` (the genre vocabulary)
- `junk.txt` — drop list (plain line = case-insensitive exact; `re:<regex>` = regex)
- `ratings.txt` — content-rating/warning vocabulary

---

## Safety model
`audit` and `apply` compute the full new state in memory and assert **no book loses its last fandom
or character** (backfill-before-strip), aborting without writing if that fails. A second guardrail
aborts if tags would **mass-shrink** (>25% of assignments and >200 lost — the signature of an
over-broad `junk.txt` rule; `--force` overrides after you've checked). A redundant tag is only
stripped when the concept already lives in that book's structured column. `audit` is read-only
(plain `python3`, fine with Calibre open); `apply`/`setup` use the Calibre API (Calibre **closed**).
**Every write automatically snapshots `metadata.db` to `/tmp/ff_<timestamp>.db` first** and prints
the path — that's your instant rollback (master rollback = a full "Export all Calibre data" backup).

## Maintenance — after new downloads / updates
New stories arrive **raw** (junky subject tags, unfolded names). **Order matters — deterministic
cleanup first, content tagging second**, because raw junk tags inflate a book's tag count and would
hide it from the classifier's "sparsely tagged" targeting:

```bash
scourgify apply --apply                # 1. wrangle FIRST: junk-drop/canonicalize the new raw tags (idempotent)
scourgify staleness --apply            # 2. free: re-derive #status from #updated age (independent, any time)
scourgify classify --incremental       # 3. cheap: content-tag only new/changed books -> proposal
                                       # 4. review data/classify_proposal.csv
scourgify classify --apply             # 5. apply the reviewed tags + stamp #wrangled
```

Or just `scourgify` and walk the wizard menu in order: **3 wrangle → 4 staleness →
5 classify → 6 review**. Re-running wrangle is always safe — it's idempotent and won't regress
curated genres (it uses the full `genres_allow.txt`).

---

## Content-based tagging — `scourgify classify`
Reads each book's description (`#comments`) and produces **two outputs**: (1) `added_tags` — tags chosen
from the **controlled vocabulary** (`defaults/classify_vocab.txt`), which get applied; and (2) `proposed_new`
— short novel tags *not* in the vocab, aggregated by frequency into `classify_newtags_ranked.csv` so you can
review and **promote** the recurring ones into the vocab. Grows the tag set deliberately, without freeform noise.

```bash
scourgify classify --engine apple --limit 50        # propose -> data/classify_proposal.csv (dry-run, read-only)
scourgify classify --apply                          # add the proposed tags (Calibre CLOSED)
```
- `--engine apple` — on-device **Apple Foundation Models** via `afm.swift` (free, private; macOS 26+,
  Apple Intelligence). Ships as source; a `swift` toolchain runs it as-is, or from a checkout build the
  faster binary once: `swiftc -O src/scourgify/afm.swift -o src/scourgify/afm`. Lower quality — prone to over-tagging,
  so the prompt caps at `--max-tags 6` and dumps (>2× cap) are rejected.
- `--engine claude|openai|gemini` — cloud APIs (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / `GEMINI_API_KEY`);
  defaults `claude-haiku-4-5` / `gpt-4o-mini` / `gemini-2.5-flash`, override with `--model`. Sharper; cheap.
- Only books with `< --min-tags` (default 2) tags **and** a description are touched. Always dry-run
  until `--apply`. Edit `defaults/classify_vocab.txt` to shape the allowed tag set.
- Long runs **save incrementally and resume** on re-run (skip books already in the proposal; `--fresh`
  to restart). `--batch N` processes only N new books per run — handy for pacing API spend/rate limits.
  A **spend gate** asks for confirmation (or `--yes`) before sending more than 200 books to a cloud engine.
- Proposals/outputs live in `data/` (gitignored): `classify_proposal.csv`, `classify_newtags_ranked.csv`,
  `classify_failures.csv`. On `--apply` the proposal is **archived** to `classify_proposal_applied_<ts>.csv`,
  so a later apply can never re-add tags you've since hand-removed in Calibre.
- **Incremental maintenance (`--incremental`):** after new FanFicFare downloads, `classify.py --incremental` (re)tags
  only books whose `#updated` is newer than their own **`#wrangled`** marker (a datetime column auto-created and
  stamped on `--apply`), plus any still untagged — cents instead of a full pass. State lives *in the library*
  (travels with `metadata.db`, no external file). A full cloud `--fresh` run is expensive; reserve it for vocab changes.

## Custom maps from your library (`overrides/`)
The bundled `defaults/` are generic. Two helper workflows mined library-specific maps into `overrides/`
(gitignored): AO3-style tag clustering (`overrides/tropes.csv`) and fandom **universe-unification**
(`overrides/fandoms.csv`, e.g. `Avengers`/`Captain America (Movies)` → `Marvel`, `Game of Thrones (TV)`
→ `A Song of Ice and Fire`). The engine loads these on top of the defaults automatically.

## Repo layout
The package lives in **`src/scourgify/`**; the single `scourgify` command (`cli.py`) dispatches
bare → wizard, `setup`/`audit`/`apply` → wrangle, `classify`, and `staleness`.
- **`cli.py`** — the `scourgify` entry point (argv dispatcher over the tools below)
- **`wrangle.py`** — the engine: `setup` / `audit` / `apply`; with no command it launches the wizard
- **`wizard.py` + `ui.py`** — the interactive wizard and its rich terminal helpers (the one
  rich-required surface)
- **`classify.py`** — content-based tagging (LLM engines) · **`staleness.py`** — `#status` re-derivation
- **`common.py`** — shared core: library resolution, read-only sqlite + custom-column reading,
  config, and `run_writer()` (the single write funnel, with automatic backup)
- **`_writer.py`** — the only file that imports Calibre; a generic ops executor invoked under
  `calibre-debug` by `run_writer()`, never by hand
- **`defaults/`** — bundled generic maps, shipped inside the package (read-only at runtime)
- Per-user files resolve against the **working directory**: `config.toml`, `overrides/` (your maps,
  gitignored), and `data/` (proposals/intermediates, gitignored)
- **`build_defaults.py`** (repo root) — maintainer tool: regenerates `defaults/` from the review maps in `data/`
- **`tests/`** — `uv run tests/test_core.py`, no library needed
- **`attic/`** — the original single-purpose pipeline, kept as provenance (see `attic/README.md`);
  `scourgify` supersedes it.

The per-library review-map CSVs in `data/` are gitignored (they contain your library's actual data);
only the generic `defaults/` ship with the repo.
