Metadata-Version: 2.4
Name: lightnovel-crawler
Version: 4.14.0
Summary: Download lightnovels from 300+ sources and export to EPUB, MOBI, PDF, DOCX, and more — with a CLI, REST API, and web UI.
Author-email: Lightnovel Crawler Team <lncrawl@pm.me>
License-Expression: GPL-3.0-or-later
Project-URL: Changelog, https://github.com/lncrawl/lightnovel-crawler/blob/master/CHANGELOG.md
Project-URL: Documentation, https://github.com/lncrawl/lightnovel-crawler/blob/master/README.md
Project-URL: Issue Tracker, https://github.com/lncrawl/lightnovel-crawler/issues
Project-URL: Source Code, https://github.com/lncrawl/lightnovel-crawler
Keywords: lightnovel,webnovel,crawler,scraper,downloader,novel,ebook,epub,mobi,pdf
Classifier: Development Status :: 5 - Production/Stable
Classifier: Natural Language :: English
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: Topic :: Games/Entertainment
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Framework :: FastAPI
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: alembic>=1.14.0
Requires-Dist: base58~=2.1.1
Requires-Dist: beautifulsoup4<5.0.0,>=4.0.0
Requires-Dist: brotli>=1.2.0
Requires-Dist: colorama<0.5.0,>=0.4.0
Requires-Dist: ebooklib<1.0.0,>=0.18.0
Requires-Dist: fastapi[standard]>=0.115.0
Requires-Dist: html5lib~=1.1
Requires-Dist: httpx[brotli,http2]>=0.27.0
Requires-Dist: humanize>=4.0.0
Requires-Dist: imap-tools>=1.10.0
Requires-Dist: lncrawl-scraper[all]~=1.7.0
Requires-Dist: lncrawl-translator>=0.2.1
Requires-Dist: lxml<7.0.0,>=5.4.0
Requires-Dist: passlib[argon2]>=1.7.0
Requires-Dist: pillow>=10.0.0
Requires-Dist: psycopg[binary]>=3.2.0
Requires-Dist: pycryptodome<4.0.0,>=3.0.0
Requires-Dist: pyease-grpc>=1.8.0
Requires-Dist: pyparsing<4.0.0,>=3.0.0
Requires-Dist: python-box<8.0.0,>=7.0.0
Requires-Dist: python-dateutil<3.0.0,>=2.9.0
Requires-Dist: python-dotenv<2.0.0,>=1.0.0
Requires-Dist: python-jose[cryptography]>=3.4.0
Requires-Dist: python-slugify<9.0.0,>=8.0.0
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: questionary<3.0.0,>=2.0.0
Requires-Dist: exejs>=0.0.7
Requires-Dist: readability-lxml<1.0.0,>=0.8.0
Requires-Dist: regex>=2024.0.0
Requires-Dist: sqlmodel>=0.0.22
Requires-Dist: tqdm<5.0.0,>=4.66.0
Requires-Dist: typer>=0.12.0
Requires-Dist: uvicorn>=0.32.0
Requires-Dist: zstd>=1.5.7.0
Provides-Extra: build
Requires-Dist: build>=1.2.0; extra == "build"
Requires-Dist: pyinstaller>=6.0.0; extra == "build"
Requires-Dist: setuptools>=75.3.0; extra == "build"
Requires-Dist: tk-tools>=0.16.0; extra == "build"
Requires-Dist: wheel>=0.40.0; extra == "build"
Provides-Extra: lsp
Requires-Dist: pylsp-rope>=0.1.17; extra == "lsp"
Requires-Dist: pyright>=1.1.409; extra == "lsp"
Requires-Dist: python-lsp-ruff>=2.3.1; extra == "lsp"
Requires-Dist: python-lsp-server[websockets]>=1.14.0; extra == "lsp"
Requires-Dist: ruff>=0.9.0; extra == "lsp"
Dynamic: license-file

<div align="center">

<img src="https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/lncrawl-icon.png" width="112" alt="Lightnovel Crawler" />

# Lightnovel Crawler

**Turn a web novel into an e-book with one command —<br>
or run a private server that keeps your whole library<br>
downloaded, translated, and readable in the browser.**

[![PyPI](https://img.shields.io/pypi/v/lightnovel-crawler.svg?logo=pypi&logoColor=white)](https://pypi.org/project/lightnovel-crawler/)
[![Python](https://img.shields.io/pypi/pyversions/lightnovel-crawler.svg?logo=python&logoColor=white)](https://pypi.org/project/lightnovel-crawler/)
[![Downloads](https://static.pepy.tech/badge/lightnovel-crawler)](https://pepy.tech/project/lightnovel-crawler)
[![Docker](https://img.shields.io/badge/ghcr.io-lightnovel--crawler-2496ed?logo=docker&logoColor=white)](https://github.com/lncrawl/lightnovel-crawler/pkgs/container/lightnovel-crawler)
[![License](https://img.shields.io/badge/license-GPLv3-blue.svg)](https://github.com/lncrawl/lightnovel-crawler/blob/master/LICENSE)
<br>
[![Build and Publish](https://github.com/lncrawl/lightnovel-crawler/actions/workflows/release.yml/badge.svg)](https://github.com/lncrawl/lightnovel-crawler/actions/workflows/release.yml)
[![Lint](https://github.com/lncrawl/lightnovel-crawler/actions/workflows/lint.yml/badge.svg)](https://github.com/lncrawl/lightnovel-crawler/actions/workflows/lint.yml)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/lncrawl/lightnovel-crawler)

[**Download**](https://github.com/lncrawl/lightnovel-crawler#install) ·
[**Supported sources**](https://github.com/lncrawl/lightnovel-crawler/blob/master/SOURCES.md) ·
[**Web app**](https://github.com/lncrawl/lightnovel-crawler#the-web-app) ·
[**Command line**](https://github.com/lncrawl/lightnovel-crawler#the-command-line) ·
[**Changelog**](https://github.com/lncrawl/lightnovel-crawler/blob/master/CHANGELOG.md) ·
[**Contributing**](https://github.com/lncrawl/lightnovel-crawler/blob/master/.github/CONTRIBUTING.md)

</div>

---

```bash
pip install -U lightnovel-crawler

lncrawl crawl "https://example.com/novel/page" -f epub --all
```

That is the whole download path: it discovers the chapter list, fetches every chapter, and
writes an EPUB you can put on an e-reader. The same package also ships a server — one command
turns it into a multi-user library with a browser reader, translations, and scheduled
re-downloads of the stories you follow.

<!-- auto generated source count -->

Currently **361 sources** across 11 languages, served by **446 crawlers**.

<!-- auto generated source count -->

[![The web app](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/novels.png)](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/novels.png)

> **Personal use only.** Sites publish fiction under their own terms and copyright. Use this
> for personal backups of content you have legitimate access to. Do not redistribute or sell
> someone else's work.

## What you get

|                                                |                                                                                                                                                                                                                                                    |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Hundreds of sites, one interface**           | Every source is a small crawler file with the same contract, so a URL is all you supply. Search runs across every source that supports it. The [full list](https://github.com/lncrawl/lightnovel-crawler/blob/master/SOURCES.md) is regenerated by CI. |
| **A library, not a downloads folder**          | Shelve novels into libraries, favourite them, and re-download when a story updates. Reading history and progress follow you between devices.                                                                                                        |
| **Reads in the browser**                       | A paginated reader with fonts, themes and text-to-speech — so a download is readable the moment it finishes, without exporting anything first.                                                                                                      |
| **Translates while it downloads**              | Multi-engine translation with failover, driven by [`lncrawl-translator`](https://pypi.org/project/lncrawl-translator/). A per-novel glossary keeps character and place names consistent across every chapter.                                       |
| **Eighteen output formats**                    | EPUB, TXT and JSON on their own; the rest through Calibre. One download can produce several at once, per volume or whole-novel.                                                                                                                     |
| **Made for more than one person**              | Accounts with tiers and quotas, e-mail notification on job completion, announcements, feedback, and admin screens for all of it.                                                                                                                    |
| **Knows _why_ a site said no**                 | Blocks are diagnosed rather than retried — see [below](https://github.com/lncrawl/lightnovel-crawler#when-a-site-says-no). A challenge page served as `200` is a failure, not a chapter.                                                             |
| **Fix a source without leaving the browser**   | The source editor gives you the crawler file with completions and diagnostics from a real language server, and a test run that streams its results back.                                                                                            |
| **Scriptable end to end**                      | Everything the web app does goes through a documented REST API — OpenAPI at `/docs`, ReDoc at `/redoc`.                                                                                                                                             |

## Install

Pick **one**. All three ship the same engine, the same sources and the same web app.

### Standalone — no Python needed

| Platform | Download                                                                |
| -------- | ----------------------------------------------------------------------- |
| Windows  | [📦 lncrawl.exe](https://go.bitanon.dev/lncrawl-windows)                |
| Linux    | [📦 lncrawl](https://go.bitanon.dev/lncrawl-linux)                      |
| macOS    | [📦 lncrawl](https://go.bitanon.dev/lncrawl-mac)                        |

Run it and the desktop app opens with no login.
[Older versions](https://github.com/lncrawl/lightnovel-crawler/releases) are on the releases
page.

[![Tutorial](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/tutorial.png)](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/tutorial.png)

### pip — the CLI and the server

Python 3.9 or newer.

```bash
pip install -U lightnovel-crawler
lncrawl -h
```

_If `pip` fails, try `python -m pip install -U lightnovel-crawler`. If `lncrawl` is not on your
PATH, `python -m lncrawl` works the same._

<details>
<summary>Install straight from the repository</summary>

```bash
# latest stable
pip install -U git+https://github.com/lncrawl/lightnovel-crawler.git#egg=lightnovel-crawler

# development branch — newest fixes, no stability promise
pip install -U https://github.com/lncrawl/lightnovel-crawler/tarball/refs/heads/dev#egg=lightnovel-crawler
```

</details>

<!-- auto generated command line output -->

```text
$ lncrawl -h
Usage: lncrawl [OPTIONS] COMMAND [ARGS]...                                     
                                                                                
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --verbose             -l              Log levels: -l = warn, -ll = info,     │
│                                       -lll = debug                           │
│ --config              -c      <path>  Config file                            │
│ --install-completion                  Install completion for the current     │
│                                       shell.                                 │
│ --show-completion                     Show completion for the current shell, │
│                                       to copy it or customize the            │
│                                       installation.                          │
│ --help                -h              Show this message and exit.            │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ app      Launches the web application.                                       │
│ version  Show current version.                                               │
│ dev      Run development commands.                                           │
│ config   View and modify configuration settings.                             │
│ sources  Manage sources.                                                     │
│ crawl    Crawl from novel page URL.                                          │
│ search   Search for novels by query string.                                  │
│ server   Run web server.                                                     │
╰──────────────────────────────────────────────────────────────────────────────╯
```

<!-- auto generated command line output -->

### Docker — the home server

```bash
mkdir -p lncrawl-data
docker run -d --name lncrawl-server \
  -v ./lncrawl-data:/data \
  -p 8181:8181 \
  -e TZ=Europe/London \
  ghcr.io/lncrawl/lightnovel-crawler -ll server
```

Open **http://localhost:8181** and sign in with `admin` / `admin`. Change that password in
**Settings → Account** before anyone else can reach the port.

> **Set `TZ`.** A container's clock is UTC, and a browser whose timezone disagrees with where
> its address geolocates reads as a robot. Measured in this image: **1 of 3 challenged sites
> cleared on the default clock, 3 of 3 with `TZ`** set to the zone your address looks like it
> is in. It matters more than it sounds — see
> [When a site says no](https://github.com/lncrawl/lightnovel-crawler#when-a-site-says-no).

## The web app

`lncrawl app` opens it as a desktop window with no login. `lncrawl server` runs it for the
network, with accounts — the same thing the Docker image starts.

1. **Crawlers** lists every supported site, or paste a novel URL straight into the search bar.
2. Pick the chapters — a range, or all of them.
3. Pick the output formats, and a language if you want it translated.
4. **Requests** tracks the job while it runs.
5. Read it in the browser, or download the file.

Saving a novel to a **Library** is what makes the next update cheap: re-download fetches only
the chapters that appeared since.

|                                                                                                                                                                                                                                              |                                                                                                                                                                                                                                                     |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [![Crawlers](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/crawlers.png)](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/crawlers.png)                              | [![Requests](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/requests.png)](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/requests.png)                                     |
| [![Reader](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/reader.png)](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/reader.png)                                    | [![Libraries](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/libraries.png)](https://raw.githubusercontent.com/lncrawl/lightnovel-crawler/master/res/screenshots/libraries.png)                                  |

## The command line

```bash
# first 10 chapters as EPUB
lncrawl crawl "https://example.com/novel/page" -f epub --first 10

# everything, in two formats
lncrawl crawl "https://example.com/novel/page" -f epub -f txt --all

# only the chapters that are not downloaded yet
lncrawl crawl "https://example.com/novel/page" -f epub --resume

# find it by title instead
lncrawl search "The Beginning After The End"

# what does this build know about a site?
lncrawl dev explain https://example.com/

# open the app, or serve it
lncrawl app
lncrawl server --port 8181
```

Any URL from a [supported source](https://github.com/lncrawl/lightnovel-crawler/blob/master/SOURCES.md)
works. `lncrawl <command> -h` documents the rest, and `lncrawl config` reads and writes the
settings the web app shows under **Settings**.

## Output formats

| Format       | Calibre | Best for                        |
| ------------ | :-----: | ------------------------------- |
| 📚 **epub**  |         | Most e-readers and reading apps |
| 📃 **txt**   |         | Any text editor                 |
| 🗂️ **json**  |         | Scripts and pipelines           |
| 📄 **pdf**   |    ✓    | Print-ready, universal          |
| 🔳 **azw3**  |    ✓    | Kindle (current)                |
| 🔲 **mobi**  |    ✓    | Kindle (older devices)          |
| 📓 **kepub** |    ✓    | Kobo                            |
| 📝 **docx**  |    ✓    | Word, LibreOffice               |
| 📑 **rtf**   |    ✓    | WordPad and others              |
| 📔 **fb2**   |    ✓    | FB2 readers                     |
| 🌐 **htmlz** |    ✓    | A book as one HTML bundle       |
| 📕 **lit**   |    ✓    | MS Reader (obsolete)            |
| 📗 **lrf**   |    ✓    | Sony readers                    |
| 🗄️ **pdb**   |    ✓    | PalmOS (legacy)                 |
| 📒 **pmlz**  |    ✓    | eReader / Palm Markup           |
| 📘 **rb**    |    ✓    | RocketBook / REB1100            |
| 📙 **tcr**   |    ✓    | Psion readers                   |
| 📓 **snb**   |    ✓    | Shanda Bambook                  |

The ticked formats are converted from the EPUB by
[Calibre](https://calibre-ebook.com/download), which you install separately. On macOS and Linux
its `ebook-convert` is found automatically; on Windows, add the Calibre folder to your `Path`.
A server that would rather not carry Calibre can point the `calibre` config section at a remote
[ebook-convert-api](https://github.com/lncrawl/ebook-convert-api) instead.

## When a site says no

Most of these sites sit behind bot detection, and a downloader that answers a block by retrying
harder is a downloader that quietly stops working. This one reads _what_ is blocking before it
reacts.

- **It escalates only as far as it has to.** A plain request first, a different address when
  the block is about where you came from, a real browser only when a challenge genuinely needs
  one. Rotating the proxy at a rate limit is the reflex that makes things worse — the new
  address gets throttled just the same, because the pacing was the problem.
- **Challenges are solved in a browser you already have.** Firefox for preference, Chrome as
  the fallback, hidden by default. The clearance is reused, so one solve covers a whole book
  instead of one page.
- **An empty page is a failure, not a success.** A challenge screen and a page of adverts both
  answer `200`. Parsed as a chapter they produce a download that reports success and contains
  nothing, which is the most common way a source breaks.

Ask what it knows about any site:

```console
$ lncrawl dev explain https://aquareader.org/
aquareader.org
  binding layer : L9 Managed JavaScript challenge — reads a hybrid property, solve
  tier          : direct (unproven)
  pacing        : 3.0s mean interval
  requests      : 0 ok / 5 failed
  clearance     : none
  ladder        : archive(0) direct(10)
  exits         : direct
```

When a download fails you get that, not `403 after 3 retries` — which layer is blocking, what
it reads, and whether anything you can configure would move it. A block on your address is
answered by adding a proxy. A block that wants an account cannot be talked around, and it says
so rather than retrying for an hour.

The engine is [`lncrawl-scraper`](https://github.com/lncrawl/scraper), built and measured
against a corpus of the source hosts in this repository. **What it will not do:** log into a
site for you, pay a captcha-solving service, or pretend a site is reachable when it is not.

## Supported sources

The generated index — every source, its crawler, what it supports and who maintains it — lives
in **[SOURCES.md](https://github.com/lncrawl/lightnovel-crawler/blob/master/SOURCES.md)**,
together with the domains that were rejected and why.

To request a site,
[open an issue](https://github.com/lncrawl/lightnovel-crawler/issues/new/choose). To add one
yourself: a source is a single file under
[sources/](https://github.com/lncrawl/lightnovel-crawler/tree/master/sources)`/<lang>/<letter>/`,
and if the site runs a CMS one of the shared templates in
[lncrawl/templates/](https://github.com/lncrawl/lightnovel-crawler/tree/master/lncrawl/templates)
already knows how to crawl it — those sources are about ten lines. Everything else declares a
handful of selectors. The walkthrough is in
[CONTRIBUTING.md](https://github.com/lncrawl/lightnovel-crawler/blob/master/.github/CONTRIBUTING.md#adding-a-source-crawler).

## Contributing

Bug fixes, new sources and documentation are all welcome.

```bash
git clone https://github.com/lncrawl/lightnovel-crawler.git
cd lightnovel-crawler
make install   # installs uv, syncs every dependency
make start     # runs the server
make lint      # pyright + ruff — run this before opening a PR
```

<details>
<summary>The rest of the Makefile</summary>

```bash
# setup
make setup            # install uv
make install          # setup + uv sync (the default target)
make sync             # uv sync only
make upgrade          # setup + uv sync --upgrade

# dev
make start            # run the server
make dev              # run with auto-reload (alias: make watch)
make lint             # pyright + ruff format --check + ruff check
make lint-fix         # ruff check --fix + ruff format
make index-gen        # regenerate the source index and SOURCES.md
make check-sources    # probe every source base URL over HTTP

# version — writes lncrawl/VERSION
make patch / minor / major

# build
make build            # wheel + executable + installer
make build-wheel
make build-exe

# dependencies
make add-dep <pkg>    # add a runtime dependency
make add-dev <pkg>    # add a dev dependency
make rm-dep <pkg>
make rm-dev <pkg>

# docker
make docker-build
make docker-up
make docker-down
make docker-logs

# misc
make clean            # remove .venv, build artifacts, caches
make version          # print the current version
```

</details>

The web UI is a separate repository,
[lncrawl-web](https://github.com/lncrawl/lncrawl-web); its build is synced into this one by CI.
[AGENTS.md](https://github.com/lncrawl/lightnovel-crawler/blob/master/AGENTS.md) holds the
architecture and the invariants that break silently, and
[CONTRIBUTING.md](https://github.com/lncrawl/lightnovel-crawler/blob/master/.github/CONTRIBUTING.md)
covers setup, code style and how to add a source.
[CI on forks](https://github.com/lncrawl/lightnovel-crawler/blob/master/.github/FORKING.md) ·
[DeepWiki overview](https://deepwiki.com/lncrawl/lightnovel-crawler)

![Repobeats](https://repobeats.axiom.co/api/embed/ecf3e93f676c27ba6404315e22523033d50aab45.svg "Repobeats analytics image")

## Get help

Questions, tips and showing off your setup:
[GitHub Discussions](https://github.com/lncrawl/lightnovel-crawler/discussions). Bugs and
source requests: [issues](https://github.com/lncrawl/lightnovel-crawler/issues/new/choose).
Licensed under [GPLv3](https://github.com/lncrawl/lightnovel-crawler/blob/master/LICENSE).
