Metadata-Version: 2.4
Name: novel-cli
Version: 2.0.0
Summary: Web novel scraper and library manager with Rich UI
Author-email: Your Name <your@email.com>
License: MIT
Project-URL: Homepage, https://github.com/YOUR_USERNAME/novel-cli
Project-URL: Repository, https://github.com/YOUR_USERNAME/novel-cli
Project-URL: Issues, https://github.com/YOUR_USERNAME/novel-cli/issues
Keywords: novel,scraper,cli,royalroad,novelfull,web-novel
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: End Users/Desktop
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.28
Requires-Dist: beautifulsoup4>=4.11
Requires-Dist: markdownify>=0.11
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.9
Requires-Dist: loguru>=0.7
Requires-Dist: tenacity>=8.0
Requires-Dist: cloudscraper>=1.2
Requires-Dist: selenium>=4.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Requires-Dist: pytest-cov>=4.0; extra == "test"
Dynamic: license-file

# novel-cli

A CLI tool for scraping and managing web novels offline, with an optional web viewer.

## Features

- **Smart Scraping**: Automatically skips completed novels with all chapters already downloaded, saving time and bandwidth
- **Offline Reading**: Download and store complete novels locally
- **Web Viewer**: Optional web-based reader for your library
- **Multiple Sources**: Support for various novel hosting sites ( Only tested on one site - RR)
- **Library Management**: Add, remove, export, and organize your collection
- **Progress Tracking**: Monitor download progress and completion status

## Installation

### From Source

```bash
git clone https://github.com/Azizgic/novel-cli.git
cd novel-cli
pip install -e .
```

### From PyPI

```bash
pip install novel-cli
```

## Quick Start

```bash
# Add a novel to your reading list
novel add https://www.royalroad.com/fiction/12345/some-novel

# Scrape all novels (smart mode - skips completed ones)
novel scrape

# Scrape a single novel
novel scrape "Novel Title"

# Update stats only (no chapter download)
novel update-stats

# List your library
novel list

# Show novel details
novel show "Novel Title"

# Remove a novel
novel remove "Novel Title"

# Export library
novel export json
novel export csv

# Show statistics
novel stats

# Clean up orphaned data
novel clean
```

## Command Reference

| Command | Description |
|---------|-------------|
| `novel add <url>` | Add a novel URL to your reading list |
| `novel scrape [title]` | Download chapters (skips completed novels by default) |
| `novel scrape --dry-run` | Preview what would be scraped without downloading |
| `novel update-stats [title]` | Update metadata without downloading chapters |
| `novel list` | Display all novels in your library |
| `novel show [title]` | Show detailed information about a novel |
| `novel remove [title]` | Remove a novel from your library |
| `novel export <json\|csv>` | Export your library to JSON or CSV |
| `novel stats` | Display library statistics |
| `novel clean` | Remove orphaned data and thumbnails |
| `novel read` | Start the web viewer |

## Web Viewer (Optional)

```bash
cd viewer && pnpm install
novel read
# Opens at http://localhost:4173
```

## Data Storage

All your data is stored locally in `~/.novel-library/`:
- `library.db` - SQLite database with chapters and metadata
- `urls.txt` - Your reading list
- `thumbnails/` - Cover images
- `library.json` - Configuration file

## Smart Scraping

The tool intelligently avoids rescraping novels that are already complete. A novel is skipped if:
- Its status is marked as "Completed"
- AND all chapters have already been downloaded

This saves time and bandwidth during regular scraping sessions. Other statuses (Ongoing, Hiatus, Dropped, etc.) are always scraped to catch updates.

## Testing

The project includes comprehensive automated tests to ensure reliability.

### Running Tests

```bash
# Install test dependencies
pip install pytest pytest-cov

# Run all tests
pytest tests/

# Run with verbose output
pytest tests/ -v

# Generate coverage report
pytest tests/ --cov=novel --cov-report=html
```

See [`tests/README.md`](tests/README.md) for detailed testing documentation.

## Recent Changes

### Smart Skip Feature (v0.1.0)
- Added `ScrapeChecker` module to prevent rescraping completed novels
- Reduces unnecessary downloads and improves performance
- Fully automated testing with 30+ test cases
- See [`docs/ENHANCEMENT_SKIP_COMPLETED.md`](docs/ENHANCEMENT_SKIP_COMPLETED.md) for details

## Development

### Project Structure

```
novel-cli/
├── novel/
│   ├── cli.py              # Command-line interface
│   ├── manage.py           # Library management
│   ├── paths.py            # Data storage paths
│   └── scraper/
│       ├── novel_scraper_indexer.py   # Main scraper
│       ├── scrape_checker.py          # Skip logic
│       ├── writers.py      # Database writer
│       └── base_scraper.py # Base scraper class
├── tests/
│   └── test_scrape_checker.py  # Test suite
├── examples/
│   └── integration_example.py   # Integration examples
└── viewer/                      # Web viewer (Svelte)
```

### Contributing

Contributions are welcome! Please ensure:
1. All tests pass: `pytest tests/`
2. Code follows the existing style
3. New features include test coverage

## Dependencies

- `requests` - HTTP requests
- `beautifulsoup4` - HTML parsing
- `cloudscraper` - Handle CloudFlare protection
- `selenium` - Browser automation (if needed)
- `markdownify` - Convert HTML to Markdown
- `lxml` - XML/HTML processing

For development:
- `pytest` - Testing framework
- `pytest-cov` - Coverage reports

## AI Use
Ai has been used to create a portion of this code.
Specifically to write the tests, and other minor improvements.
AI has also been used to improve readability and add helpful code comments.

## License

MIT License - see LICENSE file for details

## Support

Found a bug or have a suggestion? Open an issue on GitHub:
https://github.com/Azizgic/novel-cli/issues

---

**Made with ❤️ for novel readers**
