Metadata-Version: 2.4
Name: qchancrawler
Version: 0.2.0
Summary: Crawl posts from QQ channels to text files
Author-email: Ivy Xu <ivyxu1337@gmail.com>
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: aiosqlite
Requires-Dist: httpx

# QChanCrawler

QChanCrawler retrieves posts from QQ channels and saves them to a SQLite
database. It can crawl channels supplied explicitly or discover channels from
the Explore recommendations.

## Requirements

- Python 3.12 or later.

## Installation

```console
pip install qchancrawler
```

## Usage

### Crawl specified channels

```console
qchancrawler -g ID1 ID2 ID3
```

The channel identifiers may be numeric guild IDs or QQ channel numbers. More
than one identifier can be supplied; the channels are crawled concurrently.

### Crawl recommended channels

```console
qchancrawler -e
```

`-e` discovers channels from the explore section and crawls them. If
no number is supplied, it requests 200 channels. You can also supply with
`inf` or `infinite` to crawl infinitely from exploration.

### Examples

```console
qchancrawler -g 123456
qchancrawler -g 123456 Mine101 -o posts.db
qchancrawler -e -c cookies.txt -v
qchancrawler -e inf
```

Options:

- `-g`, `--guild`: QQ channel/guild IDs to crawl.
- `-e`, `--explore`: Discover and crawl recommended channels. The optional count defaults to 200.
- `-c`, `--cookie`: Path to an exported `pd.qq.com` cookie file. Optional for posts, but required to crawl comments and replies.
- `-o`, `--outfile`: SQLite database path. Defaults to `./qchan.db`.
- `-p`, `--page`: Starting page number. Crawling continues until the API reports that there are no more pages. Defaults to `1`.
- `-v`, `--verbose`: Enable verbose progress output.
- `--version`: Show the installed QChanCrawler version.

## Database

The `posts` table contains these fields:

- `id`: Auto-incrementing SQLite row ID.
- `guild_id`: QQ guild/channel identifier.
- `guild_name`: QQ guild/channel name.
- `item_id`: Unique ID of the post, comment, or reply. Rows are deduplicated by `item_id`.
- `item_type`: One of `post`, `comment`, or `reply`.
- `published_at`: Item publication time as a Unix timestamp.
- `author_name`: Author nickname.
- `author_id`: Author identifier.
- `author_location`: Author province.
- `icon_url`: Author avatar URL.
- `text`: Item content.
- `post_url`: URL of the source post. Comments and replies use the URL of their parent post.

### Item relationships

```text
Post
|
|-- Comment
|   |
|   |-- Reply
```

Each item has its own `item_id`. Comments and replies are associated with the
parent post through `post_url`.

## Cookie file

The cookie file must use the [Mozilla/Netscape `cookies.txt` format](https://curl.se/rfc/cookie_spec.html).
Browser extensions and other cookie-export tools can create files in this
format.

Cookies are optional when crawling posts, but are required for crawling comments and replies.
It's also recommended for crawling exploration, because the exploration section will be duplicated
in log-out status.

The cookie file must contain a pair of valid `p_uin` and `p_skey`. 

## Disclaimer

Use QChanCrawler only for channels and data you are authorized to access.
Protect personal data and exported cookies, avoid excessive requests, respect
service rate limits, and comply with QQ's terms and policies.
