Metadata-Version: 2.4
Name: jing
Version: 0.6.0
Summary: Download stock data and perform data analisys
Author-email: sai <dawangfei@gmail.com>
License: MIT
Keywords: jing,sai,stock
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
Requires-Dist: requests
Requires-Dist: pandas
Requires-Dist: baostock
Requires-Dist: yfinance
Requires-Dist: akshare
Requires-Dist: pyarrow
Requires-Dist: importlib-metadata; python_version < "3.10"
Provides-Extra: cos
Requires-Dist: cos-python-sdk-v5; extra == "cos"
Provides-Extra: s3
Requires-Dist: boto3; extra == "s3"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"

# jing

`jing` downloads and organizes stock market data. The examples below use China A-share stocks with BaoStock.

## Install

```bash
pip install jing
```

## Download CN data

BaoStock is the default data source for CN. Stock codes must include an exchange prefix, such as `sh.` or `sz.`.

```python
import jing

d = jing.D("cn")
d.download("sz.000807")  # Download one stock
d.download()             # Download the stocks in the default list
```

Batch downloads use `~/data/jing/list/cn_baostock.txt` by default. On first use, `jing` copies the bundled list there; edit this copy to maintain your stock list.

## Download US data

US stocks use Yahoo Finance. Yahoo blocks many datacenter and mainland-China
IPs with 403/429, so on such hosts route the requests through a proxy:

```bash
export https_proxy=http://127.0.0.1:7890 http_proxy=http://127.0.0.1:7890 all_proxy=socks5://127.0.0.1:7890
```

```python
import jing

d = jing.D("us")
d.download("AAPL")   # one stock
d.download()         # batch from ~/data/jing/list/us.txt
```

Batch runs resume per stock: each CSV is the ledger, so a re-run after an
interruption only fetches what is still missing.

## File locations

The default data directory is `~/data/jing`:

```text
~/data/jing/
├── list/cn_baostock.txt
└── raw/baostock/cn/sz.000807.csv
```

Set `JING_DATA` to use another directory:

```bash
export JING_DATA=/data/stocks
```

Or set the data directory from Python:

```python
from jing.data_paths import set_data_root

set_data_root("/data/stocks")
```

The stock list and CSV files will then be stored under `/data/stocks/list/` and `/data/stocks/raw/baostock/cn/`.

## Upload to COS (optional)

Uploading to Tencent COS is opt-in — most users have no COS bucket, and nothing
in the library touches COS unless you ask it to.

```bash
pip install jing[cos]        # the Tencent SDK is not installed by default
export JING_SYNC_COS=1       # upload each market after a batch download
```

Credentials come from a flat `~/.cos.yaml` holding only the secrets:

```yaml
cos_secret_id: <SecretId>
cos_secret_key: <SecretKey>
```

Or drive the uploader directly, which is also the only way to feed it
credentials from somewhere else (an object built with your own client):

```python
from jing import COSUploader

COSUploader(markets=["cn"]).upload()   # python -m jing.uploader_cos also works
```

A local file never overwrites a better copy on COS: if it is less than half the
remote size, or its last date is older, it is skipped and listed in
`alerts/alert-<date>.txt` on COS. `JING_COS_SHRINK_GUARD` tunes the ratio and
`0` turns the guards off.

After the run, each market also gets a manifest at `meta/<market>.json` on
COS: `generated_at` plus the key, byte size and last trading date of every
file whose COS copy is known to be identical to the local one. A consumer
syncing from COS can read the manifest instead of listing objects and probing
tails. Files blocked by the guards, or whose upload failed, are deliberately
left out, so the consumer probes those — and any missing manifest — itself.

## Run daily on a server

`deploy/` holds the cron entry point. It is a deployment script, not part of the
`jing` package: `jing` only downloads and uploads, `deploy/daily.py` schedules
that.

```bash
sudo mkdir -p /opt/jing /data/jing /var/log/jing
sudo git clone <this repo> /opt/jing
cd /opt/jing && python3 -m venv .venv && .venv/bin/pip install -e ".[cos]"

sudo install -m 600 deploy/env.example /etc/jing/env
sudo $EDITOR /etc/jing/env          # fill in JING_COS_SECRET_ID / _KEY

.venv/bin/python deploy/daily.py us   # smoke test on one market
crontab deploy/jing.cron              # weekdays 20:00 Asia/Shanghai
```

`run_daily.sh` loads `/etc/jing/env` (cron has no login shell), takes a `flock`
so a slow run cannot overlap the next one, and writes `/var/log/jing/<date>.log`
(kept 14 days).

Check on it any time with the status command — no network needed unless you add
`--cos`:

```bash
python3 deploy/status.py          # running? how far along? data up to which day?
python3 deploy/status.py --cos    # also count the objects on COS
```

## Catch up from COS before downloading

`deploy/cos_sync.py` reconciles the bucket against the local CSVs and pulls the
newer copy of each file, so a machine that fell behind — or a fresh one that
would otherwise spend days backfilling history — starts from what the bucket
already has.

```bash
python3 deploy/cos_sync.py --dry-run    # report only, write nothing
python3 deploy/cos_sync.py              # cn + us + hk
python3 deploy/cos_sync.py cn hk        # only these markets
```

`JING_COS_SYNC=1` makes `deploy/daily.py` do this automatically before each
market's download, so the resume logic decides what to fetch against current
data. It is **off by default**: the server is normally the bucket's source (it
downloads, then uploads), so the reconcile usually just reports that there is
nothing to pull. Turn it on after wiping the disk, on a new machine, or when
another host is ahead.

Which copy wins is decided per file from the **last trading date** in each:

| Local copy | Bucket copy | Result |
| --- | --- | --- |
| missing | anything | pulled |
| less than half the bucket's size | anything | pulled (local one is truncated) |
| older date | newer date | **pulled** |
| no data rows | newer date | pulled |
| same or newer date | — | left alone |
| — | date could not be read | left alone, counted as "无法比较" |

A sync never moves a market backwards. Each market reports its own state first:

```text
us 本地最新 2026-06-05，COS 最新 2026-10-02，待拉 1516 跳过 0 无法比较 0
```

Deciding costs about 2KB per file (`jing.resume.TAIL_BYTES`), not a download —
the bucket's last date comes from a ranged `GET` of the object's tail and the
size from the listing, so there is no `HEAD` per file. Only the files that win
are fetched whole. `JING_COS_SHRINK_GUARD` tunes the truncation ratio for this
direction too.

This lives in `deploy/` rather than the library: it is a deployment policy, not
a capability of `jing`. Like the uploader, credentials come from
`/etc/jing/env`, not `~/.cos.yaml`.

Notes:

- **Interrupted runs resume.** Each stock's CSV is the ledger, so the next run
  only downloads what is missing. Running out of the time budget is a normal
  outcome, not a failure.
- **Concurrency**: `JING_MAX_WORKERS` overrides the built-in worker count (3
  for BaoStock, 10 otherwise). The server sets it to 1 — one request at a
  time from one IP is the cheapest insurance against being rate limited.
  `D.download(max_workers=N)` overrides it per call.
- **akshare timeouts**: akshare's requests carry no timeout, so `jing` patches
  `requests.sessions.Session.request` to inject one (`JING_HTTP_TIMEOUT`,
  default 30s, 0 disables). `socket.setdefaulttimeout()` cannot do this —
  urllib3 passes its own `None` down to `sock.settimeout()`, overriding it.
- **cn uses BaoStock only.** `cn_ak` (AKShare) covers the same A-shares;
  running both every day doubles the request load for no new data.
