Metadata-Version: 2.4
Name: jgtpricedb-util
Version: 0.1.0
Summary: Run-to-completion jobs for the jgtpricedb price store: bootstrap, refresh, freshness, relabel probe
License: MIT
Keywords: forex,jgt,oanda,price-data,trading
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Office/Business :: Financial :: Investment
Requires-Python: >=3.10
Requires-Dist: jgtpricedb>=0.1.1
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: postgres
Requires-Dist: jgtpricedb[postgres]>=0.1.1; extra == 'postgres'
Description-Content-Type: text/markdown

# jgtpricedb-util

Run-to-completion jobs for a [`jgtpricedb`](https://pypi.org/project/jgtpricedb/)
price store.

`jgtpricedb` is **the store** — schema, bar identity, the trading-session period
grid, and the anchor-based refresh engine that writes only what moved.
`jgtpricedb-util` is **the steward** — the handful of things an operator
actually runs against that store, each one a job that starts, does a stated
amount of work, prints one greppable verdict, and stops.

Nothing here is a daemon. `light` counts its ticks and exits. `freshness` and
`probe-relabel` measure and exit. `bootstrap` and `refresh` run one pass. That is
what makes the same image safe from cron, from a compose one-shot, and from a
person's hands at six in the morning: **we run it, it stops when it is done, we
don't delete it.**

## The verdict line

Every job's last line of stdout is:

```
JGTPDB <job> OK|FAIL <summary>
```

and the exit code agrees with it:

| code | meaning |
|---|---|
| `0` | OK — the job ran and measured no failure |
| `1` | FAIL — the job ran and measured a failure |
| `2` | refused — the job could not be performed (bad arguments, absent directory, unreadable file, broker unreachable, or an abort part-way through) |

`1` and `2` are separate because "this data is wrong" and "I could not look"
call for different responses, and a caller that retries the first is wasting its
time. Grep the line, or read `$?`; they never disagree, because both come from
the same object.

The line is printed on **every** exit, including the ones nobody plans: a usage
error, an unreadable file, an unexpected exception. The traceback goes to
stderr where a person can read it; stdout still ends with the line, because a
caller that greps for it and finds nothing cannot tell a crashed job from one
that has not finished yet.

## The jobs

| job | what it does |
|---|---|
| `bootstrap` | Registers every named series from a holdings directory and runs one refresh pass. How a store first learns what exists. |
| `refresh` | One delta pass over series the store already knows. Registers nothing new. `--oanda` fetches candles into the source directory first. |
| `light` | Bounded forming-bar loop: `--loop N --interval S`, then exit. Rewrites the anchor when its values move; never appends. |
| `oanda-fetch` | OANDA v20 candles into the exact 15-column jgt CSV, at the filename `CsvSource` reads. |
| `freshness` | Does the directory a server *serves* hold the bars the writer *wrote*? Inode identity, newest-bar drift, per-timeframe staleness. |
| `probe-relabel` | Does the period grid rename the broker's bars? Any timeframe above 0% exits 1. |
| `snapshot` | Tar the paths a run is about to touch, before it touches them. Prints the archive's sha256. |
| `verify` | Byte-compare two files; names the first offset that differs. |

### `bootstrap` and `refresh` are one engine, two authorities

They share a body because `jgtpricedb` shares one: `refresh_series` reads what
the store already holds for the periods a source offers and picks its own
algorithm — bootstrap when the range is empty, incremental update when it is not.
Forcing that choice from outside would be guessing at state the store can simply
read.

What differs is **where the list of series comes from**, and that is the whole
difference the two names carry:

- `bootstrap --all-series` means *every `<INSTRUMENT>_<TF>.csv` in the holdings
  directory*, and registers them.
- `refresh --all-series` means *every series already registered in the store*,
  and registers nothing. A CSV that appears in the source directory unannounced
  is not silently adopted mid-week.

### `light` — bounded, and quiet when there is nothing to move

`--loop N` is a count of ticks, not a duration, and there is no `--forever`.
The job spends its count and exits, which is what makes the same image safe from
a timer, where the restart is somebody's stated policy rather than a while-loop's
accident.

Each tick finds the series' anchor and rewrites it if the values moved. It never
appends. Two things are therefore normal rather than wrong, and the verdict line
counts them apart from errors:

- **`deferred=`** — the series has no forming bar, so a full refresh owns it.
  Over a weekend every series is here. A run of nothing but deferrals still
  exits `0`, because a job that failed every Saturday would be a job nobody
  reads on Monday.
- **`rollovers=`** — the period advanced while the loop was running. The append
  belongs to `refresh`, so the tick says so instead of writing a second forming
  bar.

`errors=` is what fails the run, and `divergences=` fails it only under
`--strict`.

### `freshness` — the job today's incident is made of

A writer container and a reader container each mounted a **different host
directory at the same container path**, because a stray shell variable was set
when one of them was launched. Both processes were healthy. Both logs were clean.
The writer wrote current bars all day and the reader served bars from days
earlier, and nothing inside either container could tell, because from inside,
`/data/current/pds` is `/data/current/pds`.

Three questions catch it, and no one of them is enough alone:

1. **Identity** — `st_dev`/`st_ino`. Same inode: one file, no split possible.
   Different inode: not yet a fault, a copy is legitimate — but now the other two
   questions carry the weight.
2. **Drift** — does the served file's newest bar match the expected one's? This is
   the direct signature: two files, each internally consistent, describing
   different moments. One period of slack, because the writer may legitimately
   be a bar ahead of a file the server copied a moment ago — which is also this
   check's honest limit: a split whose two sides have not yet parted by a whole
   period is invisible to it, and the alternative is failing every server that
   serves from a copy.
3. **Staleness** — is the newest bar older than this timeframe's budget? This
   catches what the first two cannot: both paths being the *same* stale
   directory, where nothing diverges because nothing is being written.

The weekend is not an outage. Forex closes Friday 17:00 New York and opens Sunday
17:00, so a Saturday probe correctly finds every file hours old. The store's own
`ForexCalendar` is consulted and an age is **minutes of open market** — the
closure is not counted, because no bar could have been written in it.

Not counting it only over the weekend is not enough, and the difference is a
whole class of false alarm: holding the clock still until Sunday 17:00 and then
releasing it puts a 48-hour cliff at the reopen, where a file holding Friday's
last bar reads one minute old all weekend and 2 881 minutes old the instant the
market opens — every series on the feed failing at once, every week, with
nothing wrong and no bar missed. Elapsed open time has no such edge, and it
equals the wall clock whenever the market never shut in between.

Defaults are roughly three periods plus slack per timeframe, overridable:

```bash
jgtpdb freshness --served /data/current/pds --expected /writer/pds \
                 --max-age-minutes H1=120,m15=40
```

### `probe-relabel` — the measurement that produced `jgtpricedb` 0.1.1

A store cut on naive UTC renamed **100%** of the H4, D1, W1 and M1 bars it was
given, because the feed cuts those periods on the 17:00 America/New_York session
boundary. It renamed them silently: the bar kept its prices, `period_start` moved
its timestamp, the key came from the moved timestamp, and no exception was raised
anywhere.

The probe is one line of arithmetic run over real files, using the library's own
grid:

```
period_start(timeframe, ts, session_for(instrument)) == ts
```

Run it against holdings rather than a fixture. A fixture encodes what the grid
believed on the day it was written; the CSVs encode what the broker actually
published, which is the only authority there is.

### `oanda-fetch` — session alignment is not optional

OANDA aligns `D`, `W`, `M` and `H4` candles to `dailyAlignment` in
`alignmentTimezone`, and the default is 00:00 UTC — *not* the feed's boundary.
Left at the default, every session-anchored bar would land an offset away from
the holdings and the store would key it as a different bar. This job sends
`dailyAlignment=17`, `alignmentTimezone=America/New_York` and
`weeklyAlignment=Saturday` for exactly those timeframes, all three read off
`jgtpricedb`'s own `NY_TRADING_SESSION` rather than restated here.

**`weeklyAlignment` is `Saturday`, and the reason is worth a paragraph.** The
trading week is *labelled* Sunday — that is `session.week_opens_on` — but the
boundary that opens it is the evening before, Saturday 17:00 New York. OANDA's
parameter names the boundary's weekday, not the label's. Asking for `Sunday`
returns a weekly candle stamped one day late: measured against the live
practice API, every W1 bar of both a summer and a winter window landed off
`period_start`'s grid. The store recovers, because identity floors a stamp
before it keys anything — which is exactly what made it silent. The CSV on disk
disagreed with the holdings for the same week, and `probe-relabel` over that
directory reported 100% on W1, a defect belonging to the fetch rather than to
the grid. `tests/test_session_grid.py` keeps both responses, correct and wrong,
as captured evidence.

`Median` has no OANDA field, so it is **derived** — from bid and ask, the
authoritative sides, by the store's own formula. Not from OANDA's mid candle:
that candle is built from mid ticks rather than averaged from its own bid and
ask, so its `High` and `Low` genuinely disagree with `(bid + ask) / 2`. Those
cells are written through as what the broker published and `CsvSource` reports
them as validate divergences, which is the design — the store keeps its own
derivation and says where the source differed. A Median computed off that mid
would inherit the same disagreement and report it against a column that was
never the problem; over 149 captured live candles that was six false reports.

An `--oanda` refresh therefore prints a handful of `divergences=` on the mid
columns as a matter of course. That is a finding about the broker's mid stream,
not a failed run — and `--strict`, which turns findings fatal, will fail such a
run on purpose.

## Quickstart — pip

```bash
pip install jgtpricedb-util

# fill a store from the existing holdings
jgtpdb bootstrap --db /tmp/prices.db --holdings $JGTPY_DATA/pds \
                 --instrument EUR-USD --timeframe H1

# ask the broker for today, then take the delta
export OANDA_TOKEN=...            # practice by default
jgtpdb oanda-fetch --instrument EUR-USD --timeframe H1 \
                   --since 2026-07-29 --out /tmp/oanda
jgtpdb refresh --db /tmp/prices.db --holdings /tmp/oanda \
               --instrument EUR-USD --timeframe H1

# watch the forming bar move, ten times, then stop
jgtpdb light --db /tmp/prices.db --holdings /tmp/oanda \
             --instrument EUR-USD --timeframe H1 --loop 10 --interval 60 --oanda
```

## Quickstart — container

The image is a job container: `restart: "no"`, one job per invocation.

```bash
cd jgt-pricedb-util

# build from PyPI (the normal path)
docker compose build

# run a job — the entrypoint is jgtpdb, so the command is just the job
docker compose run --rm jgt-pricedb-util probe-relabel --holdings /data/current/pds
docker compose run --rm jgt-pricedb-util bootstrap --db /work/prices.db \
    --holdings /data/current/pds --instrument EUR-USD --timeframe H1
docker compose run --rm jgt-pricedb-util freshness \
    --served /data/current/pds --expected /data/current/pds
```

To run **this checkout** instead of the published release — before a release
exists, or to prove a change — export both variables before building, and leave
them exported for the run:

```bash
export JGT_PRICEDB_UTIL_LOCAL=1 JGT_PRICEDB_UTIL_TAG=local
docker compose build
docker compose run --rm jgt-pricedb-util probe-relabel --holdings /data/current/pds
```

Both, not one. `LOCAL` decides what gets installed and `TAG` decides what the
image is called; setting only the first builds this checkout under the released
version's name, and setting only the second is a rename of the PyPI build.
Exporting them once covers the build and every run after it — forget them on
the `run` and compose quietly reaches for the released image instead.

The plain-docker equivalent, if you would rather not export anything:

```bash
docker build --build-arg LOCAL=1 -t jgtpricedb-util:local .
docker run --rm -v "$JGTPY_DATA:/data/current:ro" -v "$PWD/work:/work" \
    jgtpricedb-util:local probe-relabel --holdings /data/current/pds
```

Mounts, as composed:

| container path | host | mode |
|---|---|---|
| `/data/current` | `${JGTPY_DATA:-/b/trading/jgtml/data/current}` | read-only |
| `/work` | `./work` | read-write — the store, fetched candles, snapshots |

`/data/current` is mounted **read-only** deliberately. This package reads the
holdings and writes to `/work`; a job that could rewrite the pipeline's own
output is a job that can cause the incident `freshness` exists to detect.

`OANDA_TOKEN`, `OANDA_ACCOUNT_ID` and `OANDA_ENVIRONMENT` pass through from the
host environment. The environment defaults to `practice` when unset, in this tool
as in every other jgt tool.

## Dependencies

`jgtpricedb`, and otherwise the standard library. No `requests` — `urllib`. No
`pandas` in this package's own code — `csv`. A job container that pulls a second
dependency tree is a job container that one day fails for a reason that has
nothing to do with the data.

`jgtpricedb` itself brings `sqlalchemy` and `pandas`, so the image contains
them; what this package adds on top is nothing.

## Related

- [`jgtpricedb`](https://pypi.org/project/jgtpricedb/) — the store this stewards.
- The session-grid decision, its evidence and its measurement live in
  `jgtpricedb`'s `rispecs/01-price-store.spec.md`, *The Session Grid*.
