Metadata-Version: 2.4
Name: orlen-price-scraper
Version: 0.1.1
Summary: Polite, resumable scraper for public ORLEN wholesale fuel prices.
Author: solek13
Project-URL: Homepage, https://github.com/solek13/orlen
Project-URL: Repository, https://github.com/solek13/orlen
Project-URL: Issues, https://github.com/solek13/orlen/issues
Keywords: orlen,fuel-prices,scraper,csv
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"

# ORLEN wholesale fuel-price scraper

This repository downloads the public price data shown on ORLEN's
[Hurtowe ceny paliw](https://www.orlen.pl/pl/dla-biznesu/hurtowe-ceny-paliw)
page. It covers every section currently visible there:

- motor fuels (all products returned by the API),
- wholesale autogas/LPG (all 16 regions),
- heavy fuel oil,
- aviation fuels (all locations).

It calls the page's JSON API sequentially. The default random 2–4 second gap,
24-hour disk cache, incremental CSV/state tracking, `Retry-After` handling,
exponential backoff, and atomic output make long runs polite and resumable. Do
not run many copies in parallel.

## Installation

From PyPI:

```powershell
python -m pip install orlen-price-scraper
```

Directly from GitHub:

```powershell
python -m pip install git+https://github.com/solek13/orlen.git
```

From a local checkout:

```powershell
python -m pip install -e .
```

After installation, the command-line tool is available as `orlen-prices`.

## Usage

Python 3.10+ is sufficient; there are no third-party dependencies.

```powershell
# After installation
orlen-prices --year 2026 --output data\prices.csv

# Current year (the default)
python .\scrape_orlen.py

# A particular year
python .\scrape_orlen.py --year 2025

# Everything exposed by the API (slowest, especially LPG)
python .\scrape_orlen.py --all

# January 1, 2013 through today (also accepts: --since 2013)
python .\scrape_orlen.py --from-year 2013

# Choose the destination
python .\scrape_orlen.py --year 2026 --output data\prices.csv
```

The default files are `data/orlen_prices_<year>.csv`,
`data/orlen_prices_since_<year>.csv`, and `data/orlen_prices_all.csv`. Each CSV
has a small `.csv.state.json` sidecar. It
records the date range already checked for every product, including products
which returned no rows. A later run loads and merges the existing CSV, requests
only dates after the recorded coverage, and skips LPG effective dates already
present. The state is bound to a hash of the CSV so deleting or modifying the
data cannot silently cause ranges to be skipped.

Product lists and LPG date indexes are checked through the 24-hour cache so new
products/dates can still be discovered without repeatedly downloading price
history. Use `--refresh` only when those missing-range requests really need to
bypass the cache. An interrupted run can simply be started again; completed
requests remain in the response cache.

The all-history run deliberately stays single-threaded. Regular, heavy-oil,
and aviation history need only a few range requests. ORLEN exposes LPG history
one effective date at a time, so that part can take several minutes. Progress is
printed to stderr.

The CSV contains category/product metadata, effective and publication dates,
price, frontend display unit, location (where applicable), exact source URL,
and fetch timestamp. ORLEN's API currently sends a null unit; `unit` is filled
from the units displayed in the corresponding ORLEN frontend tables.

## Library functions

```python
from scrape_orlen import scrape_all, scrape_since, scrape_year, update_csv, write_csv

rows = scrape_year()          # current year
# rows = scrape_all()         # complete available history
# rows = scrape_since(2013)   # 2013 through today
write_csv(rows, "prices.csv")

# Preferred for scheduled runs: merges existing rows and downloads only gaps.
rows = update_csv("prices.csv")
```

## Tests

```powershell
python -m unittest -v
```

The tests use a fake API and do not contact ORLEN.

## Packaging

Build the source distribution and wheel locally:

```powershell
python -m pip install -e .[dev]
python -m build
python -m twine check dist/*
```

Installing by package name, for example `python -m pip install orlen-price-scraper`,
requires uploading a release to PyPI first.
