Metadata-Version: 2.4
Name: worldbank-commodities
Version: 0.5.0
Summary: Python wrapper to extract World Bank Pink Sheet commodity price data (oil, gas, metals, agriculture).
Project-URL: Homepage, https://github.com/hmorao95/worldbank-commodities
Project-URL: Repository, https://github.com/hmorao95/worldbank-commodities
Project-URL: Data source, https://www.worldbank.org/en/research/commodity-markets
Author-email: Hugo Morão <hfmorao@gmail.com>
License: MIT
License-File: LICENSE
Keywords: commodities,energy,oil,pink-sheet,prices,world-bank
Requires-Python: >=3.10
Requires-Dist: fire>=0.6
Requires-Dist: openpyxl>=3.1
Requires-Dist: pandas>=2.2
Requires-Dist: requests>=2.28
Description-Content-Type: text/markdown

# worldbank-commodities

[![CI](https://github.com/hmorao95/worldbank-commodities/actions/workflows/ci.yml/badge.svg)](https://github.com/hmorao95/worldbank-commodities/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Checked with mypy](https://img.shields.io/badge/mypy-checked-2a6db2.svg)](https://mypy-lang.org/)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
[![codecov](https://codecov.io/gh/hmorao95/worldbank-commodities/graph/badge.svg)](https://codecov.io/gh/hmorao95/worldbank-commodities)
[![Keep a Changelog](https://img.shields.io/badge/changelog-Keep%20a%20Changelog-orange.svg)](CHANGELOG.md)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

A small, dependency-light Python wrapper around the World Bank "Pink Sheet"
(Commodity Markets) price data.

The World Bank publishes monthly and annual commodity price series (crude oil,
natural gas, coal, metals, fertilizers, and many agricultural products) as Excel
workbooks whose download URLs carry a hash that changes every month. This
wrapper handles that for you:

- Auto-discovers the current Monthly/Annual workbook URLs from the official
  [Commodity Markets page](https://www.worldbank.org/en/research/commodity-markets),
  so there are no hardcoded links that break every month.
- Parses the awkward multi-row-header sheets into tidy `pandas` DataFrames (long
  or wide), detecting the units row automatically.
- Caches the raw workbook on disk (24 h TTL) so repeat calls are fast.
- Exposes commodity metadata (name and unit) and filters by commodity name and
  date range.
- Cleans the World Bank's "not available" tokens (`…`, `..`) to `NaN`.

No API key required; the data is public.

## Coverage

71 monthly series back to 1960, updated monthly. Groups include:

| Group | Examples |
|---|---|
| Energy | Crude oil (average/Brent/Dubai/WTI), coal, natural gas (US/Europe/LNG Japan) |
| Metals & precious | Aluminum, copper, iron ore, nickel, zinc, tin, lead, gold, silver, platinum |
| Fertilizers | Phosphate rock, DAP, TSP, urea, potassium chloride |
| Agriculture | Grains, vegetable oils & oilseeds, cocoa/coffee/tea, meat, sugar, cotton, rubber, timber |

Series flagged `**` by the World Bank have methodology/source breaks; see the
`Description` sheet in the source workbook.

## Quickstart

New to Python tooling? These four steps take you from nothing to a spreadsheet
of commodity prices. You need [git](https://git-scm.com/downloads) and
[uv](https://docs.astral.sh/uv/) (a fast Python package manager) installed.

Install uv if you do not have it:

```bash
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
```

```powershell
# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```

Then:

```bash
# 1. Get the code
git clone https://github.com/hmorao95/worldbank-commodities.git
cd worldbank-commodities

# 2. Install it (uv creates a .venv and pulls dependencies; no manual Python setup)
uv sync

# 3. See what commodities are available
uv run worldbank-commodities list-commodities

# 4. Save every monthly price series to a CSV you can open in Excel
uv run worldbank-commodities to-csv prices.csv --freq monthly
```

That is it. `prices.csv` now holds one row per observation
(`date, mdates, commodity, price, unit`). `mdates` is a Stata-style monthly id
such as `1960m1`, handy for merging with Stata time series; it is added only for
the monthly frequency (in both long and wide output).

> Note: the `worldbank-commodities` command lives inside the project's `.venv`,
> which is why each command starts with `uv run`. If you skip `uv run` and get
> "command not found", that is the reason. To get a global command, see
> [Command line](#command-line) below.

To install the library into your own Python environment instead, use pip:

```bash
pip install .
```

Requires Python 3.10 or newer (uv installs a suitable Python for you if needed).

## Usage from Python

Once installed you can call the library from your own scripts. Run these with
`uv run python your_script.py` (or `uv run python` for an interactive session).

`get_prices()` returns a [pandas](https://pandas.pydata.org/) DataFrame, the
standard table type for data work in Python.

```python
from worldbank_commodities import WorldBankCommodities

wb = WorldBankCommodities()

# All 71 commodities + their units
wb.list_commodities()

# Tidy long DataFrame: date, mdates, commodity, price, unit
wb.get_prices(freq="monthly")

# Just crude oil + European gas since 2000, wide (one column per commodity)
wb.get_prices(
    freq="monthly",
    commodities=["Crude oil, Brent", "Natural gas, Europe"],
    start="2000-01",
    wide=True,
)

# Annual nominal or real (deflated) prices
wb.get_prices(freq="annual")  # nominal
wb.get_prices(freq="annual_real")  # real

# One-liner to CSV or Excel
wb.to_csv("brent.csv", freq="monthly", commodities=["Crude oil, Brent"], wide=True)
wb.to_excel("brent.xlsx", freq="monthly", commodities=["Crude oil, Brent"], wide=True)

# Incremental update: only append observations newer than what's already saved.
# First run writes the full history; later runs just tack on the new month(s).
wb.update_csv("commodities_monthly.csv", freq="monthly")
```

What the options mean:

- `freq` picks the dataset: `"monthly"`, `"annual"` (nominal prices), or
  `"annual_real"` (inflation-adjusted prices).
- `commodities` filters by name. Matching is case-insensitive and matches on any
  part of the name, so `commodities=["crude oil"]` returns all four crude-oil
  series. Leave it out to get everything.
- `start` and `end` limit the date range, for example `start="2000-01"`.
- `wide=False` (the default) gives one row per observation, which is easy to
  filter and plot. `wide=True` gives one column per commodity, which is handy
  for a side-by-side spreadsheet. Run `wb.list_commodities()` to see the exact
  names.

## Command line

The CLI is generated from the library with
[python-fire](https://github.com/google/python-fire): each method becomes a
subcommand and each parameter becomes a flag, so there is no separate set of
options to learn. Run `worldbank-commodities --help` to see them all.

Inside a uv project the console script lives in `.venv`, so call it with `uv run`
(or activate the venv first). `python -m worldbank_commodities` works too:

```bash
uv run worldbank-commodities list-commodities
```

To make the command available everywhere (outside this project, on your PATH),
install it as a uv tool once:

```bash
uv tool install .
worldbank-commodities list-commodities
```

The examples below use the bare command; prefix them with `uv run` if you have
not installed the tool globally.

```bash
# List every commodity and its unit
worldbank-commodities list-commodities

# Print a table to the terminal
worldbank-commodities get-prices --freq monthly

# Extract to a CSV
worldbank-commodities to-csv prices.csv --freq monthly

# Extract to an Excel workbook
worldbank-commodities to-excel prices.xlsx --freq monthly

# Filtered by one commodity (quote the name so the shell keeps it together)
worldbank-commodities get-prices --freq monthly \
    --commodities "Crude oil, Brent" --start 2010-01 --wide --out brent.csv

# Several commodities: pass a list. On Windows cmd wrap it in double quotes
# and use single quotes inside: --commodities "['Crude oil, Brent', 'Gold']"
worldbank-commodities get-prices --freq monthly \
    --commodities "['Crude oil, Brent', 'Natural gas, Europe']" --wide

# Incremental: add new and revised rows to an existing long CSV
worldbank-commodities update-csv commodities_monthly.csv --freq monthly

# Show what is missing from a CSV without writing it
worldbank-commodities new-observations commodities_monthly.csv --freq monthly
```

## Reference

Run `worldbank-commodities --help`, or `worldbank-commodities <command> --help`
for a single command, to see this from the CLI.

### CLI commands

| Command | Positional | Options |
|---|---|---|
| `list-commodities` | (none) | `--freq` |
| `get-prices` | (none) | `--freq --commodities --start --end --wide --out` |
| `to-csv` | `PATH` | `--freq --commodities --start --end --wide` |
| `to-excel` | `PATH` | `--freq --commodities --start --end --wide` |
| `update-csv` | `PATH` | `--freq --commodities --start --end` |
| `new-observations` | `EXISTING` | `--freq --commodities --start --end --include-revisions` |

For `get-prices`, `--out` writes CSV, or Excel when the path ends in `.xlsx`;
without `--out` it prints a table.

### Options

| Option | Values | Meaning |
|---|---|---|
| `--freq` | `monthly` (default), `annual`, `annual_real` | Dataset: nominal monthly, nominal annual, or real (deflated) annual. |
| `--commodities` | name or list of names | Case-insensitive substring filter. Omit for all. A single name is a plain string; several use a list (see Windows quoting above). |
| `--start`, `--end` | e.g. `2000-01`, `2000` | Inclusive date bounds. |
| `--wide` | flag | One column per commodity indexed by date (plus a leading `mdates` column for monthly). Default is long: one row per observation. |
| `--include-revisions` | flag (default on) | For `new-observations`: also return values the World Bank has since revised, not only unseen rows. |

### Python API

```python
WorldBankCommodities(cache_dir=None, cache_ttl_hours=24.0, timeout=60.0, session=None)
```

| Method | Returns |
|---|---|
| `list_commodities(freq="monthly")` | DataFrame of `commodity`, `unit` |
| `get_prices(freq="monthly", commodities=None, start=None, end=None, wide=False)` | Long or wide DataFrame |
| `to_csv(path, **kwargs)` | Writes CSV; returns the `Path` |
| `to_excel(path, **kwargs)` | Writes `.xlsx`; returns the `Path` |
| `update_csv(path, freq="monthly", commodities=None, start=None, end=None)` | Upserts the CSV; returns the added/revised rows |
| `new_observations(existing, freq="monthly", commodities=None, start=None, end=None, include_revisions=True)` | Rows missing from `existing` (DataFrame or CSV path) |

`to_csv`/`to_excel` forward `**kwargs` to `get_prices` (`freq`, `commodities`,
`start`, `end`, `wide`). The long output columns are `[date, commodity, price,
unit]`, plus `mdates` for the monthly frequency.

## Incremental updates

`update_csv()` (CLI: `update-csv`) keeps a long-format CSV current without
rewriting the whole file each run:

- First run (file missing) writes the full extract.
- Later runs read what's already saved and upsert on `(commodity, date)`: they
  append new observations and overwrite any values the World Bank has since
  revised. That covers newer months for commodities already present and
  commodities that were not extracted before.
- If nothing changed, the file is left untouched and an empty frame is returned.

```python
new_rows = wb.update_csv("commodities_monthly.csv", freq="monthly")
print(f"added {len(new_rows)} observations")
```

This works on the tidy long format (`[date, commodity, price, unit]`); wide CSVs
are not supported for incremental merging.

### Just the delta, in memory

If you keep your data somewhere other than a CSV, `new_observations()` is the
file-free building block. Give it what you already have (a long DataFrame or a
long-CSV path) and it returns only the rows you are missing: newer dates for
commodities you track, plus every row of commodities you have never extracted.

```python
have = my_store.load()  # any long DataFrame with [date, commodity, price]
missing = wb.new_observations(have, freq="monthly")
my_store.append(missing)

# Strictly-new rows only (ignore World Bank back-revisions):
missing = wb.new_observations(have, freq="monthly", include_revisions=False)
```

See [`examples/incremental.py`](examples/incremental.py) for a runnable,
commented walkthrough (`uv run python examples/incremental.py`).

## Development

Managed with [uv](https://docs.astral.sh/uv/). The package lives under a `src/`
layout (`src/worldbank_commodities/`) and tests under `tests/`.

```bash
uv sync                     # create .venv and install deps (dev + typing groups)
uv run ruff check .         # lint
uv run ruff format .        # format
uv run mypy                 # strict type checking
uv run pytest               # tests + coverage (network-mocked, no downloads)
```

Full quality gate (mirrors CI):

```bash
uv run ruff check . && uv run ruff format --check .
uv run codespell src tests README.md CHANGELOG.md
uv run deptry src
uv run interrogate -c pyproject.toml src tests   # docstring coverage >= 95%
uv run mypy
uv run pytest
```

[pre-commit](https://pre-commit.com/) runs ruff, codespell, and mypy on every
commit:

```bash
uv run pre-commit install       # one-time, enables the git hook
uv run pre-commit run --all-files
```

CI (`.github/workflows/ci.yml`) runs the same gate on every push and pull
request.

### Build and publish

Build the sdist and wheel with uv:

```bash
uv build
```

Publishing a GitHub release triggers `.github/workflows/publish.yml`, which runs
`uv build` and `uv publish` to PyPI using Trusted Publishing, so no API token is
stored in the repo. To publish by hand instead, run `uv publish` with your own
credentials.

## Changelog

See [CHANGELOG.md](CHANGELOG.md) for release notes (Keep a Changelog format).

## Citation

If you use this wrapper in your work, please cite it (see
[CITATION.cff](CITATION.cff); GitHub renders a "Cite this repository" button):

> Morão, H. (2026). *worldbank-commodities* (v0.1.0) [Software].
> https://github.com/hmorao95/worldbank-commodities

Please also credit the underlying data source, the World Bank Commodity Price
Data (The Pink Sheet).

## Data source & license

Data © The World Bank, published under
[CC BY 4.0](https://datacatalog.worldbank.org/public-licenses#cc-by).
Source: <https://www.worldbank.org/en/research/commodity-markets>.

This wrapper code is released under the MIT License.
