Metadata-Version: 2.4
Name: worldbank-commodities
Version: 0.8.1
Summary: Python wrapper to extract World Bank Pink Sheet commodity price data (oil, gas, metals, agriculture).
Project-URL: Homepage, https://github.com/hmorao95/worldbank-commodities
Project-URL: Repository, https://github.com/hmorao95/worldbank-commodities
Project-URL: Data source, https://www.worldbank.org/en/research/commodity-markets
Author-email: Hugo Morão <hfmorao@gmail.com>
License: MIT
License-File: LICENSE
Keywords: commodities,energy,oil,pink-sheet,prices,world-bank
Requires-Python: >=3.10
Requires-Dist: fire>=0.6
Requires-Dist: openpyxl>=3.1
Requires-Dist: pandas>=2.2
Requires-Dist: pyarrow>=15.0
Requires-Dist: requests>=2.28
Description-Content-Type: text/markdown

# worldbank-commodities

[![PyPI](https://img.shields.io/pypi/v/worldbank-commodities.svg)](https://pypi.org/project/worldbank-commodities/)
[![CI](https://github.com/hmorao95/worldbank-commodities/actions/workflows/ci.yml/badge.svg)](https://github.com/hmorao95/worldbank-commodities/actions/workflows/ci.yml)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Checked with mypy](https://img.shields.io/badge/mypy-checked-2a6db2.svg)](https://mypy-lang.org/)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
[![codecov](https://codecov.io/gh/hmorao95/worldbank-commodities/graph/badge.svg)](https://codecov.io/gh/hmorao95/worldbank-commodities)
[![Keep a Changelog](https://img.shields.io/badge/changelog-Keep%20a%20Changelog-orange.svg)](CHANGELOG.md)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

A small, dependency-light Python wrapper around the World Bank "Pink Sheet"
(Commodity Markets) price data.

The World Bank publishes monthly and annual commodity price series (crude oil,
natural gas, coal, metals, fertilizers, and many agricultural products) as Excel
workbooks whose download URLs carry a hash that changes every month. This
wrapper handles that for you:

- Auto-discovers the current Monthly/Annual workbook URLs from the official
  [Commodity Markets page](https://www.worldbank.org/en/research/commodity-markets),
  so there are no hardcoded links that break every month.
- Parses the awkward multi-row-header sheets into tidy `pandas` DataFrames (long
  or wide), detecting the units row automatically.
- Caches the raw workbook on disk (24 h TTL) so repeat calls are fast.
- Exposes commodity metadata (name and unit) and filters by commodity name and
  date range.
- Cleans the World Bank's "not available" tokens (`…`, `..`) to `NaN`.

No API key required; the data is public.

## Coverage

71 monthly series back to 1960, updated monthly. Groups include:

| Group | Examples |
|---|---|
| Energy | Crude oil (average/Brent/Dubai/WTI), coal, natural gas (US/Europe/LNG Japan) |
| Metals & precious | Aluminum, copper, iron ore, nickel, zinc, tin, lead, gold, silver, platinum |
| Fertilizers | Phosphate rock, DAP, TSP, urea, potassium chloride |
| Agriculture | Grains, vegetable oils & oilseeds, cocoa/coffee/tea, meat, sugar, cotton, rubber, timber |

Series flagged `**` by the World Bank have methodology/source breaks; see the
`Description` sheet in the source workbook.

## Quickstart

New to Python tooling? These four steps take you from nothing to a spreadsheet
of commodity prices. You need [git](https://git-scm.com/downloads) and
[uv](https://docs.astral.sh/uv/) (a fast Python package manager) installed.

Install uv if you do not have it:

```bash
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
```

```powershell
# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```

Then:

```bash
# 1. Get the code
git clone https://github.com/hmorao95/worldbank-commodities.git
cd worldbank-commodities

# 2. Install it (uv creates a .venv and pulls dependencies; no manual Python setup)
uv sync

# 3. See what commodities are available
uv run worldbank-commodities list-commodities

# 4. Save every monthly price series to a CSV you can open in Excel
uv run worldbank-commodities to-csv prices.csv --freq monthly
```

A bare output filename is written into the repo's `outputs/` folder
automatically (so `prices.csv` becomes `outputs/prices.csv`). That folder's
contents are gitignored, so extracts stay out of version control. Pass a path
with a directory (e.g. `data/prices.csv` or an absolute path) to write
elsewhere.

From there, the same commands take options to narrow or reshape the data:

```bash
# Save to Excel instead of CSV
uv run worldbank-commodities to-excel prices.xlsx --freq monthly

# Just one commodity (case-insensitive, matches on any part of the name)
uv run worldbank-commodities to-csv brent.csv --freq monthly --commodities "Crude oil, Brent"

# A date range, one column per commodity (wide)
uv run worldbank-commodities to-csv oil.csv --freq monthly \
    --commodities "crude oil" --start 2000-01 --end 2024-12 --wide

# Annual, real (inflation-adjusted) prices instead of nominal monthly
uv run worldbank-commodities to-csv real.csv --freq annual_real

# Keep an existing file current: add only new and revised rows
uv run worldbank-commodities update-csv prices.csv --freq monthly

# See every command and flag
uv run worldbank-commodities --help
```

See [Reference](#reference) for the full list of commands and options.

That is it. `outputs/prices.csv` now holds one row per observation
(`date, mdates, commodity, price, unit`). `mdates` is a Stata-style monthly id
such as `1960m1`, handy for merging with Stata time series; it is added only for
the monthly frequency (in both long and wide output).

> Note: the `worldbank-commodities` command lives inside the project's `.venv`,
> which is why each command starts with `uv run`. If you skip `uv run` and get
> "command not found", that is the reason. To get a global command, see
> [Command line](#command-line) below.

To install the library into your own Python environment instead, use pip:

```bash
pip install worldbank-commodities
```

Requires Python 3.10 or newer (uv installs a suitable Python for you if needed).

## Usage from Python

Once installed you can call the library from your own scripts. Run these with
`uv run python your_script.py` (or `uv run python` for an interactive session).

`get_prices()` returns a [pandas](https://pandas.pydata.org/) DataFrame, the
standard table type for data work in Python.

```python
from worldbank_commodities import WorldBankCommodities

wb = WorldBankCommodities()

# All 71 commodities + their units
wb.list_commodities()

# Tidy long DataFrame: date, mdates, commodity, price, unit
wb.get_prices(freq="monthly")

# Just crude oil + European gas since 2000, wide (one column per commodity)
wb.get_prices(
    freq="monthly",
    commodities=["Crude oil, Brent", "Natural gas, Europe"],
    start="2000-01",
    wide=True,
)

# Annual nominal or real (deflated) prices
wb.get_prices(freq="annual")  # nominal
wb.get_prices(freq="annual_real")  # real

# One-liner to CSV or Excel
wb.to_csv("brent.csv", freq="monthly", commodities=["Crude oil, Brent"], wide=True)
wb.to_excel("brent.xlsx", freq="monthly", commodities=["Crude oil, Brent"], wide=True)

# Incremental update: only append observations newer than what's already saved.
# First run writes the full history; later runs just tack on the new month(s).
wb.update_csv("commodities_monthly.csv", freq="monthly")
```

What the options mean:

- `freq` picks the dataset: `"monthly"`, `"annual"` (nominal prices), or
  `"annual_real"` (inflation-adjusted prices).
- `commodities` filters by name. Matching is case-insensitive and matches on any
  part of the name, so `commodities=["crude oil"]` returns all four crude-oil
  series. Leave it out to get everything.
- `start` and `end` limit the date range, for example `start="2000-01"`.
- `wide=False` (the default) gives one row per observation, which is easy to
  filter and plot. `wide=True` gives one column per commodity, which is handy
  for a side-by-side spreadsheet. Run `wb.list_commodities()` to see the exact
  names.

## Command line

The CLI is generated from the library with
[python-fire](https://github.com/google/python-fire): each method becomes a
subcommand and each parameter becomes a flag, so there is no separate set of
options to learn. Run `worldbank-commodities --help` to see them all.

Inside a uv project the console script lives in `.venv`, so call it with `uv run`
(or activate the venv first). `python -m worldbank_commodities` works too:

```bash
uv run worldbank-commodities list-commodities
```

To make the command available everywhere (outside this project, on your PATH),
install it as a uv tool once:

```bash
uv tool install .
worldbank-commodities list-commodities
```

The examples below use the bare command; prefix them with `uv run` if you have
not installed the tool globally.

```bash
# List every commodity and its unit
worldbank-commodities list-commodities

# Print a table to the terminal
worldbank-commodities get-prices --freq monthly

# Extract to a CSV
worldbank-commodities to-csv prices.csv --freq monthly

# Extract to an Excel workbook
worldbank-commodities to-excel prices.xlsx --freq monthly

# Filtered by one commodity (quote the name so the shell keeps it together)
worldbank-commodities get-prices --freq monthly \
    --commodities "Crude oil, Brent" --start 2010-01 --wide --out brent.csv

# Several commodities: pass a list. On Windows cmd wrap it in double quotes
# and use single quotes inside: --commodities "['Crude oil, Brent', 'Gold']"
worldbank-commodities get-prices --freq monthly \
    --commodities "['Crude oil, Brent', 'Natural gas, Europe']" --wide

# Incremental: add new and revised rows to an existing long CSV
worldbank-commodities update-csv commodities_monthly.csv --freq monthly

# Show what is missing from a CSV without writing it
worldbank-commodities new-observations commodities_monthly.csv --freq monthly

# Aggregate price indices (Energy, Agriculture, Metals, ...; base 2010=100)
worldbank-commodities get-indices --freq monthly --indices energy

# Full series definitions from the workbook's Description sheet
worldbank-commodities describe
```

## Reference

Run `worldbank-commodities --help`, or `worldbank-commodities <command> --help`
for a single command, to see this from the CLI.

### CLI commands

| Command | Positional | Options |
|---|---|---|
| `list-commodities` | (none) | `--freq` |
| `list-indices` | (none) | `--freq` |
| `describe` | (none) | `--freq` |
| `get-prices` | (none) | `--freq --commodities --start --end --wide --change --out` |
| `get-indices` | (none) | `--freq --indices --start --end --wide --change --out` |
| `to-csv` | `PATH` | `--freq --commodities --start --end --wide --change` |
| `to-excel` | `PATH` | `--freq --commodities --start --end --wide --change` |
| `to-parquet` | `PATH` | `--freq --commodities --start --end --wide --change` |
| `to-json` | `PATH` | `--freq --commodities --start --end --wide --change` |
| `update-csv` | `PATH` | `--freq --commodities --start --end` |
| `new-observations` | `EXISTING` | `--freq --commodities --start --end --include-revisions` |

For `get-prices`, `--out` picks the format by extension (`.xlsx`/`.xls` Excel,
`.parquet` Parquet, `.json` JSON, otherwise CSV); without `--out` it prints a
table. A bare output filename (no directory) is written into the `outputs/`
folder; pass a path with a directory to write elsewhere.

### Options

| Option | Values | Meaning |
|---|---|---|
| `--freq` | `monthly` (default), `annual`, `annual_real` | Dataset: nominal monthly, nominal annual, or real (deflated) annual. |
| `--commodities` | name or list of names | Case-insensitive substring filter. Omit for all. A single name is a plain string; several use a list (see Windows quoting above). |
| `--start`, `--end` | e.g. `2000-01`, `2000` | Inclusive date bounds. |
| `--wide` | flag | One column per commodity indexed by date (plus a leading `mdates` column for monthly). Default is long: one row per observation. |
| `--change` | `mom`, `yoy` | Return percentage changes instead of levels: month-over-month, or year-over-year (12 months for monthly, 1 step for annual). |
| `--include-revisions` | flag (default on) | For `new-observations`: also return values the World Bank has since revised, not only unseen rows. |

### Python API

```python
WorldBankCommodities(cache_dir=None, cache_ttl_hours=24.0, timeout=60.0, session=None)
```

| Method | Returns |
|---|---|
| `list_commodities(freq="monthly")` | DataFrame of `commodity`, `unit` |
| `list_indices(freq="monthly")` | DataFrame of index names |
| `describe(freq="monthly")` | Definitions: `group`, `commodity`, `description` |
| `get_prices(freq="monthly", commodities=None, start=None, end=None, wide=False, change=None)` | Long or wide DataFrame |
| `get_indices(freq="monthly", indices=None, start=None, end=None, wide=False, change=None)` | Aggregate price indices (base 2010=100) |
| `to_csv(path, **kwargs)` | Writes CSV; returns the `Path` |
| `to_excel(path, **kwargs)` | Writes `.xlsx`; returns the `Path` |
| `to_parquet(path, **kwargs)` | Writes `.parquet`; returns the `Path` |
| `to_json(path, **kwargs)` | Writes JSON records; returns the `Path` |
| `update_csv(path, freq="monthly", commodities=None, start=None, end=None)` | Upserts the CSV; returns the added/revised rows |
| `new_observations(existing, freq="monthly", commodities=None, start=None, end=None, include_revisions=True)` | Rows missing from `existing` (DataFrame or CSV path) |

The `to_*` writers forward `**kwargs` to `get_prices` (`freq`, `commodities`,
`start`, `end`, `wide`, `change`). The long output columns are `[date, commodity,
price, unit]`, plus `mdates` for the monthly frequency. With `change="mom"` or
`"yoy"` the values are percentage changes and the unit is `%`.

## Incremental updates

`update_csv()` (CLI: `update-csv`) keeps a long-format CSV current without
rewriting the whole file each run:

- First run (file missing) writes the full extract.
- Later runs read what's already saved and upsert on `(commodity, date)`: they
  append new observations and overwrite any values the World Bank has since
  revised. That covers newer months for commodities already present and
  commodities that were not extracted before.
- If nothing changed, the file is left untouched and an empty frame is returned.

```python
new_rows = wb.update_csv("commodities_monthly.csv", freq="monthly")
print(f"added {len(new_rows)} observations")
```

This works on the tidy long format (`[date, commodity, price, unit]`); wide CSVs
are not supported for incremental merging.

### Just the delta, in memory

If you keep your data somewhere other than a CSV, `new_observations()` is the
file-free building block. Give it what you already have (a long DataFrame or a
long-CSV path) and it returns only the rows you are missing: newer dates for
commodities you track, plus every row of commodities you have never extracted.

```python
have = my_store.load()  # any long DataFrame with [date, commodity, price]
missing = wb.new_observations(have, freq="monthly")
my_store.append(missing)

# Strictly-new rows only (ignore World Bank back-revisions):
missing = wb.new_observations(have, freq="monthly", include_revisions=False)
```

It handles the cases you would expect:

```python
full = wb.get_prices("monthly")

# Already have everything -> nothing to fetch
wb.new_observations(full, freq="monthly")  # empty

# Missing the latest months -> only those rows come back
wb.new_observations(full.iloc[:-100], freq="monthly")

# A commodity never extracted -> its whole history is returned
wb.new_observations(full[full["commodity"] != "Gold"], freq="monthly")

# Accepts a CSV path too, not just a DataFrame
wb.new_observations("commodities_monthly.csv", freq="monthly")
```

## Development

Managed with [uv](https://docs.astral.sh/uv/). The package lives under a `src/`
layout (`src/worldbank_commodities/`) and tests under `tests/`.

```bash
uv sync                     # create .venv and install deps (dev + typing groups)
uv run ruff check .         # lint
uv run ruff format .        # format
uv run mypy                 # strict type checking
uv run pytest               # tests + coverage (network-mocked, no downloads)
```

Full quality gate (mirrors CI):

```bash
uv run ruff check . && uv run ruff format --check .
uv run codespell src tests README.md CHANGELOG.md
uv run deptry src
uv run interrogate -c pyproject.toml src tests   # docstring coverage >= 95%
uv run mypy
uv run pytest
```

[pre-commit](https://pre-commit.com/) runs ruff, codespell, and mypy on every
commit:

```bash
uv run pre-commit install       # one-time, enables the git hook
uv run pre-commit run --all-files
```

CI (`.github/workflows/ci.yml`) runs the same gate on every push and pull
request.

### Build and publish

Build the sdist and wheel with uv:

```bash
uv build
```

Publishing a GitHub release triggers `.github/workflows/publish.yml`, which runs
`uv build` and `uv publish` to PyPI using Trusted Publishing, so no API token is
stored in the repo. To publish by hand instead, run `uv publish` with your own
credentials.

## Changelog

See [CHANGELOG.md](CHANGELOG.md) (Keep a Changelog format) or the
[GitHub releases](https://github.com/hmorao95/worldbank-commodities/releases)
for per-version notes.

## Citation

If you use this wrapper in your work, please cite it (see
[CITATION.cff](CITATION.cff); GitHub renders a "Cite this repository" button):

> Morão, H. (2026). *worldbank-commodities* (v0.8.1) [Software].
> https://pypi.org/project/worldbank-commodities/

Please also credit the underlying data source, the World Bank Commodity Price
Data (The Pink Sheet).

## Data source & license

Data © The World Bank, published under
[CC BY 4.0](https://datacatalog.worldbank.org/public-licenses#cc-by).
Source: <https://www.worldbank.org/en/research/commodity-markets>.

This wrapper code is released under the MIT License.
