readabs.splice
Priority splicing of mixed-frequency time series.
This module has two layers:
splice
The core primitive. Deliberately source-agnostic: it takes pandas Series
you have already fetched (by description, by ID, however you like) and
splices them into one series. It knows nothing about the ABS, ships no
static lookup table, and makes no guesses about which series belong together
— that judgement stays with the caller.
select / select_one / select_and_splice
A thin ABS-aware convenience layer over splice. Each resolves
(data, meta, selector) sources to Series via readabs.find_abs_id
(carrying each series' ABS unit on .attrs["unit"]), so the common case —
splice a few ABS series selected by description/frequency — is one call,
while select stays exposed for when you need a transform between
selecting and splicing. A selector is either a {search_value:
meta_column} dict, or a bare ABS Series ID string when you already know
exactly which series you want.
Splice design
Given an ordered list of segments (highest priority / most authoritative
first), splice():
- align — put every segment on one common
PeriodIndex. By default the grid is the finest frequency present, which dissolves anchor clashes (Q-NOV vs Q-DEC, A-JUN vs A-DEC) because every coarse period maps cleanly onto a finer one. Coarser segments are placed at their period-end; finer segments are aggregated down withagg. rebase — (opt-in; off by default) for each junction, multiplicatively scale the lower-priority segment so its level matches the running result over the overlapping date window (phase-agnostic; works even when two series never share an exact period). Falls back to a single junction point if there is no overlap, and flags it. Off by default because it transforms your data — nothing is silently rescaled unless you ask.
Rebasing assumes **ratio-scale** inputs — series whose zero is meaningful and whose discrepancy between segments is *proportional*. Indexes (CPI, price/volume indices on different base periods) are the canonical case; a proportional benchmark revision of a count works too. It is **wrong** for series that cross zero (rates of change, balances, net flows) or whose segments differ by an *additive* offset rather than a scale factor — a negative or non-finite factor is caught and raises. With ``rebase=False`` (the default) the raw levels are coalesced as-is: if two same-unit segments already agree, rebasing only invents a discrepancy to "correct".- coalesce—
combine_firstdown the priority chain: take segment 1, fill gaps from segment 2, then 3, ... The result keeps only the periods that actually carry data — a coarse back-history stays sparse on a finer grid rather than being NaN-filled, and nothing is interpolated (passfill=to densify). - resample— (optional) resample the spliced result to a chosen output frequency/anchor.
The returned join report makes every rebase factor and overlap visible, so a splice can be audited rather than trusted blindly.
1"""Priority splicing of mixed-frequency time series. 2 3This module has two layers: 4 5``splice`` 6 The core primitive. Deliberately *source-agnostic*: it takes pandas Series 7 you have already fetched (by description, by ID, however you like) and 8 splices them into one series. It knows nothing about the ABS, ships no 9 static lookup table, and makes no guesses about which series belong together 10 — that judgement stays with the caller. 11 12``select`` / ``select_one`` / ``select_and_splice`` 13 A thin ABS-aware convenience layer over ``splice``. Each resolves 14 ``(data, meta, selector)`` sources to Series via ``readabs.find_abs_id`` 15 (carrying each series' ABS unit on ``.attrs["unit"]``), so the common case — 16 splice a few ABS series selected by description/frequency — is one call, 17 while ``select`` stays exposed for when you need a transform between 18 selecting and splicing. A *selector* is either a ``{search_value: 19 meta_column}`` dict, or a bare ABS Series ID string when you already know 20 exactly which series you want. 21 22Splice design 23------------- 24Given an ordered list of segments (highest priority / most authoritative 25first), :func:`splice`: 26 271. **align** — put every segment on one common ``PeriodIndex``. By default 28 the grid is the *finest* frequency present, which dissolves 29 anchor clashes (Q-NOV vs Q-DEC, A-JUN vs A-DEC) because every 30 coarse period maps cleanly onto a finer one. Coarser segments 31 are placed at their period-*end*; finer segments are 32 aggregated down with ``agg``. 332. **rebase** — *(opt-in; off by default)* for each junction, 34 *multiplicatively* scale the lower-priority segment so its level 35 matches the running result over the *overlapping date window* 36 (phase-agnostic; works even when two series never share an exact 37 period). Falls back to a single junction point if there is no 38 overlap, and flags it. Off by default because it transforms 39 your data — nothing is silently rescaled unless you ask. 40 41 Rebasing assumes **ratio-scale** inputs — series whose zero is 42 meaningful and whose discrepancy between segments is 43 *proportional*. Indexes (CPI, price/volume indices on different 44 base periods) are the canonical case; a proportional benchmark 45 revision of a count works too. It is **wrong** for series that 46 cross zero (rates of change, balances, net flows) or whose 47 segments differ by an *additive* offset rather than a scale 48 factor — a negative or non-finite factor is caught and raises. 49 With ``rebase=False`` (the default) the raw levels are coalesced 50 as-is: if two same-unit segments already agree, rebasing only 51 invents a discrepancy to "correct". 523. **coalesce**— ``combine_first`` down the priority chain: take segment 1, 53 fill gaps from segment 2, then 3, ... The result keeps only 54 the periods that actually carry data — a coarse back-history 55 stays sparse on a finer grid rather than being NaN-filled, and 56 nothing is interpolated (pass ``fill=`` to densify). 574. **resample**— (optional) resample the spliced result to a chosen output 58 frequency/anchor. 59 60The returned join report makes every rebase factor and overlap visible, so a 61splice can be audited rather than trusted blindly. 62""" 63 64from __future__ import annotations 65 66import math 67from collections.abc import Iterable, Sequence 68from typing import Literal, cast 69 70import pandas as pd 71from pandas import DataFrame, PeriodIndex, Series 72 73from readabs.abs_meta_data import metacol as mc # used by the select() layer 74from readabs.search_abs_meta import find_abs_id # used by the select() layer 75 76# Frequency rank — higher number = finer frequency. 77_FREQ_RANK: dict[str, int] = {"Y": 0, "A": 0, "Q": 1, "M": 2, "W": 3, "D": 4} 78 79 80def _base(freqstr: str) -> str: 81 """Return the base frequency character (``"Q-NOV"`` -> ``"Q"``, ``"A-JUN"`` -> ``"Y"``).""" 82 char = freqstr.split("-", maxsplit=1)[0][0].upper() 83 return "Y" if char == "A" else char 84 85 86def _rank(freqstr: str) -> int: 87 """Return the frequency rank for a PeriodIndex freq string.""" 88 return _FREQ_RANK[_base(freqstr)] 89 90 91def _as_period_index(s: Series) -> Series: 92 """Ensure *s* has a PeriodIndex; convert from DatetimeIndex if needed.""" 93 if isinstance(s.index, PeriodIndex): 94 return s 95 if isinstance(s.index, pd.DatetimeIndex): 96 return s.set_axis(s.index.to_period()) 97 raise TypeError(f"Series '{s.name}' must have a PeriodIndex or DatetimeIndex, got {type(s.index).__name__}.") 98 99 100def _pidx(s: Series) -> PeriodIndex: 101 """Return *s*'s index as a (typed) PeriodIndex, converting if necessary.""" 102 return cast("PeriodIndex", _as_period_index(s).index) 103 104 105def _pick_target(segments: Sequence[Series]) -> str: 106 """Choose the default common-grid freq: the finest present. 107 108 If two or more segments share the *finest* rank but with different anchors 109 (e.g. ``Q-NOV`` and ``Q-DEC``) and there is nothing finer to splice them 110 onto, raise — picking one anchor would silently reanchor the other and 111 could assume wrong. Resolve it by passing a finer ``target`` (e.g. 112 ``"M"``), or by including a finer-frequency segment. 113 """ 114 freqs = [str(_pidx(s).freqstr) for s in segments] 115 ranks = [_rank(f) for f in freqs] 116 top = max(ranks) 117 top_freqs = {f for f, r in zip(freqs, ranks, strict=True) if r == top} 118 if len(top_freqs) > 1: 119 raise ValueError( 120 f"Clashing anchors at the finest frequency: {sorted(top_freqs)}. " 121 f"Pass a finer target (e.g. target='M') to splice them on a common grid." 122 ) 123 return next(iter(top_freqs)) 124 125 126def _to_grid(s: Series, target: str, agg: str) -> Series: 127 """Map *s* onto the *target* PeriodIndex frequency. 128 129 Finer-than-target segments are aggregated down with *agg*; equal-or-coarser 130 segments are placed at their period-end on the target grid. 131 """ 132 s = _as_period_index(s).dropna() 133 idx = cast("PeriodIndex", s.index) 134 src = str(idx.freqstr) 135 if _rank(src) > _rank(target): 136 # finer -> coarser: aggregate the sub-periods that fall in each target period 137 out = s.groupby(idx.asfreq(target)).agg(agg) 138 elif _rank(src) == _rank(target) and _base(src) == _base(target) and src != target: 139 # same frequency, different anchor (e.g. Q-NOV vs Q-DEC) — reanchoring 140 # would silently shift every period, so refuse rather than assume. 141 raise ValueError( 142 f"Cannot place '{s.name}' ({src}) onto a {target} grid without reanchoring. " 143 f"Use a finer target (e.g. target='M')." 144 ) 145 else: 146 # coarser (or identical) -> place each value at its period-end on the grid 147 out = Series(s.to_numpy(), index=idx.asfreq(target, how="E"), name=s.name) 148 out = out[~out.index.duplicated(keep="last")] 149 return out.sort_index() 150 151 152def _rebase_factor(result: Series, seg: Series) -> tuple[float, str, int, pd.Period | None, pd.Period | None]: 153 """Compute the factor to bring *seg* onto *result*'s level. 154 155 Measured as the ratio of mean levels over the overlapping *date span*, so 156 it is phase-agnostic — it works even when the two series share no exact 157 period (e.g. Q-NOV vs Q-DEC mapped onto a monthly grid). Falls back to a 158 single junction point when the spans do not overlap at all. 159 160 Returns ``(factor, method, overlap_n, window_start, window_end)``. 161 """ 162 r, s = result.dropna(), seg.dropna() 163 if len(r) and len(s): 164 lo = max(r.index.min(), s.index.min()) 165 hi = min(r.index.max(), s.index.max()) 166 if lo <= hi: 167 r_win, s_win = r.loc[lo:hi], s.loc[lo:hi] 168 if len(r_win) and len(s_win) and s_win.mean(): 169 return float(r_win.mean() / s_win.mean()), "window", min(len(r_win), len(s_win)), lo, hi 170 # No overlapping span — fall back to the nearest junction point. 171 r0 = result.first_valid_index() 172 if r0 is not None: 173 before = s.loc[:r0] 174 if len(before) and before.iloc[-1]: 175 return float(result.loc[r0] / before.iloc[-1]), "junction", 0, None, None 176 return 1.0, "none", 0, None, None 177 178 179def splice( 180 segments: Iterable[Series], 181 *, 182 target: str | None = None, 183 rebase: bool = False, 184 agg: str = "mean", 185 output: str | None = None, 186 fill: Literal["ffill", "interpolate"] | None = None, 187 name: str | None = None, 188) -> tuple[Series, DataFrame]: 189 """Splice mixed-frequency *segments* into one series, highest priority first. 190 191 Parameters 192 ---------- 193 segments 194 Ordered list of pandas Series (PeriodIndex or DatetimeIndex). The 195 first is highest priority: it wins where periods overlap and (when 196 ``rebase`` is on) sets the level everything else is rebased to. 197 target 198 Common-grid frequency (e.g. ``"M"``, ``"Q-DEC"``). Defaults to the 199 finest frequency present (anchor clashes step one rank finer). 200 rebase 201 Off by default — segments are coalesced at their **raw** levels, with no 202 silent transformation of your data. Set ``True`` to *multiplicatively* 203 rescale each lower-priority segment to the running result's level before 204 coalescing. Rebasing assumes **ratio-scale** inputs (meaningful zero, 205 proportional discrepancy between segments) — splicing index series on 206 different base periods (CPI, price/volume indices) is the case that 207 needs it. It is wrong for zero-crossing series (rates, balances) or 208 additive level breaks, and it *invents* a correction when same-unit 209 segments already agree — which is why it is opt-in. A non-finite or 210 non-positive factor raises. See the module docstring's *rebase* step. 211 agg 212 Aggregator used when a segment is finer than the grid (or when 213 downsampling to *output*). ``"mean"`` for index levels; use ``"sum"`` 214 for flows. 215 output 216 Optional final frequency to resample the spliced result to. 217 fill 218 Optional gap fill. By default (``None``) the result contains only the 219 periods that actually have data — no NaN rows are inserted for the gaps 220 a coarse segment leaves on a finer grid, and nothing is interpolated. 221 ``"ffill"`` or ``"interpolate"`` densify the result onto the full grid 222 first and then fill. 223 name 224 Name for the result series (defaults to the first segment's name). 225 226 Returns 227 ------- 228 tuple[Series, DataFrame] 229 The spliced series and a one-row-per-junction report. 230 231 """ 232 segments = list(segments) 233 if not segments: 234 raise ValueError("splice() needs at least one segment.") 235 236 grid = target or _pick_target(segments) 237 on_grid = [_to_grid(s, grid, agg) for s in segments] 238 239 result = on_grid[0].copy() 240 rows: list[dict[str, object]] = [] 241 for i, seg in enumerate(on_grid[1:], start=1): 242 if rebase: 243 factor, method, n, lo, hi = _rebase_factor(result, seg) 244 # Multiplicative rebasing assumes ratio-scale inputs. A non-finite 245 # factor (near-zero denominator) or a non-positive one (the overlap 246 # means have opposite signs, which would flip the back-history) means 247 # the data is not ratio-scale — fail loud rather than ship it. A 248 # large *magnitude* is fine: a legitimate base-period difference can 249 # need a 50x factor, so only sign and finiteness are guarded. 250 if not (math.isfinite(factor) and factor > 0): 251 raise ValueError( 252 f"splice: rebase factor for segment {i} ('{seg.name}') is {factor} over " 253 f"{lo}..{hi}. Multiplicative rebasing needs ratio-scale inputs (meaningful " 254 f"zero, proportional discrepancy); a non-finite or non-positive factor means " 255 f"the segments cross zero or differ additively. Pass rebase=False to coalesce " 256 f"raw levels instead." 257 ) 258 else: 259 factor, method, n, lo, hi = 1.0, "off", 0, None, None 260 seg_rebased = seg * factor 261 rows.append( 262 { 263 "segment": i, 264 "name": str(seg.name), 265 "freq_in": str(_pidx(segments[i]).freqstr), 266 "method": method, 267 "overlap_n": n, 268 "window_start": str(lo) if lo is not None else "", 269 "window_end": str(hi) if hi is not None else "", 270 "factor": round(factor, 6), 271 "fills_from": str(seg.dropna().index.min()), 272 } 273 ) 274 result = result.combine_first(seg_rebased) 275 276 # By default keep only the periods that actually carry data: do NOT reindex 277 # onto a dense grid (which would manufacture NaN for the gaps a coarse 278 # back-history leaves on a finer grid) and do NOT interpolate. A long-run 279 # series therefore stays sparse where it is old and coarse, and plots as one 280 # continuous line with no holes and no invented points. 281 result = result.dropna().sort_index() 282 283 if output and output != grid: 284 result = _to_grid(result, output, agg).dropna().sort_index() 285 grid = output 286 287 if fill in ("ffill", "interpolate") and len(result): 288 # Explicit opt-in: densify onto the full grid, then fill. 289 full = pd.period_range(result.index.min(), result.index.max(), freq=grid) 290 result = result.reindex(full) 291 result = result.ffill() if fill == "ffill" else result.interpolate() 292 293 result.name = name or str(segments[0].name) 294 report = DataFrame(rows) 295 return result, report 296 297 298# A select_and_splice() source: the fetched data dict, its meta, and either a 299# {search_value: meta_column} selector (readabs' find_abs_id convention) or a 300# bare ABS Series ID string (matched exactly against the Series ID column). 301Source = tuple[dict[str, DataFrame], DataFrame, dict[str, str] | str] 302 303 304def select_one(data: dict[str, DataFrame], meta: DataFrame, selector: dict[str, str] | str) -> Series: 305 """Select the single Series for one ``(data, meta, selector)`` — the single-source wrapper. 306 307 Convenience for the common one-selector case; equivalent to 308 ``select([(data, meta, selector)])[0]``. The *selector* is either a 309 ``{search_value: meta_column}`` dict for ``find_abs_id``, or a bare ABS 310 Series ID string, matched exactly against the metadata's Series ID column. 311 Returns the Series named by its Series ID, with its ABS unit on 312 ``.attrs["unit"]``. 313 """ 314 if isinstance(selector, str): 315 # A bare Series ID — same find_abs_id machinery, but exact-match on the 316 # Series ID column so one ID cannot substring-match another. 317 try: 318 table, series_id, unit = find_abs_id(meta, {selector: mc.id}, exact_match=True, validate_unique=True) 319 except ValueError as exc: 320 raise ValueError(f"select: series ID {selector!r} not found in the supplied metadata.") from exc 321 else: 322 table, series_id, unit = find_abs_id(meta, selector, validate_unique=True) 323 s = data[table][series_id].copy() 324 s.name = series_id 325 s.attrs["unit"] = str(unit) 326 return s 327 328 329def select(sources: Iterable[Source], *, require_same_units: bool = True) -> list[Series]: 330 """Select a series for each ``(data, meta, selector)`` — the iterable in, iterable out. 331 332 The composable selection primitive: takes the iterable of ``(data, meta, 333 selector)`` sources and returns the matching list of Series, ready to hand to 334 :func:`splice` (directly, or after a per-series transform). Each selection 335 goes through ``readabs.find_abs_id`` with ``validate_unique=True``, which 336 de-duplicates on Series ID first — so a selector matching the same series in 337 several tables resolves cleanly, while one matching two genuinely different 338 series raises rather than guessing. 339 340 Parameters 341 ---------- 342 sources 343 Iterable of ``(data, meta, selector)``: 344 345 - ``data`` — ``dict[table_name, DataFrame]`` from ``read_abs_cat``. 346 - ``meta`` — the matching metadata DataFrame. 347 - ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``, e.g. 348 ``{"Index Numbers ; All groups CPI ; Australia ;": mc.did, 349 "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series 350 ID string (e.g. ``"A2325846C"``), matched exactly. 351 require_same_units 352 If ``True`` (default) **raise** when the selected series do not all share 353 the same ABS unit — units must cohere to be spliced. Set ``False`` when 354 you deliberately select different-unit series together (e.g. two counts 355 and a rate that you will combine yourself). 356 357 Returns 358 ------- 359 list[Series] 360 One Series per source, each named by its Series ID with its ABS unit in 361 ``series.attrs["unit"]``. Unpack it (``a, b = select([...])``), map a 362 transform over it, or pass it straight to :func:`splice`. A later 363 transform drops the unit attr — correctly, since the unit is then no 364 longer the ABS one. 365 366 Raises 367 ------ 368 ValueError 369 If ``require_same_units`` and the selected series carry mixed units. 370 371 """ 372 segments = [select_one(data, meta, selector) for data, meta, selector in sources] 373 if require_same_units: 374 units = [str(s.attrs.get("unit", "")) for s in segments] 375 if len(set(units)) > 1: 376 detail = ", ".join(f"{s.name}={u!r}" for s, u in zip(segments, units, strict=True)) 377 raise ValueError( 378 f"select: selected series have mismatched units ({detail}). Pass " 379 f"require_same_units=False to select different-unit series together." 380 ) 381 return segments 382 383 384def select_and_splice( 385 sources: Iterable[Source], 386 *, 387 target: str | None = None, 388 rebase: bool = False, 389 agg: str = "mean", 390 output: str | None = None, 391 fill: Literal["ffill", "interpolate"] | None = None, 392 name: str | None = None, 393 require_same_units: bool = True, 394) -> tuple[Series, str, DataFrame]: 395 """Select one series per source and :func:`splice` them — the no-transform case. 396 397 Sugar for ``splice(select(sources))`` with a unit guard. When 398 you need a transform *between* selecting and splicing (e.g. a growth rate), 399 compose :func:`select` and :func:`splice` directly instead — that is the whole 400 reason :func:`select` is exposed separately. 401 402 Parameters 403 ---------- 404 sources 405 Ordered iterable of ``(data, meta, selector)``, **highest priority 406 first** (same priority rule as :func:`splice`): 407 408 - ``data`` — ``dict[table_name, DataFrame]`` from ``read_abs_cat``. 409 - ``meta`` — the matching metadata DataFrame. 410 - ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``, 411 e.g. ``{"Index Numbers ; All groups CPI ; Australia ;": mc.did, 412 "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series 413 ID string (e.g. ``"A2325846C"``), matched exactly. In the common case 414 the only thing differing between two sources is the frequency, so a 415 shared *base* selector composes with ``base | {"Quarter": mc.freq}``. 416 target, rebase, agg, output, fill, name 417 Passed straight through to :func:`splice`. 418 require_same_units 419 Forwarded to :func:`select`: if ``True`` (default) raise when the 420 selected segments carry mixed units; ``False`` overrides (the result is 421 then labelled with the highest-priority segment's unit). 422 423 Returns 424 ------- 425 tuple[Series, str, DataFrame] 426 The spliced series, its unit (the highest-priority segment's unit), and 427 the :func:`splice` join report, augmented with ``series_id`` and 428 ``unit`` columns recording what each segment resolved to. 429 430 """ 431 segments = select(sources, require_same_units=require_same_units) 432 units = [str(s.attrs.get("unit", "")) for s in segments] 433 434 result, report = splice(segments, target=target, rebase=rebase, agg=agg, output=output, fill=fill, name=name) 435 # Audit trail: which Series ID / unit did each reported (lower-priority) segment use? 436 if len(report): 437 seg = [int(i) for i in report["segment"]] 438 report.insert(1, "series_id", [str(segments[i].name) for i in seg]) 439 report.insert(2, "unit", [units[i] for i in seg]) 440 return result, units[0], report 441 442 443# --------------------------------------------------------------------------- 444# Self-tests — `python splice.py` 445# --------------------------------------------------------------------------- 446if __name__ == "__main__": 447 import numpy as np 448 449 def _show(title: str, s: Series, rep: DataFrame) -> None: 450 print(f"\n{'=' * 70}\n{title}\n{'=' * 70}") 451 print( 452 f"freq={cast('PeriodIndex', s.index).freqstr} n={len(s)} non-null={s.notna().sum()} " 453 f"range={s.index.min()}..{s.index.max()}" 454 ) 455 if len(rep): 456 print(rep.to_string(index=False)) 457 458 # --- Case 1: monthly (new) + quarterly (old), level shift via index rebase 459 q = Series( 460 np.arange(100, 100 + 4 * 20, dtype=float), # 20 years quarterly, base ~100 461 index=pd.period_range("2000Q1", periods=80, freq="Q-DEC"), 462 name="cpi", 463 ) 464 m = Series( 465 np.arange(50.0, 50.0 + 60) * 0.5 + 130, # monthly on a *different* base 466 index=pd.period_range("2018-01", periods=60, freq="M"), 467 name="cpi", 468 ) 469 out, rep = splice([m, q], rebase=True) # monthly priority, quarterly fills the back-history 470 _show("Case 1 — M (priority) spliced with Q-DEC, auto-grid", out, rep) 471 print(f"check: rebased Q value at 2018-03 = {out.loc['2018-03']:.3f} (monthly 2018-01 = {m.iloc[0]:.3f})") 472 473 # --- Case 2: the anchor clash — Q-NOV vs Q-DEC, overlapping in time 474 q_dec = Series( 475 np.arange(200.0, 200 + 40), 476 index=pd.period_range("2010Q1", periods=40, freq="Q-DEC"), 477 name="x", 478 ) 479 q_nov = Series( 480 np.arange(80.0, 80 + 60), # 2000Q1..2014Q4 — overlaps q_dec over 2010-2014 481 index=pd.period_range("2000Q1", periods=60, freq="Q-NOV"), 482 name="x", 483 ) 484 print(f"\n{'=' * 70}\nCase 2 — Q-DEC + Q-NOV anchor clash\n{'=' * 70}") 485 try: 486 splice([q_dec, q_nov]) # no target -> must refuse rather than reanchor 487 except ValueError as exc: 488 print(f"default (no target) correctly raised:\n {exc}") 489 out2, rep2 = splice([q_dec, q_nov], target="M", rebase=True) # resolve on a common finer grid 490 _show("Case 2b — same, resolved with target='M' (window rebase across anchors)", out2, rep2) 491 492 # --- Case 3: daily + monthly. Default grid is the finest present = D. 493 d = Series( 494 np.linspace(10, 12, 365), 495 index=pd.period_range("2023-01-01", periods=365, freq="D"), 496 name="rate", 497 ) 498 mth = Series( 499 np.linspace(12, 13, 18), # 2023-07..2024-12 — overlaps the daily over 2023-H2 500 index=pd.period_range("2023-07", periods=18, freq="M"), 501 name="rate", 502 ) 503 out3, rep3 = splice([d, mth]) # daily priority -> finest grid = D, monthly placed sparsely 504 _show("Case 3 — D (priority) + M, default finest grid = D", out3, rep3) 505 out3b, rep3b = splice([mth, d], target="M", agg="mean") # explicitly ask for a monthly result 506 _show("Case 3b — same data, target='M' so daily is aggregated down", out3b, rep3b) 507 508 # --- Case 4: CPI-style 3-way chain (new monthly + indicator + quarterly) 509 new_m = Series(np.arange(135.0, 135 + 12), index=pd.period_range("2024-01", periods=12, freq="M"), name="cpi") 510 indic = Series(np.arange(120.0, 120 + 30), index=pd.period_range("2022-07", periods=30, freq="M"), name="cpi") 511 old_q_index = pd.period_range("1995Q1", periods=120, freq="Q-DEC") 512 old_q = Series(np.arange(40.0, 40 + 120), index=old_q_index, name="cpi") 513 out4, rep4 = splice([new_m, indic, old_q], name="cpi_long", rebase=True) 514 _show("Case 4 — 3-way: new monthly + indicator + quarterly", out4, rep4) 515 print( 516 f"\nfull series spans {out4.index.min()} .. {out4.index.max()}, {out4.notna().sum()} observations present" 517 ) 518 519 # --- Case 5: same, but ask for a clean quarterly output (downsample) 520 out5, rep5 = splice([new_m, indic, old_q], output="Q-DEC", name="cpi_long_q", rebase=True) 521 _show("Case 5 — same 3-way, resampled to a clean Q-DEC output", out5, rep5) 522 523 # --- Case 6: the select() layer — dict selector vs bare Series ID string 524 fake_meta = DataFrame( 525 { 526 mc.did: ["Index Numbers ; All groups CPI ; Australia ;"] * 2, 527 mc.id: ["A2325846C", "A128478317T"], 528 mc.unit: ["Index Numbers", "Index Numbers"], 529 mc.freq: ["Quarter", "Month"], 530 mc.table: ["640101", "648601"], 531 } 532 ) 533 fake_data = { 534 "640101": DataFrame({"A2325846C": q.to_numpy()[:40]}, index=q.index[:40]), 535 "648601": DataFrame({"A128478317T": m.to_numpy()}, index=m.index), 536 } 537 by_id = select_one(fake_data, fake_meta, "A2325846C") # bare Series ID string 538 by_dict = select_one(fake_data, fake_meta, {"Month": mc.freq}) # selector dict 539 print(f"\n{'=' * 70}\nCase 6 — select_one: bare Series ID vs selector dict\n{'=' * 70}") 540 print(f"by ID: name={by_id.name} unit={by_id.attrs['unit']!r} n={len(by_id)}") 541 print(f"by dict: name={by_dict.name} unit={by_dict.attrs['unit']!r} n={len(by_dict)}") 542 out6, unit6, rep6 = select_and_splice( 543 [(fake_data, fake_meta, "A128478317T"), (fake_data, fake_meta, "A2325846C")], rebase=True 544 ) 545 _show(f"Case 6b — select_and_splice by bare Series IDs (unit={unit6!r})", out6, rep6) 546 try: 547 select_one(fake_data, fake_meta, "NOSUCHID") # unknown ID -> fail loud 548 except ValueError as exc: 549 print(f"unknown ID correctly raised:\n {exc}") 550 551 print("\nAll cases ran.")
180def splice( 181 segments: Iterable[Series], 182 *, 183 target: str | None = None, 184 rebase: bool = False, 185 agg: str = "mean", 186 output: str | None = None, 187 fill: Literal["ffill", "interpolate"] | None = None, 188 name: str | None = None, 189) -> tuple[Series, DataFrame]: 190 """Splice mixed-frequency *segments* into one series, highest priority first. 191 192 Parameters 193 ---------- 194 segments 195 Ordered list of pandas Series (PeriodIndex or DatetimeIndex). The 196 first is highest priority: it wins where periods overlap and (when 197 ``rebase`` is on) sets the level everything else is rebased to. 198 target 199 Common-grid frequency (e.g. ``"M"``, ``"Q-DEC"``). Defaults to the 200 finest frequency present (anchor clashes step one rank finer). 201 rebase 202 Off by default — segments are coalesced at their **raw** levels, with no 203 silent transformation of your data. Set ``True`` to *multiplicatively* 204 rescale each lower-priority segment to the running result's level before 205 coalescing. Rebasing assumes **ratio-scale** inputs (meaningful zero, 206 proportional discrepancy between segments) — splicing index series on 207 different base periods (CPI, price/volume indices) is the case that 208 needs it. It is wrong for zero-crossing series (rates, balances) or 209 additive level breaks, and it *invents* a correction when same-unit 210 segments already agree — which is why it is opt-in. A non-finite or 211 non-positive factor raises. See the module docstring's *rebase* step. 212 agg 213 Aggregator used when a segment is finer than the grid (or when 214 downsampling to *output*). ``"mean"`` for index levels; use ``"sum"`` 215 for flows. 216 output 217 Optional final frequency to resample the spliced result to. 218 fill 219 Optional gap fill. By default (``None``) the result contains only the 220 periods that actually have data — no NaN rows are inserted for the gaps 221 a coarse segment leaves on a finer grid, and nothing is interpolated. 222 ``"ffill"`` or ``"interpolate"`` densify the result onto the full grid 223 first and then fill. 224 name 225 Name for the result series (defaults to the first segment's name). 226 227 Returns 228 ------- 229 tuple[Series, DataFrame] 230 The spliced series and a one-row-per-junction report. 231 232 """ 233 segments = list(segments) 234 if not segments: 235 raise ValueError("splice() needs at least one segment.") 236 237 grid = target or _pick_target(segments) 238 on_grid = [_to_grid(s, grid, agg) for s in segments] 239 240 result = on_grid[0].copy() 241 rows: list[dict[str, object]] = [] 242 for i, seg in enumerate(on_grid[1:], start=1): 243 if rebase: 244 factor, method, n, lo, hi = _rebase_factor(result, seg) 245 # Multiplicative rebasing assumes ratio-scale inputs. A non-finite 246 # factor (near-zero denominator) or a non-positive one (the overlap 247 # means have opposite signs, which would flip the back-history) means 248 # the data is not ratio-scale — fail loud rather than ship it. A 249 # large *magnitude* is fine: a legitimate base-period difference can 250 # need a 50x factor, so only sign and finiteness are guarded. 251 if not (math.isfinite(factor) and factor > 0): 252 raise ValueError( 253 f"splice: rebase factor for segment {i} ('{seg.name}') is {factor} over " 254 f"{lo}..{hi}. Multiplicative rebasing needs ratio-scale inputs (meaningful " 255 f"zero, proportional discrepancy); a non-finite or non-positive factor means " 256 f"the segments cross zero or differ additively. Pass rebase=False to coalesce " 257 f"raw levels instead." 258 ) 259 else: 260 factor, method, n, lo, hi = 1.0, "off", 0, None, None 261 seg_rebased = seg * factor 262 rows.append( 263 { 264 "segment": i, 265 "name": str(seg.name), 266 "freq_in": str(_pidx(segments[i]).freqstr), 267 "method": method, 268 "overlap_n": n, 269 "window_start": str(lo) if lo is not None else "", 270 "window_end": str(hi) if hi is not None else "", 271 "factor": round(factor, 6), 272 "fills_from": str(seg.dropna().index.min()), 273 } 274 ) 275 result = result.combine_first(seg_rebased) 276 277 # By default keep only the periods that actually carry data: do NOT reindex 278 # onto a dense grid (which would manufacture NaN for the gaps a coarse 279 # back-history leaves on a finer grid) and do NOT interpolate. A long-run 280 # series therefore stays sparse where it is old and coarse, and plots as one 281 # continuous line with no holes and no invented points. 282 result = result.dropna().sort_index() 283 284 if output and output != grid: 285 result = _to_grid(result, output, agg).dropna().sort_index() 286 grid = output 287 288 if fill in ("ffill", "interpolate") and len(result): 289 # Explicit opt-in: densify onto the full grid, then fill. 290 full = pd.period_range(result.index.min(), result.index.max(), freq=grid) 291 result = result.reindex(full) 292 result = result.ffill() if fill == "ffill" else result.interpolate() 293 294 result.name = name or str(segments[0].name) 295 report = DataFrame(rows) 296 return result, report
Splice mixed-frequency segments into one series, highest priority first.
Parameters
segments
Ordered list of pandas Series (PeriodIndex or DatetimeIndex). The
first is highest priority: it wins where periods overlap and (when
rebase is on) sets the level everything else is rebased to.
target
Common-grid frequency (e.g. "M", "Q-DEC"). Defaults to the
finest frequency present (anchor clashes step one rank finer).
rebase
Off by default — segments are coalesced at their raw levels, with no
silent transformation of your data. Set True to multiplicatively
rescale each lower-priority segment to the running result's level before
coalescing. Rebasing assumes ratio-scale inputs (meaningful zero,
proportional discrepancy between segments) — splicing index series on
different base periods (CPI, price/volume indices) is the case that
needs it. It is wrong for zero-crossing series (rates, balances) or
additive level breaks, and it invents a correction when same-unit
segments already agree — which is why it is opt-in. A non-finite or
non-positive factor raises. See the module docstring's rebase step.
agg
Aggregator used when a segment is finer than the grid (or when
downsampling to output). "mean" for index levels; use "sum"
for flows.
output
Optional final frequency to resample the spliced result to.
fill
Optional gap fill. By default (None) the result contains only the
periods that actually have data — no NaN rows are inserted for the gaps
a coarse segment leaves on a finer grid, and nothing is interpolated.
"ffill" or "interpolate" densify the result onto the full grid
first and then fill.
name
Name for the result series (defaults to the first segment's name).
Returns
tuple[Series, DataFrame] The spliced series and a one-row-per-junction report.
305def select_one(data: dict[str, DataFrame], meta: DataFrame, selector: dict[str, str] | str) -> Series: 306 """Select the single Series for one ``(data, meta, selector)`` — the single-source wrapper. 307 308 Convenience for the common one-selector case; equivalent to 309 ``select([(data, meta, selector)])[0]``. The *selector* is either a 310 ``{search_value: meta_column}`` dict for ``find_abs_id``, or a bare ABS 311 Series ID string, matched exactly against the metadata's Series ID column. 312 Returns the Series named by its Series ID, with its ABS unit on 313 ``.attrs["unit"]``. 314 """ 315 if isinstance(selector, str): 316 # A bare Series ID — same find_abs_id machinery, but exact-match on the 317 # Series ID column so one ID cannot substring-match another. 318 try: 319 table, series_id, unit = find_abs_id(meta, {selector: mc.id}, exact_match=True, validate_unique=True) 320 except ValueError as exc: 321 raise ValueError(f"select: series ID {selector!r} not found in the supplied metadata.") from exc 322 else: 323 table, series_id, unit = find_abs_id(meta, selector, validate_unique=True) 324 s = data[table][series_id].copy() 325 s.name = series_id 326 s.attrs["unit"] = str(unit) 327 return s
Select the single Series for one (data, meta, selector) — the single-source wrapper.
Convenience for the common one-selector case; equivalent to
select([(data, meta, selector)])[0]. The selector is either a
{search_value: meta_column} dict for find_abs_id, or a bare ABS
Series ID string, matched exactly against the metadata's Series ID column.
Returns the Series named by its Series ID, with its ABS unit on
.attrs["unit"].
330def select(sources: Iterable[Source], *, require_same_units: bool = True) -> list[Series]: 331 """Select a series for each ``(data, meta, selector)`` — the iterable in, iterable out. 332 333 The composable selection primitive: takes the iterable of ``(data, meta, 334 selector)`` sources and returns the matching list of Series, ready to hand to 335 :func:`splice` (directly, or after a per-series transform). Each selection 336 goes through ``readabs.find_abs_id`` with ``validate_unique=True``, which 337 de-duplicates on Series ID first — so a selector matching the same series in 338 several tables resolves cleanly, while one matching two genuinely different 339 series raises rather than guessing. 340 341 Parameters 342 ---------- 343 sources 344 Iterable of ``(data, meta, selector)``: 345 346 - ``data`` — ``dict[table_name, DataFrame]`` from ``read_abs_cat``. 347 - ``meta`` — the matching metadata DataFrame. 348 - ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``, e.g. 349 ``{"Index Numbers ; All groups CPI ; Australia ;": mc.did, 350 "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series 351 ID string (e.g. ``"A2325846C"``), matched exactly. 352 require_same_units 353 If ``True`` (default) **raise** when the selected series do not all share 354 the same ABS unit — units must cohere to be spliced. Set ``False`` when 355 you deliberately select different-unit series together (e.g. two counts 356 and a rate that you will combine yourself). 357 358 Returns 359 ------- 360 list[Series] 361 One Series per source, each named by its Series ID with its ABS unit in 362 ``series.attrs["unit"]``. Unpack it (``a, b = select([...])``), map a 363 transform over it, or pass it straight to :func:`splice`. A later 364 transform drops the unit attr — correctly, since the unit is then no 365 longer the ABS one. 366 367 Raises 368 ------ 369 ValueError 370 If ``require_same_units`` and the selected series carry mixed units. 371 372 """ 373 segments = [select_one(data, meta, selector) for data, meta, selector in sources] 374 if require_same_units: 375 units = [str(s.attrs.get("unit", "")) for s in segments] 376 if len(set(units)) > 1: 377 detail = ", ".join(f"{s.name}={u!r}" for s, u in zip(segments, units, strict=True)) 378 raise ValueError( 379 f"select: selected series have mismatched units ({detail}). Pass " 380 f"require_same_units=False to select different-unit series together." 381 ) 382 return segments
Select a series for each (data, meta, selector) — the iterable in, iterable out.
The composable selection primitive: takes the iterable of (data, meta,
selector) sources and returns the matching list of Series, ready to hand to
splice() (directly, or after a per-series transform). Each selection
goes through readabs.find_abs_id with validate_unique=True, which
de-duplicates on Series ID first — so a selector matching the same series in
several tables resolves cleanly, while one matching two genuinely different
series raises rather than guessing.
Parameters
sources
Iterable of (data, meta, selector):
- ``data`` — ``dict[table_name, DataFrame]`` from ``read_abs_cat``.
- ``meta`` — the matching metadata DataFrame.
- ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``, e.g.
``{"Index Numbers ; All groups CPI ; Australia ;": mc.did,
"Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series
ID string (e.g. ``"A2325846C"``), matched exactly.
require_same_units
If True (default) raise when the selected series do not all share
the same ABS unit — units must cohere to be spliced. Set False when
you deliberately select different-unit series together (e.g. two counts
and a rate that you will combine yourself).
Returns
list[Series]
One Series per source, each named by its Series ID with its ABS unit in
series.attrs["unit"]. Unpack it (a, b = select([...])), map a
transform over it, or pass it straight to splice(). A later
transform drops the unit attr — correctly, since the unit is then no
longer the ABS one.
Raises
ValueError
If require_same_units and the selected series carry mixed units.
385def select_and_splice( 386 sources: Iterable[Source], 387 *, 388 target: str | None = None, 389 rebase: bool = False, 390 agg: str = "mean", 391 output: str | None = None, 392 fill: Literal["ffill", "interpolate"] | None = None, 393 name: str | None = None, 394 require_same_units: bool = True, 395) -> tuple[Series, str, DataFrame]: 396 """Select one series per source and :func:`splice` them — the no-transform case. 397 398 Sugar for ``splice(select(sources))`` with a unit guard. When 399 you need a transform *between* selecting and splicing (e.g. a growth rate), 400 compose :func:`select` and :func:`splice` directly instead — that is the whole 401 reason :func:`select` is exposed separately. 402 403 Parameters 404 ---------- 405 sources 406 Ordered iterable of ``(data, meta, selector)``, **highest priority 407 first** (same priority rule as :func:`splice`): 408 409 - ``data`` — ``dict[table_name, DataFrame]`` from ``read_abs_cat``. 410 - ``meta`` — the matching metadata DataFrame. 411 - ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``, 412 e.g. ``{"Index Numbers ; All groups CPI ; Australia ;": mc.did, 413 "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series 414 ID string (e.g. ``"A2325846C"``), matched exactly. In the common case 415 the only thing differing between two sources is the frequency, so a 416 shared *base* selector composes with ``base | {"Quarter": mc.freq}``. 417 target, rebase, agg, output, fill, name 418 Passed straight through to :func:`splice`. 419 require_same_units 420 Forwarded to :func:`select`: if ``True`` (default) raise when the 421 selected segments carry mixed units; ``False`` overrides (the result is 422 then labelled with the highest-priority segment's unit). 423 424 Returns 425 ------- 426 tuple[Series, str, DataFrame] 427 The spliced series, its unit (the highest-priority segment's unit), and 428 the :func:`splice` join report, augmented with ``series_id`` and 429 ``unit`` columns recording what each segment resolved to. 430 431 """ 432 segments = select(sources, require_same_units=require_same_units) 433 units = [str(s.attrs.get("unit", "")) for s in segments] 434 435 result, report = splice(segments, target=target, rebase=rebase, agg=agg, output=output, fill=fill, name=name) 436 # Audit trail: which Series ID / unit did each reported (lower-priority) segment use? 437 if len(report): 438 seg = [int(i) for i in report["segment"]] 439 report.insert(1, "series_id", [str(segments[i].name) for i in seg]) 440 report.insert(2, "unit", [units[i] for i in seg]) 441 return result, units[0], report
Select one series per source and splice() them — the no-transform case.
Sugar for splice(select(sources)) with a unit guard. When
you need a transform between selecting and splicing (e.g. a growth rate),
compose select() and splice() directly instead — that is the whole
reason select() is exposed separately.
Parameters
sources
Ordered iterable of (data, meta, selector), highest priority
first (same priority rule as splice()):
- ``data`` — ``dict[table_name, DataFrame]`` from ``read_abs_cat``.
- ``meta`` — the matching metadata DataFrame.
- ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``,
e.g. ``{"Index Numbers ; All groups CPI ; Australia ;": mc.did,
"Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series
ID string (e.g. ``"A2325846C"``), matched exactly. In the common case
the only thing differing between two sources is the frequency, so a
shared *base* selector composes with ``base | {"Quarter": mc.freq}``.
target, rebase, agg, output, fill, name
Passed straight through to splice().
require_same_units
Forwarded to select(): if True (default) raise when the
selected segments carry mixed units; False overrides (the result is
then labelled with the highest-priority segment's unit).
Returns
tuple[Series, str, DataFrame]
The spliced series, its unit (the highest-priority segment's unit), and
the splice() join report, augmented with series_id and
unit columns recording what each segment resolved to.