readabs.splice

Priority splicing of mixed-frequency time series.

This module has two layers:

splice The core primitive. Deliberately source-agnostic: it takes pandas Series you have already fetched (by description, by ID, however you like) and splices them into one series. It knows nothing about the ABS, ships no static lookup table, and makes no guesses about which series belong together — that judgement stays with the caller.

select / select_one / select_and_splice A thin ABS-aware convenience layer over splice. Each resolves (data, meta, selector) sources to Series via readabs.find_abs_id (carrying each series' ABS unit on .attrs["unit"]), so the common case — splice a few ABS series selected by description/frequency — is one call, while select stays exposed for when you need a transform between selecting and splicing. A selector is either a {search_value: meta_column} dict, or a bare ABS Series ID string when you already know exactly which series you want.

Splice design

Given an ordered list of segments (highest priority / most authoritative first), splice():

  1. align — put every segment on one common PeriodIndex. By default the grid is the finest frequency present, which dissolves anchor clashes (Q-NOV vs Q-DEC, A-JUN vs A-DEC) because every coarse period maps cleanly onto a finer one. Coarser segments are placed at their period-end; finer segments are aggregated down with agg.
  2. rebase(opt-in; off by default) for each junction, multiplicatively scale the lower-priority segment so its level matches the running result over the overlapping date window (phase-agnostic; works even when two series never share an exact period). Falls back to a single junction point if there is no overlap, and flags it. Off by default because it transforms your data — nothing is silently rescaled unless you ask.

             Rebasing assumes **ratio-scale** inputs — series whose zero is
             meaningful and whose discrepancy between segments is
             *proportional*.  Indexes (CPI, price/volume indices on different
             base periods) are the canonical case; a proportional benchmark
             revision of a count works too.  It is **wrong** for series that
             cross zero (rates of change, balances, net flows) or whose
             segments differ by an *additive* offset rather than a scale
             factor — a negative or non-finite factor is caught and raises.
             With ``rebase=False`` (the default) the raw levels are coalesced
             as-is: if two same-unit segments already agree, rebasing only
             invents a discrepancy to "correct".
    
  3. coalescecombine_first down the priority chain: take segment 1, fill gaps from segment 2, then 3, ... The result keeps only the periods that actually carry data — a coarse back-history stays sparse on a finer grid rather than being NaN-filled, and nothing is interpolated (pass fill= to densify).
  4. resample— (optional) resample the spliced result to a chosen output frequency/anchor.

The returned join report makes every rebase factor and overlap visible, so a splice can be audited rather than trusted blindly.

  1"""Priority splicing of mixed-frequency time series.
  2
  3This module has two layers:
  4
  5``splice``
  6    The core primitive.  Deliberately *source-agnostic*: it takes pandas Series
  7    you have already fetched (by description, by ID, however you like) and
  8    splices them into one series.  It knows nothing about the ABS, ships no
  9    static lookup table, and makes no guesses about which series belong together
 10    — that judgement stays with the caller.
 11
 12``select`` / ``select_one`` / ``select_and_splice``
 13    A thin ABS-aware convenience layer over ``splice``.  Each resolves
 14    ``(data, meta, selector)`` sources to Series via ``readabs.find_abs_id``
 15    (carrying each series' ABS unit on ``.attrs["unit"]``), so the common case —
 16    splice a few ABS series selected by description/frequency — is one call,
 17    while ``select`` stays exposed for when you need a transform between
 18    selecting and splicing.  A *selector* is either a ``{search_value:
 19    meta_column}`` dict, or a bare ABS Series ID string when you already know
 20    exactly which series you want.
 21
 22Splice design
 23-------------
 24Given an ordered list of segments (highest priority / most authoritative
 25first), :func:`splice`:
 26
 271. **align**   — put every segment on one common ``PeriodIndex``.  By default
 28                 the grid is the *finest* frequency present, which dissolves
 29                 anchor clashes (Q-NOV vs Q-DEC, A-JUN vs A-DEC) because every
 30                 coarse period maps cleanly onto a finer one.  Coarser segments
 31                 are placed at their period-*end*; finer segments are
 32                 aggregated down with ``agg``.
 332. **rebase**  — *(opt-in; off by default)* for each junction,
 34                 *multiplicatively* scale the lower-priority segment so its level
 35                 matches the running result over the *overlapping date window*
 36                 (phase-agnostic; works even when two series never share an exact
 37                 period).  Falls back to a single junction point if there is no
 38                 overlap, and flags it.  Off by default because it transforms
 39                 your data — nothing is silently rescaled unless you ask.
 40
 41                 Rebasing assumes **ratio-scale** inputs — series whose zero is
 42                 meaningful and whose discrepancy between segments is
 43                 *proportional*.  Indexes (CPI, price/volume indices on different
 44                 base periods) are the canonical case; a proportional benchmark
 45                 revision of a count works too.  It is **wrong** for series that
 46                 cross zero (rates of change, balances, net flows) or whose
 47                 segments differ by an *additive* offset rather than a scale
 48                 factor — a negative or non-finite factor is caught and raises.
 49                 With ``rebase=False`` (the default) the raw levels are coalesced
 50                 as-is: if two same-unit segments already agree, rebasing only
 51                 invents a discrepancy to "correct".
 523. **coalesce**— ``combine_first`` down the priority chain: take segment 1,
 53                 fill gaps from segment 2, then 3, ...  The result keeps only
 54                 the periods that actually carry data — a coarse back-history
 55                 stays sparse on a finer grid rather than being NaN-filled, and
 56                 nothing is interpolated (pass ``fill=`` to densify).
 574. **resample**— (optional) resample the spliced result to a chosen output
 58                 frequency/anchor.
 59
 60The returned join report makes every rebase factor and overlap visible, so a
 61splice can be audited rather than trusted blindly.
 62"""
 63
 64from __future__ import annotations
 65
 66import math
 67from collections.abc import Iterable, Sequence
 68from typing import Literal, cast
 69
 70import pandas as pd
 71from pandas import DataFrame, PeriodIndex, Series
 72
 73from readabs.abs_meta_data import metacol as mc  # used by the select() layer
 74from readabs.search_abs_meta import find_abs_id  # used by the select() layer
 75
 76# Frequency rank — higher number = finer frequency.
 77_FREQ_RANK: dict[str, int] = {"Y": 0, "A": 0, "Q": 1, "M": 2, "W": 3, "D": 4}
 78
 79
 80def _base(freqstr: str) -> str:
 81    """Return the base frequency character (``"Q-NOV"`` -> ``"Q"``, ``"A-JUN"`` -> ``"Y"``)."""
 82    char = freqstr.split("-", maxsplit=1)[0][0].upper()
 83    return "Y" if char == "A" else char
 84
 85
 86def _rank(freqstr: str) -> int:
 87    """Return the frequency rank for a PeriodIndex freq string."""
 88    return _FREQ_RANK[_base(freqstr)]
 89
 90
 91def _as_period_index(s: Series) -> Series:
 92    """Ensure *s* has a PeriodIndex; convert from DatetimeIndex if needed."""
 93    if isinstance(s.index, PeriodIndex):
 94        return s
 95    if isinstance(s.index, pd.DatetimeIndex):
 96        return s.set_axis(s.index.to_period())
 97    raise TypeError(f"Series '{s.name}' must have a PeriodIndex or DatetimeIndex, got {type(s.index).__name__}.")
 98
 99
100def _pidx(s: Series) -> PeriodIndex:
101    """Return *s*'s index as a (typed) PeriodIndex, converting if necessary."""
102    return cast("PeriodIndex", _as_period_index(s).index)
103
104
105def _pick_target(segments: Sequence[Series]) -> str:
106    """Choose the default common-grid freq: the finest present.
107
108    If two or more segments share the *finest* rank but with different anchors
109    (e.g. ``Q-NOV`` and ``Q-DEC``) and there is nothing finer to splice them
110    onto, raise — picking one anchor would silently reanchor the other and
111    could assume wrong.  Resolve it by passing a finer ``target`` (e.g.
112    ``"M"``), or by including a finer-frequency segment.
113    """
114    freqs = [str(_pidx(s).freqstr) for s in segments]
115    ranks = [_rank(f) for f in freqs]
116    top = max(ranks)
117    top_freqs = {f for f, r in zip(freqs, ranks, strict=True) if r == top}
118    if len(top_freqs) > 1:
119        raise ValueError(
120            f"Clashing anchors at the finest frequency: {sorted(top_freqs)}. "
121            f"Pass a finer target (e.g. target='M') to splice them on a common grid."
122        )
123    return next(iter(top_freqs))
124
125
126def _to_grid(s: Series, target: str, agg: str) -> Series:
127    """Map *s* onto the *target* PeriodIndex frequency.
128
129    Finer-than-target segments are aggregated down with *agg*; equal-or-coarser
130    segments are placed at their period-end on the target grid.
131    """
132    s = _as_period_index(s).dropna()
133    idx = cast("PeriodIndex", s.index)
134    src = str(idx.freqstr)
135    if _rank(src) > _rank(target):
136        # finer -> coarser: aggregate the sub-periods that fall in each target period
137        out = s.groupby(idx.asfreq(target)).agg(agg)
138    elif _rank(src) == _rank(target) and _base(src) == _base(target) and src != target:
139        # same frequency, different anchor (e.g. Q-NOV vs Q-DEC) — reanchoring
140        # would silently shift every period, so refuse rather than assume.
141        raise ValueError(
142            f"Cannot place '{s.name}' ({src}) onto a {target} grid without reanchoring. "
143            f"Use a finer target (e.g. target='M')."
144        )
145    else:
146        # coarser (or identical) -> place each value at its period-end on the grid
147        out = Series(s.to_numpy(), index=idx.asfreq(target, how="E"), name=s.name)
148        out = out[~out.index.duplicated(keep="last")]
149    return out.sort_index()
150
151
152def _rebase_factor(result: Series, seg: Series) -> tuple[float, str, int, pd.Period | None, pd.Period | None]:
153    """Compute the factor to bring *seg* onto *result*'s level.
154
155    Measured as the ratio of mean levels over the overlapping *date span*, so
156    it is phase-agnostic — it works even when the two series share no exact
157    period (e.g. Q-NOV vs Q-DEC mapped onto a monthly grid).  Falls back to a
158    single junction point when the spans do not overlap at all.
159
160    Returns ``(factor, method, overlap_n, window_start, window_end)``.
161    """
162    r, s = result.dropna(), seg.dropna()
163    if len(r) and len(s):
164        lo = max(r.index.min(), s.index.min())
165        hi = min(r.index.max(), s.index.max())
166        if lo <= hi:
167            r_win, s_win = r.loc[lo:hi], s.loc[lo:hi]
168            if len(r_win) and len(s_win) and s_win.mean():
169                return float(r_win.mean() / s_win.mean()), "window", min(len(r_win), len(s_win)), lo, hi
170    # No overlapping span — fall back to the nearest junction point.
171    r0 = result.first_valid_index()
172    if r0 is not None:
173        before = s.loc[:r0]
174        if len(before) and before.iloc[-1]:
175            return float(result.loc[r0] / before.iloc[-1]), "junction", 0, None, None
176    return 1.0, "none", 0, None, None
177
178
179def splice(
180    segments: Iterable[Series],
181    *,
182    target: str | None = None,
183    rebase: bool = False,
184    agg: str = "mean",
185    output: str | None = None,
186    fill: Literal["ffill", "interpolate"] | None = None,
187    name: str | None = None,
188) -> tuple[Series, DataFrame]:
189    """Splice mixed-frequency *segments* into one series, highest priority first.
190
191    Parameters
192    ----------
193    segments
194        Ordered list of pandas Series (PeriodIndex or DatetimeIndex).  The
195        first is highest priority: it wins where periods overlap and (when
196        ``rebase`` is on) sets the level everything else is rebased to.
197    target
198        Common-grid frequency (e.g. ``"M"``, ``"Q-DEC"``).  Defaults to the
199        finest frequency present (anchor clashes step one rank finer).
200    rebase
201        Off by default — segments are coalesced at their **raw** levels, with no
202        silent transformation of your data.  Set ``True`` to *multiplicatively*
203        rescale each lower-priority segment to the running result's level before
204        coalescing.  Rebasing assumes **ratio-scale** inputs (meaningful zero,
205        proportional discrepancy between segments) — splicing index series on
206        different base periods (CPI, price/volume indices) is the case that
207        needs it.  It is wrong for zero-crossing series (rates, balances) or
208        additive level breaks, and it *invents* a correction when same-unit
209        segments already agree — which is why it is opt-in.  A non-finite or
210        non-positive factor raises.  See the module docstring's *rebase* step.
211    agg
212        Aggregator used when a segment is finer than the grid (or when
213        downsampling to *output*).  ``"mean"`` for index levels; use ``"sum"``
214        for flows.
215    output
216        Optional final frequency to resample the spliced result to.
217    fill
218        Optional gap fill.  By default (``None``) the result contains only the
219        periods that actually have data — no NaN rows are inserted for the gaps
220        a coarse segment leaves on a finer grid, and nothing is interpolated.
221        ``"ffill"`` or ``"interpolate"`` densify the result onto the full grid
222        first and then fill.
223    name
224        Name for the result series (defaults to the first segment's name).
225
226    Returns
227    -------
228    tuple[Series, DataFrame]
229        The spliced series and a one-row-per-junction report.
230
231    """
232    segments = list(segments)
233    if not segments:
234        raise ValueError("splice() needs at least one segment.")
235
236    grid = target or _pick_target(segments)
237    on_grid = [_to_grid(s, grid, agg) for s in segments]
238
239    result = on_grid[0].copy()
240    rows: list[dict[str, object]] = []
241    for i, seg in enumerate(on_grid[1:], start=1):
242        if rebase:
243            factor, method, n, lo, hi = _rebase_factor(result, seg)
244            # Multiplicative rebasing assumes ratio-scale inputs.  A non-finite
245            # factor (near-zero denominator) or a non-positive one (the overlap
246            # means have opposite signs, which would flip the back-history) means
247            # the data is not ratio-scale — fail loud rather than ship it.  A
248            # large *magnitude* is fine: a legitimate base-period difference can
249            # need a 50x factor, so only sign and finiteness are guarded.
250            if not (math.isfinite(factor) and factor > 0):
251                raise ValueError(
252                    f"splice: rebase factor for segment {i} ('{seg.name}') is {factor} over "
253                    f"{lo}..{hi}. Multiplicative rebasing needs ratio-scale inputs (meaningful "
254                    f"zero, proportional discrepancy); a non-finite or non-positive factor means "
255                    f"the segments cross zero or differ additively. Pass rebase=False to coalesce "
256                    f"raw levels instead."
257                )
258        else:
259            factor, method, n, lo, hi = 1.0, "off", 0, None, None
260        seg_rebased = seg * factor
261        rows.append(
262            {
263                "segment": i,
264                "name": str(seg.name),
265                "freq_in": str(_pidx(segments[i]).freqstr),
266                "method": method,
267                "overlap_n": n,
268                "window_start": str(lo) if lo is not None else "",
269                "window_end": str(hi) if hi is not None else "",
270                "factor": round(factor, 6),
271                "fills_from": str(seg.dropna().index.min()),
272            }
273        )
274        result = result.combine_first(seg_rebased)
275
276    # By default keep only the periods that actually carry data: do NOT reindex
277    # onto a dense grid (which would manufacture NaN for the gaps a coarse
278    # back-history leaves on a finer grid) and do NOT interpolate.  A long-run
279    # series therefore stays sparse where it is old and coarse, and plots as one
280    # continuous line with no holes and no invented points.
281    result = result.dropna().sort_index()
282
283    if output and output != grid:
284        result = _to_grid(result, output, agg).dropna().sort_index()
285        grid = output
286
287    if fill in ("ffill", "interpolate") and len(result):
288        # Explicit opt-in: densify onto the full grid, then fill.
289        full = pd.period_range(result.index.min(), result.index.max(), freq=grid)
290        result = result.reindex(full)
291        result = result.ffill() if fill == "ffill" else result.interpolate()
292
293    result.name = name or str(segments[0].name)
294    report = DataFrame(rows)
295    return result, report
296
297
298# A select_and_splice() source: the fetched data dict, its meta, and either a
299# {search_value: meta_column} selector (readabs' find_abs_id convention) or a
300# bare ABS Series ID string (matched exactly against the Series ID column).
301Source = tuple[dict[str, DataFrame], DataFrame, dict[str, str] | str]
302
303
304def select_one(data: dict[str, DataFrame], meta: DataFrame, selector: dict[str, str] | str) -> Series:
305    """Select the single Series for one ``(data, meta, selector)`` — the single-source wrapper.
306
307    Convenience for the common one-selector case; equivalent to
308    ``select([(data, meta, selector)])[0]``.  The *selector* is either a
309    ``{search_value: meta_column}`` dict for ``find_abs_id``, or a bare ABS
310    Series ID string, matched exactly against the metadata's Series ID column.
311    Returns the Series named by its Series ID, with its ABS unit on
312    ``.attrs["unit"]``.
313    """
314    if isinstance(selector, str):
315        # A bare Series ID — same find_abs_id machinery, but exact-match on the
316        # Series ID column so one ID cannot substring-match another.
317        try:
318            table, series_id, unit = find_abs_id(meta, {selector: mc.id}, exact_match=True, validate_unique=True)
319        except ValueError as exc:
320            raise ValueError(f"select: series ID {selector!r} not found in the supplied metadata.") from exc
321    else:
322        table, series_id, unit = find_abs_id(meta, selector, validate_unique=True)
323    s = data[table][series_id].copy()
324    s.name = series_id
325    s.attrs["unit"] = str(unit)
326    return s
327
328
329def select(sources: Iterable[Source], *, require_same_units: bool = True) -> list[Series]:
330    """Select a series for each ``(data, meta, selector)`` — the iterable in, iterable out.
331
332    The composable selection primitive: takes the iterable of ``(data, meta,
333    selector)`` sources and returns the matching list of Series, ready to hand to
334    :func:`splice` (directly, or after a per-series transform).  Each selection
335    goes through ``readabs.find_abs_id`` with ``validate_unique=True``, which
336    de-duplicates on Series ID first — so a selector matching the same series in
337    several tables resolves cleanly, while one matching two genuinely different
338    series raises rather than guessing.
339
340    Parameters
341    ----------
342    sources
343        Iterable of ``(data, meta, selector)``:
344
345        - ``data``   — ``dict[table_name, DataFrame]`` from ``read_abs_cat``.
346        - ``meta``   — the matching metadata DataFrame.
347        - ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``, e.g.
348          ``{"Index Numbers ;  All groups CPI ;  Australia ;": mc.did,
349          "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series
350          ID string (e.g. ``"A2325846C"``), matched exactly.
351    require_same_units
352        If ``True`` (default) **raise** when the selected series do not all share
353        the same ABS unit — units must cohere to be spliced.  Set ``False`` when
354        you deliberately select different-unit series together (e.g. two counts
355        and a rate that you will combine yourself).
356
357    Returns
358    -------
359    list[Series]
360        One Series per source, each named by its Series ID with its ABS unit in
361        ``series.attrs["unit"]``.  Unpack it (``a, b = select([...])``), map a
362        transform over it, or pass it straight to :func:`splice`.  A later
363        transform drops the unit attr — correctly, since the unit is then no
364        longer the ABS one.
365
366    Raises
367    ------
368    ValueError
369        If ``require_same_units`` and the selected series carry mixed units.
370
371    """
372    segments = [select_one(data, meta, selector) for data, meta, selector in sources]
373    if require_same_units:
374        units = [str(s.attrs.get("unit", "")) for s in segments]
375        if len(set(units)) > 1:
376            detail = ", ".join(f"{s.name}={u!r}" for s, u in zip(segments, units, strict=True))
377            raise ValueError(
378                f"select: selected series have mismatched units ({detail}). Pass "
379                f"require_same_units=False to select different-unit series together."
380            )
381    return segments
382
383
384def select_and_splice(
385    sources: Iterable[Source],
386    *,
387    target: str | None = None,
388    rebase: bool = False,
389    agg: str = "mean",
390    output: str | None = None,
391    fill: Literal["ffill", "interpolate"] | None = None,
392    name: str | None = None,
393    require_same_units: bool = True,
394) -> tuple[Series, str, DataFrame]:
395    """Select one series per source and :func:`splice` them — the no-transform case.
396
397    Sugar for ``splice(select(sources))`` with a unit guard.  When
398    you need a transform *between* selecting and splicing (e.g. a growth rate),
399    compose :func:`select` and :func:`splice` directly instead — that is the whole
400    reason :func:`select` is exposed separately.
401
402    Parameters
403    ----------
404    sources
405        Ordered iterable of ``(data, meta, selector)``, **highest priority
406        first** (same priority rule as :func:`splice`):
407
408        - ``data``   — ``dict[table_name, DataFrame]`` from ``read_abs_cat``.
409        - ``meta``   — the matching metadata DataFrame.
410        - ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``,
411          e.g. ``{"Index Numbers ;  All groups CPI ;  Australia ;": mc.did,
412          "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series
413          ID string (e.g. ``"A2325846C"``), matched exactly.  In the common case
414          the only thing differing between two sources is the frequency, so a
415          shared *base* selector composes with ``base | {"Quarter": mc.freq}``.
416    target, rebase, agg, output, fill, name
417        Passed straight through to :func:`splice`.
418    require_same_units
419        Forwarded to :func:`select`: if ``True`` (default) raise when the
420        selected segments carry mixed units; ``False`` overrides (the result is
421        then labelled with the highest-priority segment's unit).
422
423    Returns
424    -------
425    tuple[Series, str, DataFrame]
426        The spliced series, its unit (the highest-priority segment's unit), and
427        the :func:`splice` join report, augmented with ``series_id`` and
428        ``unit`` columns recording what each segment resolved to.
429
430    """
431    segments = select(sources, require_same_units=require_same_units)
432    units = [str(s.attrs.get("unit", "")) for s in segments]
433
434    result, report = splice(segments, target=target, rebase=rebase, agg=agg, output=output, fill=fill, name=name)
435    # Audit trail: which Series ID / unit did each reported (lower-priority) segment use?
436    if len(report):
437        seg = [int(i) for i in report["segment"]]
438        report.insert(1, "series_id", [str(segments[i].name) for i in seg])
439        report.insert(2, "unit", [units[i] for i in seg])
440    return result, units[0], report
441
442
443# ---------------------------------------------------------------------------
444# Self-tests — `python splice.py`
445# ---------------------------------------------------------------------------
446if __name__ == "__main__":
447    import numpy as np
448
449    def _show(title: str, s: Series, rep: DataFrame) -> None:
450        print(f"\n{'=' * 70}\n{title}\n{'=' * 70}")
451        print(
452            f"freq={cast('PeriodIndex', s.index).freqstr}  n={len(s)}  non-null={s.notna().sum()}  "
453            f"range={s.index.min()}..{s.index.max()}"
454        )
455        if len(rep):
456            print(rep.to_string(index=False))
457
458    # --- Case 1: monthly (new) + quarterly (old), level shift via index rebase
459    q = Series(
460        np.arange(100, 100 + 4 * 20, dtype=float),  # 20 years quarterly, base ~100
461        index=pd.period_range("2000Q1", periods=80, freq="Q-DEC"),
462        name="cpi",
463    )
464    m = Series(
465        np.arange(50.0, 50.0 + 60) * 0.5 + 130,  # monthly on a *different* base
466        index=pd.period_range("2018-01", periods=60, freq="M"),
467        name="cpi",
468    )
469    out, rep = splice([m, q], rebase=True)  # monthly priority, quarterly fills the back-history
470    _show("Case 1 — M (priority) spliced with Q-DEC, auto-grid", out, rep)
471    print(f"check: rebased Q value at 2018-03 = {out.loc['2018-03']:.3f} (monthly 2018-01 = {m.iloc[0]:.3f})")
472
473    # --- Case 2: the anchor clash — Q-NOV vs Q-DEC, overlapping in time
474    q_dec = Series(
475        np.arange(200.0, 200 + 40),
476        index=pd.period_range("2010Q1", periods=40, freq="Q-DEC"),
477        name="x",
478    )
479    q_nov = Series(
480        np.arange(80.0, 80 + 60),  # 2000Q1..2014Q4 — overlaps q_dec over 2010-2014
481        index=pd.period_range("2000Q1", periods=60, freq="Q-NOV"),
482        name="x",
483    )
484    print(f"\n{'=' * 70}\nCase 2 — Q-DEC + Q-NOV anchor clash\n{'=' * 70}")
485    try:
486        splice([q_dec, q_nov])  # no target -> must refuse rather than reanchor
487    except ValueError as exc:
488        print(f"default (no target) correctly raised:\n  {exc}")
489    out2, rep2 = splice([q_dec, q_nov], target="M", rebase=True)  # resolve on a common finer grid
490    _show("Case 2b — same, resolved with target='M' (window rebase across anchors)", out2, rep2)
491
492    # --- Case 3: daily + monthly.  Default grid is the finest present = D.
493    d = Series(
494        np.linspace(10, 12, 365),
495        index=pd.period_range("2023-01-01", periods=365, freq="D"),
496        name="rate",
497    )
498    mth = Series(
499        np.linspace(12, 13, 18),  # 2023-07..2024-12 — overlaps the daily over 2023-H2
500        index=pd.period_range("2023-07", periods=18, freq="M"),
501        name="rate",
502    )
503    out3, rep3 = splice([d, mth])  # daily priority -> finest grid = D, monthly placed sparsely
504    _show("Case 3 — D (priority) + M, default finest grid = D", out3, rep3)
505    out3b, rep3b = splice([mth, d], target="M", agg="mean")  # explicitly ask for a monthly result
506    _show("Case 3b — same data, target='M' so daily is aggregated down", out3b, rep3b)
507
508    # --- Case 4: CPI-style 3-way chain (new monthly + indicator + quarterly)
509    new_m = Series(np.arange(135.0, 135 + 12), index=pd.period_range("2024-01", periods=12, freq="M"), name="cpi")
510    indic = Series(np.arange(120.0, 120 + 30), index=pd.period_range("2022-07", periods=30, freq="M"), name="cpi")
511    old_q_index = pd.period_range("1995Q1", periods=120, freq="Q-DEC")
512    old_q = Series(np.arange(40.0, 40 + 120), index=old_q_index, name="cpi")
513    out4, rep4 = splice([new_m, indic, old_q], name="cpi_long", rebase=True)
514    _show("Case 4 — 3-way: new monthly + indicator + quarterly", out4, rep4)
515    print(
516        f"\nfull series spans {out4.index.min()} .. {out4.index.max()}, {out4.notna().sum()} observations present"
517    )
518
519    # --- Case 5: same, but ask for a clean quarterly output (downsample)
520    out5, rep5 = splice([new_m, indic, old_q], output="Q-DEC", name="cpi_long_q", rebase=True)
521    _show("Case 5 — same 3-way, resampled to a clean Q-DEC output", out5, rep5)
522
523    # --- Case 6: the select() layer — dict selector vs bare Series ID string
524    fake_meta = DataFrame(
525        {
526            mc.did: ["Index Numbers ;  All groups CPI ;  Australia ;"] * 2,
527            mc.id: ["A2325846C", "A128478317T"],
528            mc.unit: ["Index Numbers", "Index Numbers"],
529            mc.freq: ["Quarter", "Month"],
530            mc.table: ["640101", "648601"],
531        }
532    )
533    fake_data = {
534        "640101": DataFrame({"A2325846C": q.to_numpy()[:40]}, index=q.index[:40]),
535        "648601": DataFrame({"A128478317T": m.to_numpy()}, index=m.index),
536    }
537    by_id = select_one(fake_data, fake_meta, "A2325846C")  # bare Series ID string
538    by_dict = select_one(fake_data, fake_meta, {"Month": mc.freq})  # selector dict
539    print(f"\n{'=' * 70}\nCase 6 — select_one: bare Series ID vs selector dict\n{'=' * 70}")
540    print(f"by ID:   name={by_id.name} unit={by_id.attrs['unit']!r} n={len(by_id)}")
541    print(f"by dict: name={by_dict.name} unit={by_dict.attrs['unit']!r} n={len(by_dict)}")
542    out6, unit6, rep6 = select_and_splice(
543        [(fake_data, fake_meta, "A128478317T"), (fake_data, fake_meta, "A2325846C")], rebase=True
544    )
545    _show(f"Case 6b — select_and_splice by bare Series IDs (unit={unit6!r})", out6, rep6)
546    try:
547        select_one(fake_data, fake_meta, "NOSUCHID")  # unknown ID -> fail loud
548    except ValueError as exc:
549        print(f"unknown ID correctly raised:\n  {exc}")
550
551    print("\nAll cases ran.")
def splice( segments: Iterable[pandas.Series], *, target: str | None = None, rebase: bool = False, agg: str = 'mean', output: str | None = None, fill: Literal['ffill', 'interpolate'] | None = None, name: str | None = None) -> tuple[pandas.Series, pandas.DataFrame]:
180def splice(
181    segments: Iterable[Series],
182    *,
183    target: str | None = None,
184    rebase: bool = False,
185    agg: str = "mean",
186    output: str | None = None,
187    fill: Literal["ffill", "interpolate"] | None = None,
188    name: str | None = None,
189) -> tuple[Series, DataFrame]:
190    """Splice mixed-frequency *segments* into one series, highest priority first.
191
192    Parameters
193    ----------
194    segments
195        Ordered list of pandas Series (PeriodIndex or DatetimeIndex).  The
196        first is highest priority: it wins where periods overlap and (when
197        ``rebase`` is on) sets the level everything else is rebased to.
198    target
199        Common-grid frequency (e.g. ``"M"``, ``"Q-DEC"``).  Defaults to the
200        finest frequency present (anchor clashes step one rank finer).
201    rebase
202        Off by default — segments are coalesced at their **raw** levels, with no
203        silent transformation of your data.  Set ``True`` to *multiplicatively*
204        rescale each lower-priority segment to the running result's level before
205        coalescing.  Rebasing assumes **ratio-scale** inputs (meaningful zero,
206        proportional discrepancy between segments) — splicing index series on
207        different base periods (CPI, price/volume indices) is the case that
208        needs it.  It is wrong for zero-crossing series (rates, balances) or
209        additive level breaks, and it *invents* a correction when same-unit
210        segments already agree — which is why it is opt-in.  A non-finite or
211        non-positive factor raises.  See the module docstring's *rebase* step.
212    agg
213        Aggregator used when a segment is finer than the grid (or when
214        downsampling to *output*).  ``"mean"`` for index levels; use ``"sum"``
215        for flows.
216    output
217        Optional final frequency to resample the spliced result to.
218    fill
219        Optional gap fill.  By default (``None``) the result contains only the
220        periods that actually have data — no NaN rows are inserted for the gaps
221        a coarse segment leaves on a finer grid, and nothing is interpolated.
222        ``"ffill"`` or ``"interpolate"`` densify the result onto the full grid
223        first and then fill.
224    name
225        Name for the result series (defaults to the first segment's name).
226
227    Returns
228    -------
229    tuple[Series, DataFrame]
230        The spliced series and a one-row-per-junction report.
231
232    """
233    segments = list(segments)
234    if not segments:
235        raise ValueError("splice() needs at least one segment.")
236
237    grid = target or _pick_target(segments)
238    on_grid = [_to_grid(s, grid, agg) for s in segments]
239
240    result = on_grid[0].copy()
241    rows: list[dict[str, object]] = []
242    for i, seg in enumerate(on_grid[1:], start=1):
243        if rebase:
244            factor, method, n, lo, hi = _rebase_factor(result, seg)
245            # Multiplicative rebasing assumes ratio-scale inputs.  A non-finite
246            # factor (near-zero denominator) or a non-positive one (the overlap
247            # means have opposite signs, which would flip the back-history) means
248            # the data is not ratio-scale — fail loud rather than ship it.  A
249            # large *magnitude* is fine: a legitimate base-period difference can
250            # need a 50x factor, so only sign and finiteness are guarded.
251            if not (math.isfinite(factor) and factor > 0):
252                raise ValueError(
253                    f"splice: rebase factor for segment {i} ('{seg.name}') is {factor} over "
254                    f"{lo}..{hi}. Multiplicative rebasing needs ratio-scale inputs (meaningful "
255                    f"zero, proportional discrepancy); a non-finite or non-positive factor means "
256                    f"the segments cross zero or differ additively. Pass rebase=False to coalesce "
257                    f"raw levels instead."
258                )
259        else:
260            factor, method, n, lo, hi = 1.0, "off", 0, None, None
261        seg_rebased = seg * factor
262        rows.append(
263            {
264                "segment": i,
265                "name": str(seg.name),
266                "freq_in": str(_pidx(segments[i]).freqstr),
267                "method": method,
268                "overlap_n": n,
269                "window_start": str(lo) if lo is not None else "",
270                "window_end": str(hi) if hi is not None else "",
271                "factor": round(factor, 6),
272                "fills_from": str(seg.dropna().index.min()),
273            }
274        )
275        result = result.combine_first(seg_rebased)
276
277    # By default keep only the periods that actually carry data: do NOT reindex
278    # onto a dense grid (which would manufacture NaN for the gaps a coarse
279    # back-history leaves on a finer grid) and do NOT interpolate.  A long-run
280    # series therefore stays sparse where it is old and coarse, and plots as one
281    # continuous line with no holes and no invented points.
282    result = result.dropna().sort_index()
283
284    if output and output != grid:
285        result = _to_grid(result, output, agg).dropna().sort_index()
286        grid = output
287
288    if fill in ("ffill", "interpolate") and len(result):
289        # Explicit opt-in: densify onto the full grid, then fill.
290        full = pd.period_range(result.index.min(), result.index.max(), freq=grid)
291        result = result.reindex(full)
292        result = result.ffill() if fill == "ffill" else result.interpolate()
293
294    result.name = name or str(segments[0].name)
295    report = DataFrame(rows)
296    return result, report

Splice mixed-frequency segments into one series, highest priority first.

Parameters

segments Ordered list of pandas Series (PeriodIndex or DatetimeIndex). The first is highest priority: it wins where periods overlap and (when rebase is on) sets the level everything else is rebased to. target Common-grid frequency (e.g. "M", "Q-DEC"). Defaults to the finest frequency present (anchor clashes step one rank finer). rebase Off by default — segments are coalesced at their raw levels, with no silent transformation of your data. Set True to multiplicatively rescale each lower-priority segment to the running result's level before coalescing. Rebasing assumes ratio-scale inputs (meaningful zero, proportional discrepancy between segments) — splicing index series on different base periods (CPI, price/volume indices) is the case that needs it. It is wrong for zero-crossing series (rates, balances) or additive level breaks, and it invents a correction when same-unit segments already agree — which is why it is opt-in. A non-finite or non-positive factor raises. See the module docstring's rebase step. agg Aggregator used when a segment is finer than the grid (or when downsampling to output). "mean" for index levels; use "sum" for flows. output Optional final frequency to resample the spliced result to. fill Optional gap fill. By default (None) the result contains only the periods that actually have data — no NaN rows are inserted for the gaps a coarse segment leaves on a finer grid, and nothing is interpolated. "ffill" or "interpolate" densify the result onto the full grid first and then fill. name Name for the result series (defaults to the first segment's name).

Returns

tuple[Series, DataFrame] The spliced series and a one-row-per-junction report.

Source = tuple[dict[str, pandas.DataFrame], pandas.DataFrame, dict[str, str] | str]
def select_one( data: dict[str, pandas.DataFrame], meta: pandas.DataFrame, selector: dict[str, str] | str) -> pandas.Series:
305def select_one(data: dict[str, DataFrame], meta: DataFrame, selector: dict[str, str] | str) -> Series:
306    """Select the single Series for one ``(data, meta, selector)`` — the single-source wrapper.
307
308    Convenience for the common one-selector case; equivalent to
309    ``select([(data, meta, selector)])[0]``.  The *selector* is either a
310    ``{search_value: meta_column}`` dict for ``find_abs_id``, or a bare ABS
311    Series ID string, matched exactly against the metadata's Series ID column.
312    Returns the Series named by its Series ID, with its ABS unit on
313    ``.attrs["unit"]``.
314    """
315    if isinstance(selector, str):
316        # A bare Series ID — same find_abs_id machinery, but exact-match on the
317        # Series ID column so one ID cannot substring-match another.
318        try:
319            table, series_id, unit = find_abs_id(meta, {selector: mc.id}, exact_match=True, validate_unique=True)
320        except ValueError as exc:
321            raise ValueError(f"select: series ID {selector!r} not found in the supplied metadata.") from exc
322    else:
323        table, series_id, unit = find_abs_id(meta, selector, validate_unique=True)
324    s = data[table][series_id].copy()
325    s.name = series_id
326    s.attrs["unit"] = str(unit)
327    return s

Select the single Series for one (data, meta, selector) — the single-source wrapper.

Convenience for the common one-selector case; equivalent to select([(data, meta, selector)])[0]. The selector is either a {search_value: meta_column} dict for find_abs_id, or a bare ABS Series ID string, matched exactly against the metadata's Series ID column. Returns the Series named by its Series ID, with its ABS unit on .attrs["unit"].

def select( sources: Iterable[tuple[dict[str, pandas.DataFrame], pandas.DataFrame, dict[str, str] | str]], *, require_same_units: bool = True) -> list[pandas.Series]:
330def select(sources: Iterable[Source], *, require_same_units: bool = True) -> list[Series]:
331    """Select a series for each ``(data, meta, selector)`` — the iterable in, iterable out.
332
333    The composable selection primitive: takes the iterable of ``(data, meta,
334    selector)`` sources and returns the matching list of Series, ready to hand to
335    :func:`splice` (directly, or after a per-series transform).  Each selection
336    goes through ``readabs.find_abs_id`` with ``validate_unique=True``, which
337    de-duplicates on Series ID first — so a selector matching the same series in
338    several tables resolves cleanly, while one matching two genuinely different
339    series raises rather than guessing.
340
341    Parameters
342    ----------
343    sources
344        Iterable of ``(data, meta, selector)``:
345
346        - ``data``   — ``dict[table_name, DataFrame]`` from ``read_abs_cat``.
347        - ``meta``   — the matching metadata DataFrame.
348        - ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``, e.g.
349          ``{"Index Numbers ;  All groups CPI ;  Australia ;": mc.did,
350          "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series
351          ID string (e.g. ``"A2325846C"``), matched exactly.
352    require_same_units
353        If ``True`` (default) **raise** when the selected series do not all share
354        the same ABS unit — units must cohere to be spliced.  Set ``False`` when
355        you deliberately select different-unit series together (e.g. two counts
356        and a rate that you will combine yourself).
357
358    Returns
359    -------
360    list[Series]
361        One Series per source, each named by its Series ID with its ABS unit in
362        ``series.attrs["unit"]``.  Unpack it (``a, b = select([...])``), map a
363        transform over it, or pass it straight to :func:`splice`.  A later
364        transform drops the unit attr — correctly, since the unit is then no
365        longer the ABS one.
366
367    Raises
368    ------
369    ValueError
370        If ``require_same_units`` and the selected series carry mixed units.
371
372    """
373    segments = [select_one(data, meta, selector) for data, meta, selector in sources]
374    if require_same_units:
375        units = [str(s.attrs.get("unit", "")) for s in segments]
376        if len(set(units)) > 1:
377            detail = ", ".join(f"{s.name}={u!r}" for s, u in zip(segments, units, strict=True))
378            raise ValueError(
379                f"select: selected series have mismatched units ({detail}). Pass "
380                f"require_same_units=False to select different-unit series together."
381            )
382    return segments

Select a series for each (data, meta, selector) — the iterable in, iterable out.

The composable selection primitive: takes the iterable of (data, meta, selector) sources and returns the matching list of Series, ready to hand to splice() (directly, or after a per-series transform). Each selection goes through readabs.find_abs_id with validate_unique=True, which de-duplicates on Series ID first — so a selector matching the same series in several tables resolves cleanly, while one matching two genuinely different series raises rather than guessing.

Parameters

sources Iterable of (data, meta, selector):

- ``data``   — ``dict[table_name, DataFrame]`` from ``read_abs_cat``.
- ``meta``   — the matching metadata DataFrame.
- ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``, e.g.
  ``{"Index Numbers ;  All groups CPI ;  Australia ;": mc.did,
  "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series
  ID string (e.g. ``"A2325846C"``), matched exactly.

require_same_units If True (default) raise when the selected series do not all share the same ABS unit — units must cohere to be spliced. Set False when you deliberately select different-unit series together (e.g. two counts and a rate that you will combine yourself).

Returns

list[Series] One Series per source, each named by its Series ID with its ABS unit in series.attrs["unit"]. Unpack it (a, b = select([...])), map a transform over it, or pass it straight to splice(). A later transform drops the unit attr — correctly, since the unit is then no longer the ABS one.

Raises

ValueError If require_same_units and the selected series carry mixed units.

def select_and_splice( sources: Iterable[tuple[dict[str, pandas.DataFrame], pandas.DataFrame, dict[str, str] | str]], *, target: str | None = None, rebase: bool = False, agg: str = 'mean', output: str | None = None, fill: Literal['ffill', 'interpolate'] | None = None, name: str | None = None, require_same_units: bool = True) -> tuple[pandas.Series, str, pandas.DataFrame]:
385def select_and_splice(
386    sources: Iterable[Source],
387    *,
388    target: str | None = None,
389    rebase: bool = False,
390    agg: str = "mean",
391    output: str | None = None,
392    fill: Literal["ffill", "interpolate"] | None = None,
393    name: str | None = None,
394    require_same_units: bool = True,
395) -> tuple[Series, str, DataFrame]:
396    """Select one series per source and :func:`splice` them — the no-transform case.
397
398    Sugar for ``splice(select(sources))`` with a unit guard.  When
399    you need a transform *between* selecting and splicing (e.g. a growth rate),
400    compose :func:`select` and :func:`splice` directly instead — that is the whole
401    reason :func:`select` is exposed separately.
402
403    Parameters
404    ----------
405    sources
406        Ordered iterable of ``(data, meta, selector)``, **highest priority
407        first** (same priority rule as :func:`splice`):
408
409        - ``data``   — ``dict[table_name, DataFrame]`` from ``read_abs_cat``.
410        - ``meta``   — the matching metadata DataFrame.
411        - ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``,
412          e.g. ``{"Index Numbers ;  All groups CPI ;  Australia ;": mc.did,
413          "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series
414          ID string (e.g. ``"A2325846C"``), matched exactly.  In the common case
415          the only thing differing between two sources is the frequency, so a
416          shared *base* selector composes with ``base | {"Quarter": mc.freq}``.
417    target, rebase, agg, output, fill, name
418        Passed straight through to :func:`splice`.
419    require_same_units
420        Forwarded to :func:`select`: if ``True`` (default) raise when the
421        selected segments carry mixed units; ``False`` overrides (the result is
422        then labelled with the highest-priority segment's unit).
423
424    Returns
425    -------
426    tuple[Series, str, DataFrame]
427        The spliced series, its unit (the highest-priority segment's unit), and
428        the :func:`splice` join report, augmented with ``series_id`` and
429        ``unit`` columns recording what each segment resolved to.
430
431    """
432    segments = select(sources, require_same_units=require_same_units)
433    units = [str(s.attrs.get("unit", "")) for s in segments]
434
435    result, report = splice(segments, target=target, rebase=rebase, agg=agg, output=output, fill=fill, name=name)
436    # Audit trail: which Series ID / unit did each reported (lower-priority) segment use?
437    if len(report):
438        seg = [int(i) for i in report["segment"]]
439        report.insert(1, "series_id", [str(segments[i].name) for i in seg])
440        report.insert(2, "unit", [units[i] for i in seg])
441    return result, units[0], report

Select one series per source and splice() them — the no-transform case.

Sugar for splice(select(sources)) with a unit guard. When you need a transform between selecting and splicing (e.g. a growth rate), compose select() and splice() directly instead — that is the whole reason select() is exposed separately.

Parameters

sources Ordered iterable of (data, meta, selector), highest priority first (same priority rule as splice()):

- ``data``   — ``dict[table_name, DataFrame]`` from ``read_abs_cat``.
- ``meta``   — the matching metadata DataFrame.
- ``selector`` — ``{search_value: meta_column}`` for ``find_abs_id``,
  e.g. ``{"Index Numbers ;  All groups CPI ;  Australia ;": mc.did,
  "Index Numbers": mc.unit, "Quarter": mc.freq}``; or a bare ABS Series
  ID string (e.g. ``"A2325846C"``), matched exactly.  In the common case
  the only thing differing between two sources is the frequency, so a
  shared *base* selector composes with ``base | {"Quarter": mc.freq}``.

target, rebase, agg, output, fill, name Passed straight through to splice(). require_same_units Forwarded to select(): if True (default) raise when the selected segments carry mixed units; False overrides (the result is then labelled with the highest-priority segment's unit).

Returns

tuple[Series, str, DataFrame] The spliced series, its unit (the highest-priority segment's unit), and the splice() join report, augmented with series_id and unit columns recording what each segment resolved to.