Metadata-Version: 2.5
Name: market-data-normalizer
Version: 1.48.0
Summary: Normalize heterogeneous market-data feeds (CSV, WebSocket JSON, FIX) into one exchange-agnostic schema.
Project-URL: Homepage, https://harvestgroup360.com
Project-URL: Repository, https://github.com/Harvestgroup360/market-data-normalizer
Project-URL: Changelog, https://github.com/Harvestgroup360/market-data-normalizer/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/Harvestgroup360/market-data-normalizer/issues
Author-email: HarvestGroup360 <github@harvestgroup360.com>
License: MIT License
        
        Copyright (c) 2026 HarvestGroup360 (AMII LTD)
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: fix-protocol,market-data,normalization,quantitative-finance,tick-data,trading
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial :: Investment
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Description-Content-Type: text/markdown

# market-data-normalizer (`mdnorm`)

[![CI](https://github.com/Harvestgroup360/market-data-normalizer/actions/workflows/ci.yml/badge.svg)](https://github.com/Harvestgroup360/market-data-normalizer/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)
[![PyPI](https://img.shields.io/pypi/v/market-data-normalizer.svg)](https://pypi.org/project/market-data-normalizer/)

Normalize heterogeneous market-data feeds — CSV tick dumps, exchange
WebSocket JSON, and FIX — into a single, exchange-agnostic event schema, so
downstream research and execution code never has to care where a tick came
from.

Zero runtime dependencies. Pure Python (3.10+). `Decimal` prices, integer
nanosecond timestamps.

## Why

Every venue spells the same thing differently: `BTCUSDT` vs `XBT/USD`,
millisecond epochs vs FIX `UTCTimestamp`, `is_buyer_maker` booleans vs side
codes. Research notebooks and backtesters end up littered with per-venue
parsing branches. `mdnorm` pushes that mess to the edge and hands the rest of
your stack one clean type.

## Install

```console
pip install market-data-normalizer
```

The distribution is named `market-data-normalizer`; the import name is
`mdnorm`:

```python
import mdnorm
```

Pure Python, no runtime dependencies, Python 3.10+.

## Quick start

```python
from mdnorm import from_csv_row, from_ws_json, from_fix

# CSV row (ISO-8601 timestamp)
from_csv_row(
    {"symbol": "btc/usd", "ts": "2026-01-02T00:00:00Z",
     "price": "42000.5", "size": "0.25", "side": "buy"},
    venue="coinbase",
)

# Exchange WebSocket trade message
from_ws_json({"s": "BTCUSDT", "p": "42000.5", "q": "0.25",
              "T": 1767312000000, "m": False}, venue="binance")

# FIX execution report (SOH-delimited in the wild; "|" here for readability)
from_fix("55=BTC/USD|31=42000.5|32=0.25|54=1|60=20260102-00:00:00",
         venue="lmax", sep="|")
```

All three calls above produce the **same** `MarketEvent`.

### Quotes (bid/ask)

```python
from mdnorm import from_ws_quote

q = from_ws_quote(
    {"s": "BTCUSDT", "b": "41999.5", "B": "1.2",
     "a": "42000.5", "A": "0.8", "T": 1767312000000},
    venue="binance",
)
q.mid_price   # Decimal("42000.0")
q.spread      # Decimal("1.0")
```

`from_csv_quote` does the same for CSV rows with bid/ask columns.

### OHLCV bars

```python
from mdnorm import time_bars

bars = time_bars(events, interval_ns=60_000_000_000)  # 1-minute bars
bars[0].open, bars[0].high, bars[0].low, bars[0].close, bars[0].volume, bars[0].vwap
```

`time_bars` reduces a stream of trade events into fixed-interval OHLCV `Bar`s
(with VWAP and trade count), sorting out-of-order input and skipping quotes.

`resample_bars(bars, interval_ns)` downsamples bars to a coarser interval
(e.g. 1-minute → 5-minute) with correct OHLC aggregation and volume-weighted
VWAP.

`fill_gaps(bars)` returns a gapless series, inserting flat zero-volume bars
(OHLC = previous close) for any interval with no trades — a continuous grid for
backtests and feature pipelines.

### Event-driven bars

Time bars are not the only clock. Sample by activity instead:

```python
from decimal import Decimal
from mdnorm import count_bars, volume_bars, dollar_bars

count_bars(events, every=500)                       # tick bars
volume_bars(events, min_volume=Decimal("100"))      # volume bars
dollar_bars(events, min_notional=Decimal("1e6"))    # dollar bars
```

### Trading sessions

Filter a feed down to the hours that matter, with daylight saving handled
for you:

```python
from mdnorm import US_EQUITY_RTH, filter_session, group_by_session_date

rth = filter_session(events, US_EQUITY_RTH)        # 09:30-16:00 New York
by_day = group_by_session_date(events, US_EQUITY_RTH)
```

Overnight windows (a session that opens at 18:00 and closes at 17:00 the
next day) are supported, and `session_date` keeps a whole night in one
bucket. From the command line:

```console
$ mdnorm bars trades.csv --interval 5m --session 09:30-16:00 --tz America/New_York -o rth.csv
```

### Holidays, half-days, and the year that is not 252 sessions

A session is a recurring window. A calendar is that window plus the exceptions
to it, and the exceptions are the part that quietly breaks things.

```python
from mdnorm import read_calendar_csv, US_EQUITY_RTH

cal = read_calendar_csv("us_2026.csv", US_EQUITY_RTH)
cal.is_trading_day(date(2026, 7, 3))                     # False, from the file
cal.close_time(date(2026, 11, 27))                       # 13:00, a half-day
cal.trading_minutes_between(jan, dec)                    # not sessions x 390
```

**A missing holiday looks exactly like missing data.** A pipeline that does not
know a date is a holiday sees a day-long gap and reports an outage, or fills
it, or drops the instrument for poor coverage. All three are wrong in the same
way: the data was never supposed to be there.

**A half-day is not half a problem.** An early close shortens the session and
changes nothing else, so bars keep being cut against a 6.5-hour assumption, a
volatility annualised on session length is overstated for that day, and a
staleness check fires on every instrument at once an hour before it should.

**A calendar cannot answer outside the range it was given.** A file listing
2026 says nothing about 2027, and a calendar that treats an unknown weekday as
open turns a missing file into a confident wrong answer. Every query outside
`covers` raises instead — noisy exactly once, and then correct.

**252 is a convention, not a count.** How many sessions a year holds depends on
where the weekends and holidays fell; how many *minutes* it holds depends on
how many of those sessions closed early. Both are computable, and both rescale
every annualised figure in a report while leaving its shape untouched.

```console
$ mdnorm calendar us_2026.csv --session 09:30-16:00 --tz America/New_York
trading days         251
early closes         2
trading minutes      97530
note: early closes cost 360 minute(s) against a flat 390-minute session.
for `mdnorm features`: --sessions-per-year 251 --session-length 23400s
note: 2026 has 251 sessions here, not the conventional 252. Annualising a
volatility on 252 overstates it by 0.20%.
```

### A price is a number and a currency

The moment a study spans venues that quote in different currencies, every
figure in it depends on a second series nobody was watching.

```python
from mdnorm import CurrencyPair, FxRates, convert_series, decompose_return

rates = FxRates({CurrencyPair("EUR", "USD"): eurusd})
usd, dropped = convert_series(prices, rates, base="EUR", to="USD",
                              max_age_ns=MINUTE)
```

**There is no default conversion time.** Converting at the observation's own
timestamp, at a daily fix, or at the end of the study are three different
questions, and only the first was available to someone standing at that
moment. The last is the one that gets used by accident, because one rate is
easier to obtain than a series — and it restates the whole history using a
number that did not exist until the end of it. Nothing here accepts a scalar
rate.

**Staleness is the ordinary failure.** FX stops over weekends while other
venues keep trading, so an as-of join with no age limit converts a Sunday
print with Friday's close. `max_age_ns` is required, and every conversion
carries the age of the rate it used.

**Direction is not guessable from a name.** Vendors disagree about which way
round to publish a pair, and a rate applied upside-down is either wrong by a
factor of thousands or — near parity — wrong by a few per cent and entirely
plausible. Pairs carry an explicit base and quote, inversion is recorded in
the result, and it can be refused outright.

**A cross is not free, and no path is searched for.** Going through a vehicle
currency multiplies two quotes and inherits both spreads and both staleness
windows. State the vehicle with `via=` or the conversion is refused: a library
that finds its own way through the currency graph is choosing which spreads
you pay, invisibly.

**A converted return is not a converted price.** `(1 + total) = (1 + asset)(1
+ fx)` holds exactly; the familiar shorthand adds the two and drops the
product. `decompose_return` returns both and the difference between them.

```console
$ mdnorm fx prices.csv rates.csv --from EUR --to USD --max-age 1m -o usd.csv
note: the rate moved +9.09% across this span, so a single-rate conversion
would have restated the whole series by a number that did not exist until the
end of it.
```

### Prices live on a grid, and the grid is data

A venue does not accept any price. It accepts multiples of a tick, and the
tick depends on the price band, the instrument and the year.

```python
from mdnorm import TickTable, TickBand, grid_report, spread_in_ticks

table = TickTable([TickBand(D("0"), D("0.0001")),
                   TickBand(D("1"), D("0.01"))])
grid_report(prices, table).looks_raw      # could the venue have quoted these?
spread_in_ticks(bid, ask, table)          # 1.0 is the floor, not a tight market
```

**A price off the grid is telling you something.** Raw prints sit on the grid
by construction — the venue would not have accepted them otherwise. So a
series that does not is a mid, a VWAP, an average of venues, a back-adjusted
history, or an error, and those are indistinguishable by eye. `grid_report` is
one pass over the data and answers a question most pipelines never ask.

**Back-adjustment takes a series off the grid permanently, and that is
correct.** An adjusted history is a returns object, not a price object. It
stops being correct when someone rounds it back on to make it look tidy. The
grid is the cheapest way to tell the two apart after the fact.

**There is no default tick size.** The familiar penny is wrong below a dollar
on most venues, wrong for sub-penny programmes, wrong for crypto by orders of
magnitude, and wrong for the same instrument before the last regime change —
so tick tables are point-in-time data, and `TickSchedule` refuses to answer
before the first one it was given.

**Ties are not an edge case here.** On a continuous scale an exact half is a
curiosity; on a tick grid a mid between adjacent ticks is a half-tick every
single time. `Rounding` has no default and no tie shortcut.

**Round against yourself, or say that you did not.** `executable` rounds a buy
down and a sell up, so the grid never makes an order more aggressive than the
strategy asked for. Rounding to the nearest tick does the opposite about half
the time, which lifts the fill rate in any backtest that fills limit orders at
their limit. The clearest case is the mid: a market quoted one tick wide has a
mid exactly half a tick from both sides, so it is not a price the venue could
ever accept, and filling there understates cost by half the spread on every
trade.

```console
$ mdnorm ticks prices.csv --table ticks.csv
off the grid         2
note: 2 price(s) could not have been quoted on this grid, so this series is
not raw prints.
```

### Corporate actions and contract rolls

A raw price series is not continuous. A 4-for-1 split divides the printed
price by four overnight, a cash dividend drops it by the amount paid, and a
futures roll steps it by the spread between the two contracts. None of them
are market moves, but all of them look like returns:

```python
from decimal import Decimal
from mdnorm import adjust_bars, split, dividend, roll, iso_to_ns

actions = [
    split(iso_to_ns("2026-06-06T00:00:00Z"), Decimal("4")),
    dividend(iso_to_ns("2026-05-09T00:00:00Z"), Decimal("0.25")),
]
clean = adjust_bars(bars, actions)
```

Back-adjustment leaves the most recent segment at the prices that actually
printed and restates everything before each event, so the joins are seamless:

```text
raw closes    500   502   498   504  │  126  125.5   127  126.5
raw returns       +0.4% -0.8% +1.2%  │ -75.0% -0.4% +1.2% -0.4%
                                     ^ the split, not a crash

adj closes    125  125.5 124.5  126  │  126  125.5   127  126.5
adj returns       +0.4% -0.8% +1.2%  │  +0.0% -0.4% +1.2% -0.4%
```

Splits scale volume as well as price. Dividends take their reference price
from the last print before the ex-date unless you pass one. Rolls support
both conventions — `AdjustMethod.RATIO` (default, preserves returns) and
`AdjustMethod.DIFFERENCE` (preserves price differences, the usual choice for
futures). Factors are composed as exact rationals, so a 1-for-2 followed by a
1-for-3 restates 600 to exactly 100 rather than 99.999...96.

Actions can come from a file, and the CLI wires it up:

```console
$ mdnorm bars trades.csv --interval 1d --actions actions.csv -o adjusted.csv
$ mdnorm bars tape.jsonl --infer-sides --every-imbalance 500 -o imbalance.csv
$ mdnorm book deltas.csv --symbol BTC-USD -o quotes.jsonl
$ mdnorm nbbo quotes.jsonl --max-age 2s -o top.jsonl
$ mdnorm tca fills.csv --market tape.jsonl --decision-price 100
```

```text
ts,kind,value,ref_price
2026-06-06T00:00:00Z,split,4,
2026-05-09T00:00:00Z,dividend,0.25,190.50
2026-03-14T00:00:00Z,roll,5312.50,5290.25
```

### Who crossed the spread

Most trade tapes give you a price and a size but not the aggressor. That one
missing field is what separates a price series from an order-flow series, and
signed volume, order imbalance and imbalance bars are all defined in terms of
it. `mdnorm.micro` infers it, using the three rules the literature settled on:

```python
from mdnorm import SideRule, infer_sides, trade_imbalance, mean_effective_spread

classified = infer_sides(events)                       # Lee-Ready by default
print(trade_imbalance(classified))                     # -1 selling .. +1 buying
print(mean_effective_spread(classified))               # 2 * |price - mid|
```

`SideRule.TICK` compares each trade with the previous different price and
needs trades only. `SideRule.QUOTE` compares the trade with the prevailing
mid. `SideRule.LEE_READY` — the default — uses the quote rule and falls back
to the tick rule at the mid. A side reported by the venue always wins;
inference only fills gaps, and trades it cannot resolve stay `None` rather
than being guessed at. Published accuracy of these rules is roughly 75-85% on
liquid names, so treat an inferred side as an estimate.

`roll_spread` estimates the effective spread from trade prices alone, via the
serial covariance that bid-ask bounce induces. It needs no quotes, which makes
it a useful cross-check on the rest — and it returns `None` rather than zero
when the covariance comes out non-negative and the estimator is undefined.

### Imbalance bars

Once trades carry a side, the sampling clock can follow order flow instead of
time or volume:

```python
from mdnorm import Pipeline

bars = Pipeline().infer_sides().imbalance_bars(Decimal("500")).run(events)
```

A bar runs until buyers have outbought sellers, or the reverse, by the
threshold. Balanced two-sided periods produce one long bar; a sustained
one-sided push produces several short ones. `by="tick"` measures the imbalance
in trade count rather than size. From the command line:

```console
$ mdnorm bars tape.jsonl --infer-sides --every-imbalance 500 -o imbalance.csv
```

### Rebuilding the order book

Exchanges do not send you a book. They send a snapshot and then a stream of
deltas, and the book only exists if you apply every one of them, in order:

```python
from mdnorm import BookDelta, OrderBook, Side, replay_book

book = OrderBook("BTC-USD", "binance")
book.apply_snapshot(ts, bids=[(D("100"), D("2"))], asks=[(D("101"), D("3"))], seq=10)

quotes = list(replay_book(book, deltas))     # one quote per change in the top
print(book.best_bid, book.spread, book.imbalance(levels=5))
```

Two failure modes make a reconstructed book silently untrue, and this
implementation refuses to hide either.

A **sequence gap** means a message was missed, and no later update repairs the
damage — the book is simply wrong from then on, in a way that looks completely
normal. `OrderBook` raises `SequenceGapError` the moment a number is skipped,
naming how many updates went missing, because the correct response is to
resynchronise from a snapshot rather than carry on. Duplicated or replayed
messages are rejected the same way. Feeds without sequence numbers work fine;
pass `strict_sequence=False` to opt out entirely.

A **crossed book** — best bid at or above best ask — is not a market state but
a symptom: a dropped delete, a stale snapshot, two venues merged by mistake.
It is exposed as `is_crossed`, and the spread goes negative rather than being
quietly clamped to zero.

`to_quote()` turns the top of the book into an ordinary `MarketEvent`, so a
reconstructed book feeds straight into session filtering, trade classification
and effective spreads with nothing in between. From the command line:

```console
$ mdnorm book deltas.csv --symbol BTC-USD --venue binance -o quotes.jsonl
```

### One instrument, several venues

When something trades in more than one place, "the price" is a question. The
consolidated top of book is the answer, and it is where three problems live
that a maximum over venues will not warn you about:

```python
from mdnorm import consolidate

top = consolidate(quotes, max_age_ns=2_000_000_000)   # 2s staleness cutoff
```

**A venue that goes quiet keeps voting.** When a feed disconnects, its last
quote stays in the consolidation forever — and a stale price is very often the
*best* price, so the dead venue ends up setting the top of book. This is the
failure that produces a consolidated feed which looks excellent and is
fiction. `max_age_ns` retires a venue that has not spoken recently;
`stale_venues()` names them.

**A consolidated book can appear crossed.** A bid on one venue above the offer
on another looks like free money and is almost always clock skew between two
feeds timestamped by different machines. `is_crossed` reports it and
`crossed_updates` counts it, because the useful response is to check the
clocks rather than to trade the spread.

**Ties need a rule.** Equal best prices are broken by size, then by venue
name, so the same input always produces the same output.

Which venue actually sets the price is a measurement in its own right, and
`leadership` counts it. The pieces compose: an order book becomes a quote,
quotes from several venues consolidate into one, and the result feeds trade
classification and effective spreads unchanged.

```console
$ mdnorm nbbo quotes.jsonl --symbol BTC-USD --max-age 2s -o top.jsonl
```

### Did I execute well?

Once the tape is clean the next question is about you rather than the market,
and every standard benchmark has a way of quietly flattering the person
running it:

```python
from mdnorm import Fill, Side, evaluate

report = evaluate(my_fills, market_trades, decision_price=D("100"))
print(report.slippage_vs_vwap_bps, report.participation_rate)
```

**Your own trades are in the benchmark.** A VWAP over the public tape includes
the prints you just made, so you end up partly benchmarking yourself against
yourself — and the bigger your share of volume, the more the benchmark bends
toward your own average price. `evaluate` removes your fills from the tape
before computing anything; `exclude_fills` does it on its own if you want the
benchmark separately. In the library's own test suite, leaving them in turns a
100 VWAP into 109 and a losing execution into a winning one.

**Participation decides whether the number means anything.** Beating VWAP by
two basis points on 0.1% of volume is a result; the same number on 30% of
volume mostly measures your own impact. The summary always reports the two
together, and the CLI says so out loud above 10%.

**Sign conventions are stated, not assumed.** Positive basis points always
mean better than the benchmark — paying below it on a buy, selling above it on
a sell. Mixed-side fills are refused rather than netted, because one number
covering both directions has no meaning.

By default the window runs from your first fill to your last. That is right
for a worked order and wrong for a single fill — the only print in the window
is then your own — so `start_ns` and `end_ns` let you score against an
interval you chose instead.

TWAP skips intervals that never traded instead of carrying the last price
forward, for the same reason nothing else here invents data.

```console
$ mdnorm tca fills.csv --market tape.jsonl --decision-price 100
```

### Several instruments, one time grid

Research wants a matrix — one row per timestamp, one column per instrument —
and building it from independent tick streams is where look-ahead bias gets in,
because every mistake here makes the backtest *better* rather than raising:

```python
from mdnorm import Field, align

rows = align({"BTC": btc_events, "ETH": eth_events},
             interval_ns=60_000_000_000,       # a one-minute grid
             max_age_ns=5 * 60_000_000_000)    # nothing older than 5 minutes
rows[0].values      # {"BTC": Decimal("60000"), "ETH": Decimal("3000")}
rows[0].ages_ns     # how old each value was at that grid point
rows[0].complete    # False if any column had nothing to show
```

**The join only looks backwards.** A value is visible at a grid point only if
it was observed at or before it. "Nearest observation" is the expensive default
in this area: on a one-minute grid it lets a print from 09:30:20 be read at
09:30:00, and twenty seconds of hindsight is enough to make a mediocre signal
look tradeable.

**A bar labelled 09:30 is not knowable at 09:30.** It contains everything that
traded until 09:31, so joining bars on their label imports an interval of the
future. `AsOfSeries.from_bars` timestamps each bar at its *end*, and
`align_bars` therefore gives you the last **closed** bar per column — one
interval further back than the naive join, and the version you could have
traded.

**Forward-filling has no natural end.** A halted or delisted stream otherwise
contributes its last price forever, and a frozen price correlates with nothing,
which reads as diversification. With `max_age_ns` a quiet column becomes
`None`; the age is still reported, so `row.stale` (had data, too old) and
`row.missing` (never had data) stay distinguishable.

**A feed you get late was not available on time.** `AsOfSeries.delayed(250ms)`
shifts observation times forward by the delivery delay, so alignment reflects
when you could have acted rather than when the source stamped it. If you do
not know what your delay is, the next section measures it.

Nothing interpolates or smooths. `align_on` takes timestamps you supply, for
one row per print of a reference instrument, per signal, or per fill.

```console
$ mdnorm align BTC=btc.csv ETH=eth.jsonl --interval 1m --max-age 5m -o matrix.csv
```

### When the venue says it happened, and when you found out

`AsOfSeries.delayed` has always taken a delay and said, in its own docstring,
that a delay of zero is a claim about your infrastructure rather than a
default. It never offered a way to find out what yours is. This is that half:

```python
from mdnorm import Arrival, delay_report, as_received, as_stamped, view_gap

report = delay_report(arrivals)     # what the transport actually costs
report.median_ns, report.p95_ns     # the typical case, and the one to size for
report.negative                     # rows received before they happened
report.out_of_order                 # messages that overtook the one before

knowable = as_received(arrivals)    # the series you could have acted on
optimistic = as_stamped(arrivals)   # the series research usually builds
view_gap(arrivals, grid).share      # how often those two disagree
```

**A venue timestamp is not an arrival.** Keying research on it claims the
information reached you instantly, and the error only ever points one way:
every signal looks actionable slightly earlier than it was, every cross-venue
lead is inflated by the difference in transport, and a fill is priced at a
quote that had not reached the machine placing the order. Nothing fails. The
result is simply better than it should be.

**There is no default delay.** A file with no receipt column gets no invented
one. State what you decided to believe with `assume_delay_ns=` and the report
comes back with `assumed=True` — a report that hides which of the two it used
is worse than no report.

**A negative delay is a fact, not an outlier.** Receipt before the venue stamp
means the clocks disagree, which is usually the more interesting finding. It
is counted separately and never clamped, because clamping turns a clock
problem into a latency figure that looks fine.

**The mean latency is the least useful summary there is.** A transport
distribution has a tail and the mean mostly measures it. The report gives the
median and the p95 by nearest rank, so every figure it prints is a delay that
actually happened, and `tail_ratio` says whether the typical case and the bad
case are the same problem.

`view_gap` asks both views what they knew at each grid point and reports where
they differ, with `largest_gain_ns` — the most unearned foresight found. That
number is the one to compare against the horizon your signal acts on: a
quarter of a second is nothing to a daily rebalance and everything to a queue
position.

```console
$ mdnorm arrival feed.csv --interval 1s
observations         184203
median delay         412us
p95 delay            3.9ms
p95 / median         9.47x
received before sent 0
out of order         37
for `AsOfSeries.delayed`: by_ns=412000 for the typical case, 3900000 for the
case worth sizing against.
```

### The average of your returns is not the return you earned

An average monthly return of one per cent does not annualise to 12.68 per
cent, which is what compounding the average gives. It annualises to what the
account did:

```python
from mdnorm import compound_report, Convention

rep = compound_report(monthly, convention=Convention.SIMPLE)
rep.arithmetic                 # 0.010000 — the number in the deck
rep.geometric                  # 0.009529 — the rate that compounds
rep.drag                       # 0.000471 per month

year = rep.annualised(periods_per_year=12)
year.naive                     # 0.126825 — the average, compounded
year.actual                    # 0.120539 — what the account did
year.overstatement             # 0.006286 — sixty-three basis points
```

**The direction is guaranteed and the size is not.** The geometric mean never
exceeds the arithmetic one — an inequality, not a tendency — so compounding
the average always overstates. How much depends on the variance alone, which
means **the flattery grows with the risk**. At ten per cent annual volatility
the drag is about half a point a year; at forty per cent it is about eight.
That is the wrong way round for a statistic to behave.

**Leverage is worse than proportional.** Doubling every period return roughly
quadruples the drag, because the variance term is squared while the mean is
only doubled. `leverage_drag` applies the multiple and recomputes rather than
scaling a rule of thumb — 3.97× and 8.87× on the example below, against the
2× and 3× a reader would guess. Only the returns are scaled: financing, borrow
and the path-dependence of a daily reset are real and are not modelled, so it
is a lower bound on what leverage costs rather than an estimate of it.

**One comparison has no fixed direction, and it is the one people reach for.**
The sum of the returns is not reliably above or below the compounded total:
compounding adds every cross-product, which helps a positive series and hurts a
volatile one. `total_gap` reports it and says so. An earlier draft of this
module called that figure "the overstatement", which would have been wrong
about half the time; the comparison with a guaranteed sign is the annualised
one above.

```console
$ mdnorm compounding monthly.csv --convention simple --periods 12 --leverage 2 3
observations           12
convention             simple
arithmetic mean        0.010000
geometric mean         0.009529
drag per period        0.000471
  sigma squared / 2    0.000475
volatility             0.030822
compounded total       0.120539
sum of returns         0.120000
annualised at 12 periods
  average compounded   0.126825
  actual               0.120539
  overstated by        0.006286
  as a share           5.22%
  leverage 2x drag     0.001869  (3.97x the unlevered drag)
  leverage 3x drag     0.004177  (8.87x the unlevered drag)
```

The σ²/2 rule of thumb is reported beside the exact figure rather than used in
place of it. They agree to a basis point on monthly equity returns and part
company as soon as periods get large, and the size of that disagreement is
itself worth seeing, since the approximation is what most reports are built on.

**Nothing here guesses your convention.** A log return and a simple return are
different numbers living in identically shaped files, and compounding one as
though it were the other is exactly the kind of silent error this library
exists to catch. `Convention` is required on every call, and there is no
default annualisation factor either — for the reason
[ROADMAP.md](ROADMAP.md) already gives.

### Alpha is what is left after the things you already knew about

A strategy with a Sharpe ratio worth reporting is sometimes a strategy, and
sometimes it is a factor everybody can already buy, wearing a new name. The
return series of an exposure and the return series of an edge look identical:

```python
from mdnorm import factor_regression, alpha_stream, sharpe_ratio

rep = factor_regression(strategy, {"market": mkt, "momentum": mom,
                                   "value": val})
rep.r_squared        # 0.8562
rep.alpha            # 0.00010503 per period
rep.alpha_t_stat     # 1.223 — not distinguishable from zero
rep.alpha_share      # 0.18 — 18% of the mean return survives

sharpe_ratio(strategy)                       # 1.16 annualised
sharpe_ratio(alpha_stream(strategy, facs))   # 0.55 annualised
```

Five years of daily returns. The headline Sharpe is 1.16 and the market
carries 0.00044607 of the 0.00058426 mean — three quarters of the return.
What is left over is worth 0.55, and its t-statistic is 1.2.

**The absence of an exposure is not evidence of alpha.** It is evidence about
your factor list. A residual that no factor explains means the strategy is
orthogonal to *the factors you supplied*, which is a much smaller claim than
the one people make with it. This module will never tell you a strategy has
alpha; it tells you how much survives a list you chose.

**No factor data ships with this library and none ever will.** Bundling a
factor set would make every answer partly a property of whose definition of
momentum we vendored, and [ROADMAP.md](ROADMAP.md) already rules out tying the
library to one feed. Bring your own series, and record where they came from —
`mdnorm.provenance` exists for that.

**One trap is in the arithmetic rather than the data.** A least-squares
residual computed with an intercept has a mean of exactly zero, always, so a
Sharpe ratio on it is zero whatever the alpha was. `residuals` returns that
series, for looking at the *shape* of what the factors missed; `alpha_stream`
returns the strategy with the factor contributions removed and the intercept
kept, whose mean *is* the alpha. That is the series to put a ratio on, and it
is the one nobody computes.

```console
$ mdnorm exposure returns.csv --strategy strategy
observations           1260
factors                3
R squared              0.8562
mean return            0.00058426
alpha                  0.00010503
  t-statistic          1.223
  share of the mean    18.0%
loadings
  market             beta     0.6851  t    81.34  carries   0.00044607
  momentum           beta     0.4157  t    28.19  carries   0.00003139
  value              beta     0.0119  t     0.68  carries   0.00000177
dominant factor        market (0.00044607)
Sharpe, as reported    0.0729
Sharpe, alpha only     0.0346
```

`dominant_factor` ranks by the return a factor carried, not by its beta and
not by its t-statistic: a large loading on a factor that went nowhere carries
nothing, and a significant coefficient is a statement about precision rather
than about magnitude.

**Choosing a factor list after seeing the strategy is a search.** Four
candidates in every combination are fifteen regressions, and the one with the
flattering residual is the best of fifteen. `mdnorm.multiverse` will enumerate
that grid and hand you the count. **The loadings are also full-sample and
constant** — a strategy that was fully exposed in the first half and flat in
the second reports an average beta describing neither half, so run the
regression across `mdnorm.windows` and watch whether the betas move.

### Five hundred names is not five hundred bets

`mdnorm.independence` counts how many independent observations an
overlapping-label study really has, along the time axis. This asks the same
question across the cross-section, and almost nobody asks it:

```python
from mdnorm import correlation_matrix, breadth_report

rep = breadth_report(correlation_matrix(returns_by_symbol))
rep.names                  # 40
rep.average_correlation    # 0.5035
rep.effective_bets         # 3.628
rep.overstatement          # 11.02x
rep.ratio_overstatement    # 3.32x
```

**Breadth enters performance arithmetic under a square root, so the error
compounds twice.** The fundamental law puts an information ratio at the
information coefficient times the root of the number of independent bets.
Forty names driven by one market are not forty bets and not twenty — they are
three and a half, and a ratio computed on the position count is overstated by
a factor of three.

**The error is silent because the position count is a fact.** There really are
forty names, the trades really happened, the reconciliation really balances.
Nothing in the accounting is wrong. What is wrong is the claim implied by
quoting a statistic against forty, and no line of the books contradicts it.

**Two counts are reported and they are not two estimates of one thing.**
`effective_bets` is the participation ratio of the eigenvalues, `(Σλ)²/Σλ²` —
how concentrated risk is across independent directions. `effective_observations`
is `n / (1 + (n-1)ρ̄)` — what an average of n correlated series is worth as a
sample size, which is the number a cross-sectional t-statistic needs, and
`BreadthReport.as_sample()` hands it straight to `deflate_t_stat`. They
coincide only at the extremes: both give n for the identity and one when every
correlation is one. Three names at ρ = 0.5 are **two bets and one and a half
observations**, and as n grows at fixed ρ the first tends to 1/ρ² and the
second to 1/ρ. Quote the one that matches the claim being made.

```console
$ mdnorm breadth panel.csv --eigenvalues --list-limit 5
names                  40
observations           500
average correlation    0.5035
effective bets         3.628
effective observations 1.938
position count over    11.02x the bets
  information ratio    3.32x overstated
eigenvalues
    1     20.747915   51.87%
    2      0.812768    2.03%
    3      0.777735    1.94%
    4      0.750583    1.88%
    5      0.748383    1.87%
  ... (35 more)
```

One direction carries fifty-two per cent of the variance and the next
thirty-nine share the rest. That is the whole finding, and it is visible
before any of the summary numbers are computed.

**A short sample flatters the count upward.** With fewer observations than
names the sample correlation matrix is singular and part of its spectrum is
noise, which inflates the apparent number of bets. `thin_sample` reports the
condition; nothing here corrects for it, because the correction needs a model
of the return process and this library does not have one.

The eigenvalues come from a cyclic Jacobi rotation written for `Decimal`, so
there is still no runtime dependency. It stops when the off-diagonal mass
reaches the round-off floor of the working precision and raises if the floor
it reaches is large enough to matter, rather than returning a
half-diagonalised answer dressed up as a spectrum.

### Every cleaning decision is a fork, and nobody counts the forks

The window is not the only thing that was chosen. So was the staleness
threshold, the clipping sigma, whether the scale was ordinary or robust,
whether repeated prints were dropped. Each decision was defensible and made
once. Together they define a grid, and the published number is one cell of it:

```python
from mdnorm import Choice, specifications, explore, choice_effect

choices = [
    Choice("clip",  [("none", None), ("5 sigma", D(5)), ("3 sigma", D(3))]),
    Choice("scale", [("ordinary", False), ("robust", True)]),
    Choice("stale", [("keep", False), ("drop", True)]),
]
curve = explore(specifications(choices), run_pipeline)
curve.highest        # 0.8682 — clip nothing, ordinary scale, keep repeats
curve.lowest         # 0.4254 — clip at 3 sigma, robust scale, drop repeats
curve.trials         # 12
choice_effect(curve, "clip").spread    # 0.3096 of the 0.4428 total
```

**The grid multiplies, which is the point.** Three decisions here; six binary
ones would be sixty-four pipelines. Nothing samples the grid for you and there
is no default set of decisions, because which forks a pipeline contains is a
property of that pipeline and a library that guessed would be reporting on one
it invented.

**Which decision is doing the work is usually answerable, and that is the
useful output.** `choice_effect` gives the median result under each option of
one decision; `dominant_choice` names the one that pulls them furthest apart.
Above, the clipping threshold accounts for 0.3096 of the 0.4428 spread — so
the sentence to write is not *the number is unstable* but *the number is
mostly a function of a threshold we chose in a meeting*.

**A specification count is a trial count, with a caveat we will not bury.**
`SpecCurve.trials` goes to `deflated_sharpe_ratio` the way
`SensitivityReport.trials` does, and it is an **upper bound** on the effective
number of trials: two pipelines differing in one choice out of six see nearly
the same data. Deflating by the raw count errs toward caution, which is the
direction to be wrong in, and it is still wrong. Nothing here estimates the
effective count, because that needs a model of how the choices correlate and
this library does not have one.

```console
$ mdnorm multiverse pnl.csv --clip 5 3 --scale --stale --periods 252 --deflate
observations         1000
specifications       12
  clip               none | 5 sigma | 3 sigma
  scale              ordinary | robust
  stale              keep | drop
lowest               0.4254
median               0.7851
highest              0.8682
spread               0.4428
positive             12/12 (100.0%)
highest from         clip=none, scale=ordinary, stale=keep
lowest from          clip=3 sigma, scale=robust, stale=drop
attribution
  clip               none=0.8266  5 sigma=0.7929  3 sigma=0.5170   spread 0.3096
  scale              ordinary=0.7851  robust=0.7485   spread 0.0366
  stale              keep=0.8345  drop=0.7485   spread 0.0860
dominant decision    clip (0.3096 of 0.4428)
trials               12
note: these specifications share a grid and are not independent, so this is
an upper bound on the effective number of trials. Deflating by it errs toward
caution rather than toward being right.
best specification deflated
  as one trial       0.9579
  as 12 trials       0.8874
```

`best` is named for the number and not for the pipeline. Which cleaning is
correct is not a question this library can answer — the cell that flatters a
result is simply the one most in need of a reason, and the parameters that
produced it belong in a `mdnorm.provenance` manifest.

### Choosing where the sample starts is a trial

A backtest is reported as one number over one window. The window was chosen
too — by when the data happened to begin, by which vendor file was to hand, or
by somebody sliding the start forward until the curve looked right. The last
of those is a search, and nobody counts it:

```python
from mdnorm import trimmed_starts, sweep, sharpe_ratio

rep = sweep(returns, trimmed_starts(len(returns), step=21, count=24),
            sharpe_ratio, kind=WindowKind.TRIMMED_START)
rep.full          # -0.0329 — a losing strategy over everything
rep.highest       # +0.0524 — profitable, starting eleven months in
rep.changes_sign  # True
rep.trials        # 25 — the number to hand a deflated Sharpe
```

**A window count is a trial count.** Twenty-four start dates are
twenty-four alternatives that could have been reported. Quoting the best of
them without deflating for that count is the same error as quoting the best of
twenty-four strategies, and it is harder to see because only one strategy was
ever written down. `SensitivityReport.trials` exists so the number reaches
`deflated_sharpe_ratio` instead of somebody's memory of how the window was
picked.

**The spread is the finding, not the best value in it.** A metric that is
positive on seven windows out of twenty-four and negative on the other
seventeen has not been measured badly — it has been measured, and the answer
is that the headline figure is mostly a function of where the window opens.

**Shorter windows are noisier, and that is part of what you see.** A metric
over a third of the data carries roughly √3 times the standard error, so some
of the spread is sampling noise rather than instability. Nothing here
separates the two; that would need a model of the return process and this
library does not have one. Every window carries its observation count so the
shrinkage is visible, and a wide spread is consistent with instability rather
than proof of it.

```console
$ mdnorm windows pnl.csv --metric sharpe --trim-start 21 --count 24 --deflate
observations         1000
windows              24 (trimmed_start)
  shortest           496
  longest            979
full sample          -0.0329
lowest               -0.0351
median               -0.0168
highest              0.0524
spread               0.0875
positive             7/24 (29.2%)
note: the metric is positive on some windows and negative on others. That is
not a matter of degree.
trials               25
best window deflated
  as one trial       0.9510
  as 25 trials       0.4730
```

Read the last two lines together. The best window deflates to 0.95 if you
pretend it was the only thing you ever looked at, and to 0.47 once the
twenty-five windows are counted — and the strategy loses money over the full
sample either way.

**The sensitivity is usually to a handful of observations.** If dropping the
first four months changes the answer, find out what was in those four months
before concluding anything about regimes. `mdnorm.extremes` counts how few
observations a result rests on, and a moved start date is often just a dropped
outlier wearing a different hat.

### A fat tail and a fat finger look identical

One is the risk you are paid to carry, the other is a typo, and nothing in the
number distinguishes them. So this measures what removing them would cost and
removes nothing:

```python
from mdnorm import flag_extremes, clip_effect, concentration

flag_extremes(returns, sigma=5, robust=True)     # 30 found
flag_extremes(returns, sigma=5)                  # 0 found
concentration(returns, share=Decimal("0.5"))     # 3 observations
```

**The outliers hide inside the ruler used to find them.** A z-score divides by
a standard deviation computed from the same sample, and every extreme
observation inflates it. Thirty contaminated points in a thousand raised the
scale by 1.42x, which pushed all thirty from six sigma down to four and a bit
— so at a five-sigma cut the ordinary score found **none of them** and the
robust score found **all thirty**. That is masking, and it starts as soon as
contamination is more than about one per cent.

**So the scale is computed two ways and you choose.** `robust=True` is the
median and the median absolute deviation scaled by 1.4826, which the extremes
cannot move. Right for detection, wrong for description — a robust scale
deliberately ignores the tail you may be trying to measure. Both are reported
and neither is assumed.

**Clipping always lowers the measured volatility; what it does to the Sharpe
depends on which side the tail was on.** Symmetric extremes leave the mean
alone and the ratio rises, which is the case people have in mind. A one-sided
tail takes the profit with it and the ratio falls — 0.88x on the run below.
Both are distortions of the same size, so `clip_effect` reports the shift
without asserting a sign and hands back no data.

**Concentration is the question behind all of it.** Three observations out of
a thousand make half the total here. That strategy is a bet on three days, and
whether those three were real is the only question that matters.

```console
$ mdnorm extremes pnl.csv --sigma 5 --robust --tail 10 \
    --concentration 0.5 --clip 3
observations         1000
ordinary centre +0.00034184  scale 0.01422197
robust   centre +0.00024111  scale 0.01004799
note: the ordinary scale is 1.42x the robust one, which means the extremes
are inflating the ruler they would be measured against.
at 5 sigma
  ordinary score     0
  robust score       30
largest 10 by size
  their total        0.12
  whole total        0.3418381129
  without them       0.2218381129
  share of the total 35.10%
concentration        3 observation(s) make 50% of the total
clipping at 3 sigma
  would touch        32
  volatility         0.01422197 -> 0.01100780
  understated to     0.7740x
  Sharpe             0.8838x (lower)
note: the clip lowered the Sharpe rather than raising it, which means the
tail was one-sided and the profit went with it.
```

Nothing was clipped. `winsorise` exists and is separate, so trimming a sample
is a visible act rather than a side effect of measuring it — and the sigma you
picked is a parameter chosen after seeing the data, which is what
`mdnorm.provenance` is for.

### A result you cannot reproduce is not a result

Every other module here refuses to guess a constant. `metrics` will not invent
the number of trials a search ran, `coverage` will not pick a gap threshold,
`halts` will not infer a pause, `independence` will not choose a truncation
lag. Each refusal hands the caller a decision — and until now nothing wrote
down what they decided:

```python
from mdnorm import manifest, verify, read_manifest, write_manifest

m = manifest(command="sharpe", inputs=["pnl.csv"],
             parameters={"trials": 500, "risk_free": "0.04"})
write_manifest(m, "run.json")          # fingerprint ed731c772bdd

v = verify(read_manifest("run.json"), parameters={"trials": 50})
v.reproducible                          # False
v.drifts                                # pnl.csv: digest, trials: 500 -> 50
```

**The parameters are the part that goes missing.** An input that changes is
usually noticed, because somebody had to change it. A trial count that was 500
in the run and 50 in the write-up is noticed by nobody, and it is the whole
difference between a deflated Sharpe that survives and one that does not. A
manifest records the arguments beside the data because they are the same kind
of fact.

**A fingerprint that includes the clock answers no question.** Two runs of the
same pipeline over the same inputs with the same arguments must fingerprint
identically, so `created_ns` and the free-text note are excluded and only what
would change the numbers is covered. Outputs are excluded too, deliberately:
the same fingerprint with different results is the finding.

**Floats are refused.** `0.1` is not a value, it is a rendering of one, and the
rendering differs by platform. Pass a `str` or a `Decimal` and the manifest
records what you meant; pass a float and it raises rather than promising to
reproduce something it can only approximate.

**An edited manifest is refused.** `read_manifest` re-derives the fingerprint
from the contents and raises if the file disagrees with itself. A manifest
somebody has corrected by hand is worse than no manifest, because it carries
the authority of a record while stating something that never happened.

**Nothing here judges.** `verify` reports what moved and stops. Whether a
changed input is a correction or a corruption is not a question a library can
answer, and a tool that decided would be trusted for a judgment it is not
entitled to make.

```console
$ mdnorm provenance run.json --verify --parameter trials=50
command              sharpe
recorded with        market-data-normalizer 1.35.0
fingerprint          ed731c772bdd
inputs               1
result               2 difference(s)
  pnl.csv: digest 'b264dd52615d' -> 'db1608907c45'
  trials: parameter '500' -> '50'
note: nothing here says which side is right. A changed input may be a
correction or a corruption, and that is not a question this library can answer.
```

Exit status is non-zero when a run does not reproduce, so this drops into a
scheduled check without any parsing.

### The absence of a row is not the absence of an event

A feed that stops delivering and a market that stops trading produce the same
thing: nothing. Every calculation downstream reads the silence as information:

```python
from mdnorm import coverage_report, panel_coverage

rep = coverage_report(events, min_gap_ns=5 * MINUTE, calendar=cal,
                      halts=halts, start_ns=first, end_ns=last)
rep.explained_share        # 97.27% of the silence was the venue being shut
rep.unexplained_ns         # 5h54m nobody has accounted for
rep.longest_unexplained_ns # 3h10m — a feed that stopped mid-session
```

**A gap has to be explained before it is a gap.** Most silences are ordinary,
and a raw gap count is mostly weekends. `explain_gaps` subtracts closed
sessions using a `TradingCalendar` and pauses using the `halts` windows, then
reports the residual — which is the only part that says anything about the
feed. The three components are times rather than one label, because a Friday
outage running into a weekend is part missing data and part closed venue.

**Give the bounds, or a feed that stopped is invisible.** Without `start_ns`
and `end_ns` a gap only exists between two observations, so a symbol that goes
quiet and never returns produces nothing at all — its last print has nothing
after it to be distant from. That is the case worth catching, because
instruments stop printing when something has happened to them. With the
bounds, a named symbol that never appears is one gap the width of the period.

**Names go missing when they are in trouble.** A cross-sectional rank or
z-score over "the instruments that printed" is computed on a universe whose
width moves, and the ones that drop out are not a random sample.
`panel_coverage` counts the width at every point instead of averaging it away.

**No default threshold, and the calendar is recorded.** Five minutes without a
print is remarkable on a liquid future and unremarkable on a corporate bond,
so `min_gap_ns` is required — the same objection `halts` makes to inferring a
pause. A report built without a calendar carries `calendar=False`, because
counting every night as missing data is right for a venue that never closes
and badly wrong for one that does.

```console
$ mdnorm coverage feed.csv --min-gap 5m --calendar us_2026.csv \
    --session 09:30-16:00 --tz America/New_York --panel 1d \
    --since 1773149400000000000 --until 1773432000000000000
events               5481
symbols              3
covered span         307h30m
gaps over 5m         15
  total silence      216h24m
  venue was shut     210h
  halted             30m
  unexplained        5h54m
longest unexplained  3h10m
explained            97.27% of the silence
```

### A price you could not have traded at

When an instrument is halted the tape goes quiet, and a backtest reading that
tape sees nothing unusual. The last print stands, the features keep updating
off it, and every fill placed in that silence is a fill that could not have
happened:

```python
from mdnorm import halt_report, reopen_gaps, unfillable

halt_report(events, halts).halted_share      # 25.6% of the covered span
reopen_gaps(events, halts)[0].move_bps       # -1,834.8 across one pause
unfillable(decisions, halts).value_share     # 91.6% of the money
```

**Halts are concentrated in exactly the wrong place.** An instrument is not
paused on a quiet afternoon. It is paused on the day of the earnings leak, the
guidance cut, the tender offer — the days carrying the largest moves in the
sample. The fills a backtest invents during a halt are not a random slice of
its trades; they come from the fattest part of the tail, and they land on the
right side of it, because the strategy is reading a price that has not yet
absorbed the news.

**The count understates it and the value does not.** One decision in two on
this input fell inside the pause, and those decisions carried 91.6 per cent of
the money. A `value_share` above a `count_share` is the signature of the whole
problem, which is why `unfillable` reports both and sums in absolute terms, so
a short and a long of the same size cannot cancel into a reassuring zero.

**The reopening move belongs to nobody.** A name halted at 41.15 and reopened
at 33.60 fell eighteen per cent with no tradable print in between. Whoever
held it took the loss; whoever "entered" during the pause entered at the stale
price and was marked at the new one, which is not a trade. `reopen_gaps`
prices every one of those.

**Nothing is inferred.** There is no rule here that a long enough quiet
stretch is a halt — on an illiquid name that rule fires constantly, and the
resulting statistic is a property of the threshold rather than of the market.
Either the windows are supplied, or the report says it has none. Symbols with
no halt record at all are named rather than assumed clean.

```console
$ mdnorm halts trades.csv --halts halts.csv --decisions fills.csv
events               30
halts                1 across 1 symbol(s)
halted time          10m
longest halt         10m
share of covered     25.64%
reopening moves      (no tradable price existed across these)
  AAA 10m  41.15 -> 33.6  -1834.8 bps
decisions            4
  unfillable         2 (50.00%)
  by value           91.60%
note: the value share exceeds the count share, which means the decisions taken
while halted were the large ones.
```

Prints stamped inside a halt window are counted and not interpreted: usually a
late report of a pre-halt execution, occasionally a cross that is allowed to
print, sometimes a vendor with a broken clock.

### A price that stopped moving is not a price that stopped being risky

`align` has warned since it was written that a frozen price is uncorrelated
with everything and therefore reads as diversification. It gave no way to find
out how much of that you have:

```python
from mdnorm import staleness_report, smoothing_bias

rep = staleness_report(marks, min_run=3)
rep.unchanged_share, rep.longest_run    # 75%, 31 observations

bias = smoothing_bias(returns)
bias.weight_current                     # 0.721 — the rest arrives tomorrow
bias.volatility_understated             # 0.773x
bias.sharpe_inflation                   # 1.294x
```

**Repeated values are a fact; staleness is an interpretation.** An illiquid
instrument genuinely does not trade for an hour, and a vendor carrying
yesterday's mark forward produces identical rows. Nothing here decides which
you have — `runs` and `staleness_report` count, and what makes a flat stretch
suspicious is the instrument and the sampling interval, which is why there is
no default `min_run`.

**Smoothing is where the money is.** A reported series that partly reflects
the previous period's move is a moving average of the true one, and a moving
average has lower variance than what it averages. Lower measured volatility
with the same mean is a higher Sharpe, a lower beta and a smaller correlation
with everything else — four numbers moving the flattering way from one cause,
with nothing raising an objection.

**The adjustment is a model and says so.** `smoothing_bias` assumes the
reported return is a two-period weighted average, infers the weights from the
first-order autocorrelation and reports the implied understatement. The result
carries `modelled=True`, so it can never be confused with the run counts,
which are arithmetic. A negative autocorrelation is bid-ask bounce rather than
staleness and returns weights of one and zero rather than pretending.

**Where the model cannot fit, it refuses.** A two-period average cannot
produce a first-order autocorrelation above one half, so a larger value is
evidence of something else — a trend, a longer memory, an overlapping sampling
window. `fits` comes back False and no figure is offered.

```console
$ mdnorm staleness returns.csv --returns
observations         2999
unchanged            0 (0.00%)
autocorrelation      +0.3368
weight on today      0.721
variance reported    0.5975 of the truth (modelled)
volatility           0.7730x understated
Sharpe               1.2937x overstated
```

Nothing is unsmoothed in place and no repeated value is dropped. Both would be
corrections applied to data whose cause has not been established.

### How many observations you actually have

A five-day forward return sampled every day gives you a thousand rows and
about two hundred pieces of information. Every t-statistic, Sharpe,
confidence interval and p-value computed on the thousand is overstated by
roughly the square root of five, and nothing in the arithmetic complains:

```python
from mdnorm import label_spans, effective_sample_size, deflate_t_stat

sample = effective_sample_size(label_spans(1_000, horizon=5))
sample.nominal, sample.effective     # 1000, 200.8
sample.inflation                     # 2.232x on every t-statistic
deflate_t_stat(Decimal("2.1"), sample)   # 0.941
```

`forward_returns` produces exactly those overlapping labels and
`purged_splits` already removes the training rows whose windows reach into a
test block. This is the other half of the same problem: purging stops the
overlap leaking *across* a split, and nothing stops it inflating the sample
*within* one.

**Overlap is arithmetic, not an estimate.** Given each label's window you can
count how many are live at every point. A label sharing its window with four
others is worth a fifth of an observation; summing that gives the effective
count exactly, with no model and no assumption. `uniqueness` exposes the
per-label figure and `concurrency` the step function underneath it.

**Autocorrelation is an estimate, and it says so.** For a return series with
no explicit windows there is only the sample autocorrelation, which is itself
noisy, so `effective_sample_size_series` marks its answer `estimated`. The
sum stops at the first non-positive autocorrelation — past that the terms are
noise whose signs cancel arbitrarily and can produce an effective sample
*larger* than the real one, which is the one direction this module exists to
rule out. On an AR(1) it lands on the closed form `(1-φ)/(1+φ)`, which the
test suite checks.

**No default lag, no default horizon, no silent correction.** How far the
dependence reaches is a property of your data. `deflate_t_stat` returns the
adjusted figure and the report keeps both counts, because a statistic that
has quietly been divided by something is harder to argue with than one that
shows its working.

```console
$ mdnorm independence --count 1000 --horizon 5 --t-stat 2.1
nominal sample       1000
effective sample     200.80  (exact)
ratio                20.1%
t-statistic inflated 2.232x
t-statistic adjusted 0.941
note: that crosses the conventional two-sigma line in the wrong direction.
The overlap did it, not the strategy.
```

### The crosses are not points on the tape

The opening and closing auctions are single prints, at a single price,
aggregating orders that never met each other in a book. Treating them as
ordinary trades is wrong in a direction that flatters:

```python
from mdnorm import auction_windows, auction_report, vwap_gap

windows = auction_windows(days, calendar)     # from the calendar, not a constant
auction_report(trades, windows).volume_share  # what went through the crosses
vwap_gap(trades, windows).difference_bps      # what that does to the benchmark
```

**An auction print has no aggressor.** Nobody crossed a spread; a clearing
price was computed. The tick rule and the quote rule will still return a side,
because those functions always return a side, and it is an artefact of where
the last continuous print happened to sit.

**Execution benchmarks are where it costs money.** A VWAP with the closing
cross in it is dominated by one print. A strategy that never touches the
auction, scored against that benchmark, is being measured against a price it
could not have obtained; a strategy that only trades the auction beats it by
construction. `vwap_gap` reports both numbers and the distance between them.

**Auctions are not inferred.** No condition-code guessing, no rule that a
print ten times the median must be a cross — on a busy day that rule
reclassifies ordinary blocks and the resulting statistic describes the
threshold rather than the market. Windows come from a `TradingCalendar`, so a
half-day's cross lands where the venue actually closed rather than three hours
later. The window extents default to zero: wide enough for a print stamped at
the bell and nothing else, because "thirty seconds, everyone uses that" is a
constant that differs by venue and by decade.

Nothing is deleted. `split_auctions` hands back both halves, and
`largest_print_share` is measured even with no windows at all — a file where
one print is a tenth of the day has a cross in it whether or not anything has
been told where.

```console
$ mdnorm auctions trades.csv --calendar us_2026.csv --session 09:30-16:00 \
      --tz America/New_York --ts-unit ns
trades               3730
  in an auction      20
volume in auctions   67.86%
notional in auctions 67.89%
VWAP with auctions   100.103008
VWAP without         100.008100
benchmark difference +9.49 bps
```

### Stored in nanoseconds is not measured in nanoseconds

Every timestamp here is an integer nanosecond. That is a storage decision. A
vendor that stamps to the millisecond and hands you nanoseconds has multiplied
by a million, and the extra six digits are zeros wearing the clothes of
precision:

```python
from mdnorm import detect_resolution, classification_risk

res = detect_resolution(ts)
res.granularity_ns, res.overstated_digits   # 1_000_000, 6
res.tied_share                              # 76% of rows share a timestamp

risk = classification_risk(events)
risk.same_tick, risk.changed                # exposure, and what it moves
```

**Resolution is detectable, and detection is divisibility.** A millisecond
feed leaves every timestamp a multiple of a million. The module walks the
decimal ladder — nanosecond up to a second — and reports the coarsest unit
that divides everything. Only decimal units, because those are the units a
clock is read in; a divisor of 2,000,000 would be arithmetic rather than a
statement about the venue.

**Divisibility needs enough observations to mean anything.** Twenty distinct
timestamps all being multiples of ten is a one-in-10²⁰ coincidence on a real
nanosecond feed, so twenty is plenty — three is not. Below the threshold the
answer is *undetermined*, which is not the same answer as one nanosecond. The
threshold counts distinct values, since a thousand copies of one round number
is one reading of the clock.

**A tie is not an ordering.** Rows sharing a timestamp are in the order the
writer used — an unstable sort, a queue that interleaved, a buffer flushed
however it was held. Reading sequence out of them is reading the writer.

**Where it costs something is trade classification.** The quote rule matches a
trade against the newest quote at or before it. When that quote carries the
same timestamp as the trade, which came first is not in the data.
`classification_risk` counts those trades, then re-runs the rule against the
last quote that is provably earlier and reports how many sides actually move.

```console
$ mdnorm resolution trades.jsonl
distinct timestamps  3000
resolution           1ms
padding digits       6
tied timestamps      3704 (76.34%)
largest tie          2
trades               1852
  same-tick quote    1852 (100.00%)
  side would change  815 (44.01%)
```

Nothing here re-sorts, jitters, or invents a finer timestamp. The resolution
you have is the resolution you have.

### The shape of the day, fitted without the rest of the year

Volume, spread and volatility all follow a curve inside a session — heavy at
the open, thin at midday, heavy into the close. Any statistic computed across
a day without removing that curve is mostly measuring the time of day:

```python
from mdnorm import Sample, US_EQUITY_RTH, session_profile, deseasonalise

shape = session_profile(volumes, US_EQUITY_RTH, bucket_ns=5 * 60 * 10**9)
shape.factor_at(0)                      # 2.09x — the first five minutes
shape.sessions, shape.excluded          # what the curve is actually built on

adjusted = deseasonalise(volumes, US_EQUITY_RTH,
                         bucket_ns=5 * 60 * 10**9, min_sessions=20)
```

**Removing the shape is the easy half; not reading the future while you do it
is the hard half.** The usual recipe fits one profile over the whole sample
and divides every day by it, including the first. That profile contains days
that had not happened yet, so an unusually heavy open in January is judged
against a curve that already knows December. `expanding_profiles` gives each
session a profile built only from the sessions before it, and `deseasonalise`
uses it. `full_sample_deseasonalise` is the other one, kept deliberately: it
is the better estimate for describing a market and the wrong input to
anything that trades, and shipping both is what lets `profile_leak` measure
the difference instead of arguing about it.

**No default bucket size.** Five minutes over a 6½-hour session is 78
buckets; the same five minutes on a venue that never closes is 288. Finer
buckets describe the curve better and put less evidence in each, and where
that trade sits is a property of your data.

**A thin bucket reports nothing rather than the average.** Filling it with
the overall mean makes the adjusted series look well-behaved in exactly the
places where nothing is known about it, so `min_observations` sets the bar and
the bucket comes back empty below it. A point with no factor leaves the
output; a point silently divided by one would be a point claiming to have
been adjusted.

**A short session is not a quiet one.** Bucketing by offset from the open
puts a half-day's closing surge into a bucket that is mid-afternoon on every
other day. Given a `TradingCalendar`, early closes are left out and counted.

```console
$ mdnorm seasonality volume.csv --session 09:30-16:00 --tz America/New_York --bucket 5m
sessions used        60
buckets              78 of 5m
heaviest bucket      +6h25m into the session (2.09x)
lightest bucket      +3h55m into the session (0.47x)
comparable samples   3120 of 4680
factor differs by    >0.01: 2533 (81.19%)
median gap           3.59%
largest gap          30.46%
```

### Features that cannot see the future

With the matrix built, the next step is turning prices into returns, z-scores,
volatility and correlations. This is the second place look-ahead gets in, and
it gets in just as quietly:

```python
from mdnorm import ReturnMethod, column, returns, rolling_zscore, realized_volatility

px  = column(rows, "BTC")
r   = returns(px, method=ReturnMethod.LOG)
z   = rolling_zscore(px, window=60)          # trailing, never full-sample
vol = realized_volatility(r, window=60)      # per period until you annualise it
qty = rolling_sum(volumes, window=60)        # O(n), not O(n x window)
```

**A full-sample z-score is look-ahead.** Subtracting the mean and dividing by
the standard deviation *of the whole series* gives every observation knowledge
of the distribution it sits in — including the part that had not happened yet.
It is one line of code and it is everywhere. Every statistic here is trailing:
the value at `i` comes from `values[i-window+1 : i+1]` and nothing else. There
is a test that pins this as a property — change the tail of the input and every
earlier output must be byte-identical — and a second test that shows the
full-sample form failing it.

**A partial window is not a result.** Until the window fills you get `None`,
not a twenty-period statistic computed from three observations. A gap inside a
window propagates for the same reason: stepping over the hole would compute a
twenty-period number from nineteen and label it twenty.

**A frozen series has no z-score and no correlation.** Zero dispersion makes
both undefined, so both return `None` rather than `0`. Reading that zero as a
correlation is how a dead feed becomes an apparent diversifier.

**There is no default annualisation factor.** √252 is right for daily bars on a
252-day calendar and wrong for almost everything else. `realized_volatility`
returns per-period volatility unless you pass a factor, and `periods_per_year`
makes you state the calendar rather than assume one — the same minute bars are
525,600 periods a year on a continuous venue and 98,280 on a cash equity
session.

**The trailing sum is slid, and only where sliding is exact.** `rolling_sum`
and `rolling_mean` no longer resum the window at every index, which took them
from O(n × window) to O(n) — window 250 now costs the same as window 60, 21×
faster than before. The reason libraries avoid this is drift: a running
`Decimal` total rounds differently from a fresh sum. That rounding is
observable, so every update runs with the `Inexact` flag cleared and is thrown
away the moment it would round, falling back to a full sum of the window. On
ordinary data every output is unchanged; where one does change it is because
the forward recomputation rounded an intermediate partial and the slid total
did not, and the test suite checks against rational arithmetic that the slid
answer is the exact one. The variance pass inside `rolling_std` and
`rolling_zscore` is deliberately *not* slid — the identity that would allow it
is a different sequence of roundings, and that is a trade we decline.
[BENCHMARKS.md](BENCHMARKS.md) has the numbers and the argument.

```console
$ mdnorm features matrix.csv --returns log --zscore 60 --vol 60 \
      --interval 1m --sessions-per-year 365 --session-length 24h -o feats.csv
```

### Labels, and a split that does not leak

A label is the one series in a research dataset that is *allowed* to look
forward — it is the thing you are predicting. That makes it the series which
quietly contaminates every split it touches:

```python
from mdnorm import forward_returns, purged_splits

y = forward_returns(prices, horizon=5)
for split in purged_splits(len(prices), n_splits=5, horizon=5, embargo=60):
    train, test = split.train, split.test
```

**A label with a horizon makes neighbouring rows overlap.** If the label at
row `i` spans the next five bars, rows `i` through `i+5` all describe the same
stretch of future. Put row `i` in train and row `i+3` in test and the model has
already seen most of the answer. Shuffling does not help — the rows genuinely
are different rows, they merely share an outcome. `purged_splits` drops the
training samples whose label window reaches into each test block, and reports
how many it dropped.

**A gap after the test block is not enough, because features have memory.** A
rolling statistic computed just after a test period is built partly from
observations inside it. The `embargo` removes the training rows immediately
following each block; set it to at least your longest feature window. It
defaults to 0 because the right value is a property of your features, not of
this function.

**`forward_returns` looks forward on purpose.** It is the only function in the
library that does, which is why it lives in `mdnorm.labels` rather than
`mdnorm.features`. Its output belongs on the left-hand side of a model; feeding
it back in as an input is not a subtle mistake.

The purging and embargo scheme follows López de Prado, *Advances in Financial
Machine Learning* (2018), ch. 7.

```console
$ mdnorm labels feats.csv --column BTC --horizon 5 --splits 5 --embargo 60 -o ml.csv
```

### The instruments that existed then

Two of the biases this library guards against are about time. The third is
about membership: a universe assembled today did not exist in the past.

```python
from mdnorm import Universe, Listing, cross_section, cross_sectional_rank

pit = Universe([Listing("AAA", listed_ns=...), Listing("BBB", listed_ns=..., delisted_ns=...)])
ranks = cross_section(rows, cross_sectional_rank, universe=pit)
```

**Survivorship bias produces no strange values anywhere.** Take the names
listed and liquid now, pull their history, rank them against each other over
ten years, and every instrument in the study is one that survived. Unlike a
look-ahead bug there is nothing odd to spot — the numbers are all real, the
sample is just wrong.

**Excluding a name is not the same as it having no data.** A symbol that has
not listed yet, or delisted last month, belongs outside the cross-section
rather than inside it as a blank — because a blank gets treated as missing at
random, and the instruments that disappear from a market are the opposite of
random. `mask_to_universe` returns the number of cells it removed; over a long
window a count of zero usually means the listings file is present-day
membership.

**The size of the cross-section changes, and that is correct.** Percentile
ranks are computed against the members present at that moment, so the
denominator moves as instruments list and delist.

Ties share an average rank, missing names are ranked neither last nor middle,
and a flat cross-section has no z-score rather than a row of zeros.

```console
$ mdnorm universe matrix.csv --listings listings.csv --pct-rank -o pit.csv
$ mdnorm revisions gdp.csv -o published.csv
```

### Two feeds that disagree

`quality` inspects one feed and reports what looks wrong inside it. A second
feed asks the question desks actually use to decide whether a vendor can be
trusted.

```python
from mdnorm import reconcile, suggest_shift

report, mismatches = reconcile(primary, secondary,
                               relative_tolerance=D("0.0001"))
report.agreement            # of the shared timestamps, how many matched
report.coverage_difference  # timestamps only one of them had
```

**The two kinds of disagreement are not one number.** A timestamp one feed has
and the other does not is a coverage difference — a dropped message, a
filtered print, a venue one side does not carry. A timestamp both have with
different values is a content difference, and at least one of them is wrong
about something checkable. `agreement` is computed over shared timestamps
only, so a feed that simply carries less does not look like a feed that lies.

**There is no default tolerance.** Two feeds of the same instrument differ in
the last digits for reasons that are not errors, and a constant deciding how
much is acceptable is a judgement about your data rather than a property of
it. With none given, values must match exactly — the strictest reading, and
one that states its own assumption.

**Zero overlap almost never means total disagreement.** It usually means a
clock offset: one feed stamps at the venue, the other on receipt, exact
matching finds nothing in common, and the naive conclusion is that the feeds
are unrelated. `suggest_shift` looks for the constant offset that lines them
up and reports how much of the sample it would explain. It does not apply it —
a clock difference is a fact about two systems that somebody should confirm.

```console
$ mdnorm reconcile primary.csv vendor.csv --relative 0.0001 -o breaks.csv
```

### Who was in the index, and when they were told

`universe` applies a membership record. Producing one from the files a vendor
actually ships is a separate job, and it is where survivorship gets in.

```python
from mdnorm import MembershipHistory, Basis, survivorship_gap

history = MembershipHistory.from_changes(changes)
history.members_at(t, basis=Basis.EFFECTIVE)   # who was in the index
history.report()                                # what the file cannot say
survivorship_gap(history, t)                    # what a today-list would cost
```

**Two dates, and they answer different questions.** An addition is announced
on one day and takes effect on another. *Who was in the index* is the
effective date; *when could anyone have known* is the announcement. Rank on
one and trade the other and the file will never object, because both columns
are correct. `Basis` has no default, so the question has to be named.

**A snapshot cannot express a deletion.** Names that leave do not appear as
departures, they simply stop being listed, and the last file that showed them
is not the day they left. `from_snapshots` therefore dates each inferred change
at the *later* snapshot — never claiming membership earlier than the file
supports — and records the window it really fell inside. On a monthly file
that window is a month, which is longer than many holding periods.

**A today-list is the classic error, and it is measurable.**
`survivorship_gap` returns both directions: the names a today-list drops
(they left, usually not for good reasons) and the names it holds too early
(they had not joined yet). The two do not cancel — one removes losers and the
other adds winners — which is why the effect is large and one-directional.

**The report names the tell.** If nothing ever left, the file is a list of
today's members wearing a history's clothes, and `mdnorm membership` says so
out loud rather than computing quietly on it.

```console
$ mdnorm membership index_changes.csv --at 1770000000000000000
```

### Values that get corrected later

Every observation so far has had one timestamp: when it happened. A lot of real
data has two — the period it describes, and the moment it became knowable — and
then it gets revised.

```python
from mdnorm import Revision, RevisionSeries

series = RevisionSeries([
    Revision(event_ts_ns=q1, known_ts_ns=april, value=D("2.1")),
    Revision(event_ts_ns=q1, known_ts_ns=may,   value=D("1.6")),   # revised down
])
series.as_of(event_ts_ns=q1, known_ts_ns=april_20)   # 2.1 — what you knew
series.final(event_ts_ns=q1)                          # 1.6 — what is true now
```

**Using the corrected value is look-ahead, and no timestamp check will catch
it.** The row is dated correctly. The value is a real number that was genuinely
published. Nothing marks it as unavailable until three weeks later. Every guard
in `mdnorm.align` passes and the study is still wrong.

**Two honest questions, two different objects.** *What was the newest published
number at time t* is a feature — `known_series()` is keyed by publication time
and joins like any other stream. *What did the whole table look like at time t*
is a vintage — `vintage_at(t)` is keyed by event time and reproduces the sheet
as it appeared that day. Reading a vintage at the wrong moment gives a value
nobody had; there is a test that shows exactly that.

**Measure it rather than assuming.** `revision_summary()` reports how many
events were ever revised and how far first releases sat from final values. If
that number is large, every backtest built on final data has been reading
answers.

```console
$ mdnorm revisions gdp.csv -o published.csv
```

### A daily number on an intraday grid

A daily close, a settlement price, an overnight risk figure — slow series meet
fast grids constantly, and they almost always arrive labelled with the period
they *describe* rather than the moment they became *knowable*.

```python
from mdnorm import PeriodSeries, leak_report, US_EQUITY_RTH, grid

series = PeriodSeries.from_sessions(daily_closes, US_EQUITY_RTH)
feature = series.knowable_series()        # keyed at the close — safe to join
report  = leak_report(series, grid(...))  # what the label join would have cost
```

**A daily bar labelled Tuesday is not knowable on Tuesday morning.** It is
knowable once Tuesday's session closes — Tuesday evening, and later still if
the number has to be published. Join it by its label and every minute of
Tuesday sees a value that summarises, among other things, the rest of Tuesday.

**The session decides the close, not the file.** Daily bars are frequently
stamped midnight to midnight regardless of when the market was open.
`from_sessions` and `from_daily_bars` take a `Session`, so a 16:00 New York
close is 21:00 UTC in January and 20:00 in July without the caller thinking
about it.

**Publication lag is a separate claim.** A settlement price exists at the
close; it reaches you when the vendor sends it. `publication_lag_ns` is where
that goes and it defaults to zero, because a lag of zero is a statement about
your feed that only you can make.

**Measure the leak instead of arguing about it.** `leak_report` counts the grid
points where the label join shows a value that did not yet exist, and how far
ahead the worst one was read. On back-to-back periods the answer is *every
point*: the moment one value becomes readable the label has already moved to
the next. Whether that ruins a study depends on the signal, which is exactly
why the number is worth having.

```console
$ mdnorm mixfreq daily.csv --interval 60000000000 --lag 900000000000 -o joined.csv
```

### The square root of time is a claim about independence

Annualising a Sharpe ratio means multiplying by the square root of the
calendar. That step looks like a unit conversion and is an assumption: it is
correct only if the returns are serially uncorrelated. A smoothed series — an
appraisal mark, a stale quote, a model price on something that did not trade —
has an autocorrelation well above zero, and then the familiar factor is simply
the wrong number.

```python
from mdnorm import serial_report

rep = serial_report(monthly, periods_per_year=12, max_lag=11)

rep.first_order              # 0.238055 — last month is still in this month
rep.factor.naive             # 3.464102 — the square root of twelve
rep.factor.corrected         # 2.509901 — what this series supports
rep.factor.naive_overstates  # True

rep.sharpe                   # 0.193901 per month
rep.naive_annualised         # 0.671692 — the figure that gets reported
rep.corrected_annualised     # 0.486672 — the one the series earns
rep.variance_ratio           # 1.911987 — the same fact without a Sharpe
```

**The correction is Lo's, and it reduces to the familiar factor.** For a
horizon of `q` the honest multiplier is `q / sqrt(q + 2 * sum (q - k) * rho_k)`.
Set every autocorrelation to zero and it returns `sqrt(q)` exactly. The square
root of time is the special case, not the general rule.

**The direction is not fixed.** Negative autocorrelation — bid-ask bounce, a
mean-reverting spread — makes `sqrt(q)` *understate*. `naive_overstates` says
which case a series is in, and the gap is called `difference` rather than an
overstatement because a name asserting a direction would be wrong half the
time.

**Truncation is reported, because it flatters.** Lo's factor wants `q - 1`
autocorrelations; annualising daily returns wants 251, and past the first few
those rest on too little data to mean anything. Supplying fewer is normal and
treats the rest as zero, which pulls the answer back toward the naive one.
`ScalingFactor` carries `lags_used`, `lags_needed` and `truncated`, so a
truncated result reads as a lower bound on the correction rather than an
estimate of it.

**Two independent corroborations, not one number.** `variance_ratio` asks
whether `q`-period variance is `q` times one-period variance, and
`long_run_variance` is the Newey-West figure that belongs under the square
root of a standard error. On the series above they agree with each other —
1.91 and 1.90 against an ordinary variance — which is the check worth running
before trusting any of the three.

**The estimator is biased toward independence, and we say so.** The variance
ratio drifts toward one at long horizons because overlapping windows share
observations: on four thousand draws from a process whose asymptotic ratio is
1.86, it returns about 1.66. So a ratio near one is weak evidence of
independence and a ratio far from it is strong evidence against — the error
runs in the direction that makes a dependent series look clean.

```console
$ mdnorm serial monthly.csv --periods 12 --max-lag 11 --show 4 --hac 11
```

### A maximum drawdown is a maximum

The drawdown figure on a tear sheet is the worst single observation in the
sample. That makes it an order statistic, and order statistics grow with how
long you look: run the same unchanged strategy for ten years instead of two
and its worst decline gets deeper, because there were more chances for a bad
run to happen. Two backtests of different lengths are not reporting the same
quantity.

```python
from mdnorm import underwater_report, resampled_max_drawdown

rep = underwater_report(equity)

rep.deepest              # 0.208839 — the number that gets published
rep.ulcer                # 0.088358 — reads every observation, not one
rep.pain                 # 0.071522
rep.concentration        # 0.423089 — ulcer over deepest
rep.underwater_share     # 0.9294 — 93% of the sample below a prior peak
rep.longest_underwater   # 281 observations without a new high
rep.open_at_end          # True — the sample ends in a decline
```

**The headline rests on one observation and the alternatives do not.**
`ulcer_index` is the root mean square depth over the whole curve and
`pain_index` is its mean. Neither can be moved by a single day. Their ratio to
the maximum says which kind of strategy you have: a concentration near one
means the curve spent the sample close to its worst, near zero means the
maximum was a single excursion the rest of the sample knows nothing about.

**Depth is not the part anyone lives through.** Down eight per cent for three
weeks and down eight per cent for three years report the same drawdown. On the
series above the maximum is 20.9 per cent and the strategy spent 93 per cent
of five years below a previous high, with one stretch of 281 trading days
without a new one. That second fact is usually absent from the report.

**A decline still open at the end is not a recovered one.** Nothing here
closes an open drawdown at the final observation, because that turns "we do
not know yet" into "it ended here".

**The length effect, measured rather than assumed.** `resampled_max_drawdown`
draws from your own returns to ask what the worst decline looks like over a
horizon you name:

| Horizon | Median worst drawdown | 90th percentile |
| --- | --- | --- |
| 1 year | 0.1344 | 0.2166 |
| 5 years | 0.2392 | 0.3732 |
| 10 years | 0.2982 | 0.4271 |

Same returns, same process, no change in risk. A two-year backtest reporting
13 per cent and a ten-year one reporting 30 per cent can describe the same
strategy.

**The resampling assumption is stated, and it flatters.** Drawing with
replacement destroys serial correlation, and losses that arrive in runs make
drawdowns deeper than independent losses do. On a positively autocorrelated
series — which is what [the previous section](#the-square-root-of-time-is-a-claim-about-independence)
measures — the resampled distribution therefore sits shallower than the truth.
It is a lower bound, not an estimate. No default horizon, path count or seed:
the caller states all three.

```console
$ mdnorm underwater pnl.csv --returns --horizon 252 1260 2520 --paths 1000 --seed 7
```

### A return has to beat something

Every ratio above measures a return against a hurdle, and the hurdle is
usually left at zero or set to one constant for the whole sample. Cash paid
close to nothing for a decade and then five per cent, so the first choice
credits a strategy with the cash return and the second one misdates it.

The series below is twenty years of a cash-plus book — the kind that holds
collateral, so its gross return mechanically contains the rate it was financed
at. Monthly returns are `rate + gauss(0.0025, 0.0085)` from
`random.Random(20260918)`, against a rate path that sits at 0.5 per cent for
eleven years, rises to 5 per cent over twenty months, holds, then eases.

```python
from mdnorm import hurdle_comparison, excess_report

cmp = hurdle_comparison(returns, rates, ddof=1)

cmp.sharpe_raw         # 0.486326 — no hurdle at all
cmp.sharpe_constant    # 0.260610 — the mean rate, subtracted once
cmp.sharpe_series      # 0.257868 — the rate actually paid, period by period
cmp.zero_hurdle_gap    # 0.228458
cmp.rate_correlation   # 0.051507

excess_report(returns, rates, ddof=1).share_credited_to_cash   # 0.464125
```

Annualised at twelve periods, those are the numbers a reader would see:

| Hurdle | Annualised Sharpe |
| --- | --- |
| None | 1.6847 |
| The mean rate, subtracted once | 0.9028 |
| The rate series, period by period | 0.8933 |

A book that presents as 1.68 is 0.89 once it is charged the cash it was
financed at, and 46 per cent of its gross return was the hurdle. Nothing in
the return series is wrong; the top row is answering a different question from
the bottom one.

**Not subtracting anything has a direction, and it is the flattering one.** A
non-negative rate can only make the raw figure the larger, so `zero_hurdle_gap`
is reported as a gap with a known sign.

**Subtracting a constant and subtracting a series are different operations.**
The first moves the mean and leaves the volatility alone; the second changes
both, because a moving rate has a variance of its own and a covariance with
the returns. `constant_series_gap` is 0.002742 here — the constant reads
*higher* — and the module refuses to promise that sign, because it follows the
correlation and the rate's variance in the sample rather than any general
argument. On a series that is mostly the rate, it flips.

**A quoted rate is an annual number and returns are not.** Five per cent over
252 periods is 0.000198413 divided and 0.000193631 compounded. The gap is
small per period, runs one way for the whole sample, and lands on the hurdle.
`per_period_rate` does both and makes you name which; there is no default,
for the same reason there is no default annualisation factor.

**Day-count bases are not interchangeable.** SOFR, EURIBOR and most deposit
quotes are on a 360-day year. Used against a 365-day calendar the same quote
understates the hurdle by 365/360 — 1.39 per cent of the rate, every period,
on the flattering side. `rebase` converts and refuses to guess which basis a
number arrived on.

**A benchmark is a hurdle too, and a worse-behaved one.** `active_returns`,
`tracking_error` and `information_ratio` measure against a series somebody
chose. Both means are reported, because an information ratio is a statement
about the strategy *and* about the benchmark, and a tracking error computed
with `ddof=0` on a two-year sample is not the one computed with `ddof=1`.

No cash curve ships here, for the same reason no factor data ships with
`exposure`: bundling one would make every answer partly a property of whose
curve we picked.

```console
$ mdnorm hurdle pnl.csv --rates cash.csv --benchmark index.csv
```

### How often the book is traded back is an assumption

A weight vector is a decision made once. What happens to it afterwards is
arithmetic: winners grow, losers shrink, and by the end of a month an
equal-weight book is not equal-weight any more. Most backtests quietly snap the
weights back to target at every observation, which earns a return nobody could
have had without trading, and then report no turnover at all.

Five names over five years, equal weight, one of them markedly more volatile
than the rest. Returns are `gauss` draws from `random.Random(20260919)` with
the per-name parameters in the test file. Nothing differs between these rows
except how often the book was assumed to be traded back:

| Schedule | Total return | One-sided turnover | Rebalances | Max drift |
| --- | --- | --- | --- | --- |
| Every period | 11.7901% | 5.6217 | 1259 | 0.0000 |
| Every 5 | 11.3488% | 2.4133 | 251 | 0.0213 |
| Every 21 | 10.1833% | 1.0955 | 59 | 0.0376 |
| Every 63 | 11.3573% | 0.6863 | 19 | 0.0726 |
| Every 252 | 10.9495% | 0.2868 | 4 | 0.1156 |
| Band, 2pp | 10.8574% | 0.9783 | 39 | 0.0199 |
| Never | 11.3466% | 0.0000 | 0 | 0.1523 |

```python
from mdnorm import periodic_rebalance, buy_and_hold, compare_schedules

cmp = compare_schedules({
    "every 1": periodic_rebalance(target, returns, every=1),
    "never": buy_and_hold(target, returns),
})

cmp.return_spread        # 0.00443514 — 44 basis points over five years
cmp.breakeven_cost_bps   # 7.8893
```

**The daily advantage is inside the cost of getting it.** Rebalancing at every
observation beats never rebalancing by 44 basis points and turns the book 5.6
times to do it. At 7.89 basis points per unit of one-sided turnover the
advantage is exactly gone, and above it the ranking reverses. That figure is
what `breakeven_cost_bps` reports: not a recommendation, the cost level at
which the argument changes sides.

**Turnover is monotone in the frequency and the return is not.** Read the
table again: 63 periods beats 5, which beats 252, which beats 21. There is no
ordering to find, because the differences are noise and the trading is not.
This is why the module reports both columns and recommends nothing.

**A band rule holds the book tighter than a calendar rule and trades less.**
The 2-percentage-point band keeps the worst drift at 0.0199 with 0.98 of
turnover; rebalancing every five periods allows 0.0213 and spends 2.41. A
calendar does not know whether anything has moved, and most of the time
nothing has.

**No costs are charged here.** [`costs`](#what-the-trade-costs) prices a trade;
this module says how much trading a rule implies. Keeping them apart means the
cost model stays something you state rather than something a schedule smuggled
in. The residual weight is cash at a zero return, which is an assumption —
[`hurdle`](#a-return-has-to-beat-something) is where it gets priced.

There is no default schedule, no default band and no default turnover
convention. One-sided and two-sided turnover differ by a factor of two and
both appear in print under the same word, so `turnover_between` makes you name
which.

```console
$ mdnorm rebalance panel.csv --every 1 21 252 --band 0.02 --hold --drift-band 0.02
```

### A backtest reports what the strategy earned; an investor keeps less

Every figure above is gross of the fees a fund charges its investors. That is
the right default for research and the wrong number to put in front of anyone
deciding whether to invest, because the fee is not a constant subtracted from
the return. A management fee compounds against the investor, and an incentive
fee is a share of the upside with no share of the downside.

Ten years of monthly gross returns, `gauss(0.0095, 0.035)` from
`random.Random(20260921)`: 116.92 per cent in total, about 8.05 per cent a
year. The same series through eight contracts:

| Contract | Net total | Share of the profit taken |
| --- | --- | --- |
| No fees | 116.92% | 0% |
| 2 and 20, annual, high-water mark | 55.44% | 52.59% |
| 2 and 20, quarterly, high-water mark | 53.19% | 54.51% |
| 2 and 20, monthly, high-water mark | 52.74% | 54.89% |
| 2 and 20, annual, no mark | 53.31% | 54.41% |
| 2 and 20, monthly, no mark | 19.54% | 83.29% |
| 1.5 and 15, annual, high-water mark | 67.98% | 41.86% |
| 2 and 0 | 77.77% | 33.48% |
| 0 and 20, annual, high-water mark | 83.62% | 28.48% |

```python
from mdnorm import FeeSchedule, apply_fees

res = apply_fees(gross, FeeSchedule(management=D("0.02"), incentive=D("0.20"),
                                    periods_per_year=12, crystallise_every=12,
                                    high_water_mark=True))

res.gross_total           # 1.169200
res.net_total             # 0.554361
res.fee_share_of_profit   # 0.525863
```

**The headline rate is not the share of profit.** Under the most ordinary
contract in the table the investor kept less than half of what the strategy
earned: 8.05 per cent a year gross became 4.51 net. Even the row with no
management fee at all took 28.48 per cent of the profit on a twenty per cent
incentive, because a gain crystallised in one year is not returned when the
next one loses.

**How often the fee crystallises is a free parameter that moves the answer.**
Nothing in the gross returns changes down the table. Monthly crystallisation
without a high-water mark took 83 per cent of the profit on the same returns
that an annual schedule with a mark took 53 per cent of. This is the argument
from [the previous section](#how-often-the-book-is-traded-back-is-an-assumption)
one layer up: an unstated schedule that decides how much is taken.

**The Sharpe ratio falls by less than the return, and that is not good
news.** Annualised, the ratio goes from 0.6699 to 0.4063 — down 39 per cent —
while the total return falls by 53. The ratio is built from the average
monthly return, and averages do not compound: the fees cut the average month
by 39 per cent, and ten years of compounding turn that into a total 53 per cent
smaller. Volatility is not the reason. Under annual crystallisation it rises
slightly, because the fee lands as a few large deductions; crystallise monthly
and it falls a little. `FeeResult.sharpe` offers both ratios so that a 39 per
cent fall in one is not read as a fee that took 39 per cent.

An earlier version of this paragraph, in 1.46.0, attributed the gap to the
incentive fee trimming volatility. On this series that is wrong, and the
CHANGELOG says so.

**The investor leaves at the end of the sample.** An incentive fee accrued
since the last crystallisation is charged at the final observation rather than
dropped, because a net figure that ignores an accrued fee is one nobody could
have redeemed at. A hurdle, when given, grows the high-water mark period by
period; [`hurdle`](#a-return-has-to-beat-something) turns a quoted rate into
the per-period series it needs.

There is no default schedule. "Two and twenty" is four decisions, and the two
that are not in the name are the ones the table shows moving the answer. The
module is `fundfees`, not `fees`, so it is not mistaken for `costs.Fees`,
which prices a trade.

```console
$ mdnorm fees gross.csv --management 0.02 --incentive 0.20 \
    --periods-per-year 12 --crystallise-every 12 3 1 --compare-hwm
```

### Choosing the best backtest is a procedure, and it can be tested

Every research process ends with a choice: of the variants tried, keep the one
that looked best. The deflated Sharpe ratio asks how good that winner would
have looked by luck. `selection` asks a question that needs no distributional
assumption: does the *procedure* of picking the in-sample winner choose
something that does well out of sample?

It uses combinatorially symmetric cross-validation (Bailey, Borwein, López de
Prado and Zhu, 2017). Cut the sample into an even number of contiguous blocks;
for every way of using half of them in-sample and half out-of-sample, find the
in-sample winner and record where it finished out of sample.

Twenty variants, a thousand daily returns each, every one drawn with the same
true edge — `gauss(0.0002, 0.01)` from `random.Random(20260922)`, an
annualised Sharpe of about 0.32. Ten blocks, 252 splits:

```python
from mdnorm import cscv

rep = cscv(variants, blocks=10, metric="sharpe", ddof=1)

rep.pbo                    # 0.7540 — the probability of backtest overfitting
rep.mean_is_best           # 1.7260 annualised: how good the winner looked
rep.mean_oos_of_is_best    # 0.2128: what the same winner went on to do
rep.mean_oos_all           # 0.4979: what an average variant did
rep.degradation_slope      # -0.5604: a better-looking winner did worse
```

| | Probability of overfitting | Winner in-sample | Winner out-of-sample | Average variant out-of-sample |
| --- | --- | --- | --- | --- |
| Twenty variants, identical edge | 0.7540 | 1.7260 | 0.2128 | 0.4979 |
| Same, with one genuinely better variant | 0.3889 | 2.0117 | 0.7707 | 0.5530 |

Annualised Sharpe ratios at 252 periods a year.

**On identical strategies, picking the winner was worse than picking at
random.** The chosen variant earned 0.21 out of sample where an average pick
earned 0.50. The probability is above one half rather than at it because the
two halves of each split are complements: for a given whole-sample result, a
variant that did well in one half did relatively worse in the other.

**It measures the search, not a strategy.** Replace one of the twenty with a
variant that has a real edge and the probability falls to 0.39; that variant is
chosen in 130 of the 252 splits. Same method, same noise, different answer,
because the search now contains something worth finding.

**The winner's in-sample figure is flattering by construction.** It is the
maximum of twenty noisy estimates. On the full sample, the best of the twenty
identical variants shows an annualised Sharpe ratio of 1.26 against a true
value of about 0.32.

Blocks are contiguous so serial structure inside them survives, and a sample
that does not divide into the stated number of blocks is refused rather than
trimmed. Ties in-sample go to the first name in sorted order, and the
out-of-sample rank counts only variants strictly below the winner, so a tie
never flatters it. No default block count and no default metric: both decide
the answer.

```console
$ mdnorm selection variants.csv --blocks 10 --metric sharpe --ddof 1 --annualise 252
```

### A strategy earns a return. An investor earns a rate on the money that was there

Every return series above describes one unit of capital held from the first
observation to the last. Nobody invests that way. Money arrives after a good
year and leaves after a bad one, and the chain-linked figure in the backtest is
deliberately blind to all of it.

`flows` reports both rates over one window: the time-weighted return, which
judges the strategy, and the money-weighted return — the internal rate of the
investor's cash flows — which is what the money earned.

The textbook case first, because it needs no simulation. A strategy doubles and
then halves: chain-linked, it returned exactly zero. An investor who put in 100
at the start and another 100 after the good period ends with 150 out of 200
contributed.

```python
from mdnorm import flow_report

rep = flow_report([D(1), D("-0.5")], [D(100), D(100)], when="start")

rep.time_weighted          # 0.0000  — what the strategy did
rep.money_weighted         # -0.1771 — per period, what the money did
rep.money_weighted_total   # -0.3229 — over the window
rep.profit                 # -50: the whole of it belongs to the timing
```

Then five years of one unchanged strategy — sixty monthly returns from
`random.Random(20260923)`, `gauss(0.008, 0.045)` — with three flow paths over
it. The chasing path contributes 20 a month after a positive trailing year and
5 after a negative one; the contrarian path does the opposite; the level path
contributes 10 every month regardless.

| Flow path | Contributed | Money-weighted | Against the strategy |
| --- | --- | --- | --- |
| Level, 10 a month | 600 | 0.6292 | +0.0856 |
| Chasing the trailing year | 975 | 0.6101 | +0.0664 |
| Contrarian | 465 | 0.6661 | +0.1224 |

The strategy returned **0.5437** over the same sixty months in every row.

**Every difference in that column belongs to the schedule, not to the
manager.** The spread between the best and worst path is 5.60 percentage
points on returns that never changed. Chasing earned the least of the three
and contributed the most to do it.

**Part of the gap is arithmetic, not behaviour.** A money-weighted rate is a
capital-weighted average of the periods and a chain-linked one is a geometric
average, so the two differ even on a level schedule with no timing in it at
all. `timing_effect` holds the amount fixed and contributes it in equal parts,
leaving only the part a decision could have changed: **-0.0192** for the
chasing path.

**The timing convention is required.** A contribution at the start of a period
earns that period's return and the same contribution at the end does not, so
`when` has no default.

**The approximation in every performance report is shown with its error.**
Modified Dietz divides the gain by a weighted capital base and never
compounds, so it is exact only when a single flow opens the window. Twelve
monthly contributions at a flat one per cent a month come out at 12.4512 per
cent against a true 12.6825, and on the sixty-month level path it reads 0.5791
against 0.6292 — an error of five percentage points in the direction that
understates.

An internal rate of return can have more than one root when the cash flows
change sign more than once. `root_unique` says whether that condition holds,
and `npv` lets any rate be checked directly rather than taken on trust. The
rate itself is found by bisection, so it does not depend on a starting guess.

```console
$ mdnorm flows account.csv --when start --periods-per-year 12 --level
```

### How much of the result is the search

Everything above is about getting the data right. The last step is a correct
dataset that still produces a misleading number, because the number was
chosen. A Sharpe ratio from one strategy is an estimate; the same ratio kept
after trying two hundred parameter sets is a maximum, and the maximum of two
hundred draws from noise is not small.

```python
from mdnorm import sharpe_report

rep = sharpe_report(daily_returns, periods_per_year=D(252),
                    trials=500, trial_sharpe_variance=D("0.004"))

rep.sharpe_annualised   # 0.56  — the figure that goes in the deck
rep.probabilistic       # 0.92  — probability the true ratio is above zero
rep.deflated            # 0.008 — after accounting for 500 attempts
rep.demonstrated        # False — the sample is shorter than it needs to be
rep.warnings            # what the headline number does not say
```

**Ratios are per period until you state the calendar.** `sharpe_ratio` divides
mean by standard deviation and stops; `annualise_sharpe` needs a factor, for
the same reason `realized_volatility` does. Being wrong by a constant is the
hardest kind of wrong to notice, because the shape of the series is unchanged.

**A short track record is not evidence.** `min_track_record_length` says how
many periods a ratio needs before it is distinguishable from the benchmark.
A strategy whose minimum is nine years and whose backtest is eighteen months
has not been demonstrated, however good the ratio looks.

**Selection is measurable.** `expected_max_sharpe(trials, variance)` is the
best ratio a search of that size produces from strategies that are all
worthless. `deflated_sharpe_ratio` measures your result against that instead
of against zero, following Bailey and López de Prado (2014). Pass the whole
search, not the survivors.

**Nothing here returns a flattering placeholder.** A series that never moved
has no Sharpe, a sample with no losing period has no measurable downside, a
curve that never fell has no drawdown — all `None`, not zero and not infinity.
Each of them is a statement about the sample being short.

```console
$ mdnorm metrics pnl.csv --column ret --interval 1d \
    --sessions-per-year 252 --session-length 6h \
    --trials 500 --trial-variance 0.004
```

### What the trade costs

`mdnorm.execution` measures what your fills actually cost. This is the other
question: what a backtest should charge itself for a trade it never made. It
is the crudest way a result flatters you and it survives every other check,
because nothing in the data is wrong — the strategy is simply being priced at
a level nobody trades at.

```python
from mdnorm import CostModel, Fees, ImpactModel, Liquidity, estimate, capacity

model = CostModel(fees=Fees(taker_bps=D(1)),
                  impact=ImpactModel(coefficient=D("0.5")))   # no default
liq = Liquidity(adv=D(1_000_000), volatility=D("0.02"), spread_bps=D(4))

b = estimate(model, notional=D(500_000), quantity=D(20_000), liquidity=liq)
b.commission_bps   # 1.0
b.spread_bps       # 2.0   — half of the quoted spread, because you crossed
b.impact_bps       # 14.1  — 2% of daily volume, square-root law
b.total_bps        # 17.1

capacity(D(20), model=model, liquidity=liq)   # 28,900 a day at a 20 bps edge
```

**Zero cost is not a default, it is a claim.** A backtest that charges nothing
has asserted that it trades at the midpoint, in unlimited size, for free.
Written down that way nobody would sign it.

**A cost that does not depend on size is not a cost model.** A flat five basis
points says a strategy can trade a thousand dollars and a billion on identical
terms, so every capacity question has the same answer. `estimate` says so in
its warnings every time an impact model is absent.

**There is no default impact coefficient.** The square-root law is well
supported; the constant in front of it is not universal, and a plausible wrong
one rescales every cost in the report while changing nothing about its shape.
Calibrate it against your own fills — that is what `evaluate` is for.

**The useful output is not the cost.** `breakeven_participation` is the
fraction of daily volume at which the edge is exactly consumed, and `capacity`
is the same figure as a quantity. A two-basis-point edge that breaks even at
0.3% of volume is a different object from the same edge breaking even at 30%,
and no Sharpe ratio distinguishes them.

```console
$ mdnorm costs pnl.csv --column ret --turnover-column turnover \
    --cost-bps 5 --edge-bps 20 --adv 1000000 --volatility 0.02 \
    --spread-bps 4 --fee-bps 1 --impact-coefficient 0.5
```

### A ticker is not an identifier

`canonical_symbol` makes `BTCUSDT` and `XBT/USD` agree on a spelling. This is
the other problem: the same spelling, at two different times, meaning two
different things. Exchanges reuse ticker strings — a company delists and its
symbol is reassigned, a venue renames a pair and the old name reappears
elsewhere.

```python
from mdnorm import SymbolAssignment, SymbolMap, key_by_instrument, series_segments

smap = SymbolMap([
    SymbolAssignment("ABC", "US0000000001", start_ns=t0, end_ns=t1),
    SymbolAssignment("ABC", "US0000000002", start_ns=t2),   # reused later
])

smap.reused_symbols()                 # [("ABC", 2)] — the finding
smap.instrument_at("ABC", t_mid)      # None: in the gap it named nothing
rows, counts = key_by_instrument(rows, smap)
counts["reassigned"]                  # rows the string would have mis-joined
segments, unresolved = series_segments("ABC", timestamps, smap)
```

**The bias is a join, not a bad value.** Every price in a spliced series
genuinely traded, at its own timestamp, under the ticker it carries. What is
wrong is the assumption that the column header names one thing — made once,
silently, when the matrix is built.

**Reuse looks like a merger, and mergers look profitable.** A delisting is
usually a fall and a new listing starts at a normal price, so splicing one onto
the other inserts a jump. Half the time it is upward, and an upward jump in a
name you were holding is indistinguishable from a takeover premium. The series
does not look broken; it looks lucky.

**A gap is not filled with the next owner.** Between the delisting and the
reassignment the ticker named nothing, and `instrument_at` returns `None` there
rather than the instrument that took the letters afterwards. That substitution
is the splice.

**Overlaps are refused.** One ticker bound to two instruments at the same
moment is a broken reference file, and picking one of them quietly is how the
error reaches a study. `SymbolMap` raises instead.

**No reuse in a long history is a finding, not a pass.** A file with one
open-ended binding per ticker cannot express reuse at all, so a zero means the
file rather than the market — the same shape of diagnostic as a purge that
removes nothing.

```console
$ mdnorm instruments symbol_map.csv trades.csv --segments ABC -o keyed.csv
```

### Data quality

```python
from mdnorm.quality import find_issues, clean

find_issues(events)          # list of QualityIssue (outlier / gap / out_of_order / non_positive)
cleaned, issues = clean(events)  # drop bad ticks & invalid rows, keep a report
```

`clean` removes price outliers and non-positive price/size records and returns
the surviving events plus everything it flagged.

### Serialization

```python
from mdnorm import to_records

to_records(events)                 # list of flat dicts (Decimals as strings)
to_records(bars, as_float=True)    # numeric output for DataFrames
```

`to_records` (and `event_to_dict` / `bar_to_dict`) flatten events and bars into
plain, JSON-serialisable dicts — drop straight into `pandas.DataFrame`, a
`csv.DictWriter`, or `json.dumps`.

### Consolidating streams

```python
from mdnorm import merge_streams, dedupe

timeline = dedupe(merge_streams(binance_events, coinbase_events))
```

`merge_streams` interleaves multiple venue feeds into one timestamp-ordered
timeline; `dedupe` drops exact duplicate events left behind by reconnects and
replays.

### CSV files

```python
from mdnorm import read_csv_trades, write_records_csv

events = read_csv_trades("trades.csv", venue="coinbase")   # file -> events
write_records_csv(bars, "bars.csv", as_float=True)          # events/bars -> file
```

`read_csv_trades` parses a whole CSV of trades into normalized events;
`write_records_csv` writes events or bars back out. Standard library only.

### NDJSON / JSON Lines

```python
from mdnorm import write_jsonl, read_jsonl_events

write_jsonl(events, "events.jsonl")          # one JSON object per line
events2 = read_jsonl_events("events.jsonl")  # lossless round-trip

# large files: stream lazily, .gz handled transparently
for e in iter_jsonl_events("dump.jsonl.gz"):
    ...
```

### Pipelines

Declare a processing chain once, reuse it everywhere:

```python
from decimal import Decimal
from mdnorm import Pipeline

pipe = (
    Pipeline()
    .dedupe()
    .clean(max_return=Decimal("0.1"))
    .time_bars(60_000_000_000)   # 1-minute bars
    .fill_gaps()
)
bars = pipe.run(events)
print(pipe.last_issues)          # quality report from clean()
```

### Command line

The common conversions ship as a zero-dependency CLI:

```console
$ mdnorm bars trades.csv --venue binance --interval 1m -o bars.csv
$ mdnorm quality trades.csv --max-gap 5m
$ mdnorm convert trades.csv -o trades.jsonl
$ mdnorm bars trades.csv --interval 1d --actions actions.csv -o adjusted.csv
$ mdnorm align BTC=btc.csv ETH=eth.csv --interval 1m --max-age 5m -o matrix.csv
$ mdnorm features matrix.csv --returns log --zscore 60 --vol 60 -o feats.csv
$ mdnorm labels feats.csv --column BTC --horizon 5 --splits 5 -o ml.csv
$ mdnorm universe matrix.csv --listings listings.csv --pct-rank -o pit.csv
$ mdnorm revisions gdp.csv -o published.csv
$ mdnorm metrics pnl.csv --column ret --trials 500 --trial-variance 0.004
$ mdnorm costs pnl.csv --cost-bps 5 --edge-bps 20 --adv 1e6 --volatility 0.02
$ mdnorm instruments symbol_map.csv trades.csv -o keyed.csv
$ mdnorm calendar us_2026.csv --session 09:30-16:00 --tz America/New_York
$ mdnorm fx prices.csv rates.csv --from EUR --to USD --max-age 1m -o usd.csv
$ mdnorm ticks prices.csv --table ticks.csv
$ mdnorm arrival feed.csv --interval 1s
$ mdnorm seasonality volume.csv --session 09:30-16:00 --bucket 5m
$ mdnorm resolution trades.jsonl
$ mdnorm auctions trades.csv --calendar us_2026.csv --session 09:30-16:00
$ mdnorm independence --count 1000 --horizon 5 --t-stat 2.1
$ mdnorm staleness marks.csv --min-run 3
$ mdnorm halts trades.csv --halts halts.csv --decisions fills.csv
$ mdnorm coverage feed.csv --min-gap 5m --calendar us_2026.csv
$ mdnorm provenance run.json --verify --parameter trials=500
$ mdnorm extremes pnl.csv --sigma 5 --robust --tail 10
$ mdnorm windows pnl.csv --metric sharpe --trim-start 21 --deflate
```

Also available as `python -m mdnorm`.

## The unified schema

```python
@dataclass(frozen=True, slots=True)
class MarketEvent:
    symbol: str          # canonical "BASE-QUOTE", e.g. "BTC-USD"
    venue: str           # source venue
    event_type: EventType  # TRADE | QUOTE
    ts_ns: int           # nanoseconds since Unix epoch (UTC)
    price: Decimal | None
    size:  Decimal | None
    side:  Side | None     # BUY | SELL
    # ... plus bid/ask fields for quotes
```

## Benchmarks

Throughput for the hot paths, measured by a script in this repository rather
than asserted: [BENCHMARKS.md](BENCHMARKS.md). The headline finding is that
exact `Decimal` arithmetic costs 3.1× a float loop, not the order of magnitude
usually assumed — so the cost of this library is mostly Python and the
algorithm, not the exactness.

```console
$ python bench/benchmark.py
```

## Roadmap

What exists, what has been asked for, and what we have decided against is in
[ROADMAP.md](ROADMAP.md) — including the native Rust port two people have now
asked for, with an honest account of what it would and would not solve.

## Design notes

- **Money is `Decimal`.** Prices and sizes never touch binary floats, so
  `42000.10` stays `42000.10`.
- **Time is integer nanoseconds, UTC.** One comparable integer regardless of
  whether the source gave seconds, milliseconds, or a FIX timestamp string.
- **Symbols are canonicalized** to `BASE-QUOTE`, with venue aliases resolved
  (`XBT` → `BTC`) and quote currencies detected longest-match-first so
  `USDT` wins over `USD`.
- **Normalizers are pure functions** — one raw record in, one `MarketEvent`
  out — which keeps them trivial to unit-test and compose into any streaming
  or batch pipeline.

## Architecture

```
raw feed ──► normalizer ─────────────► MarketEvent ──► your pipeline
 (CSV /      (from_csv_row /            (unified,       (research,
  WS JSON /   from_ws_json /             immutable)      backtest,
  FIX)        from_fix)                                  execution)
                    │
                    ├── symbols.canonical_symbol()   BTCUSDT → BTC-USDT
                    ├── SymbolMap.instrument_at()    which instrument the ticker named then
                    ├── timeutil.*_to_ns()           any time → ns UTC
                    ├── adjust.adjust_events()       splits/divs/rolls
                    ├── TradingCalendar.is_open()    the holidays and half-days
                    ├── FxRates.convert()            a price in another currency
                    ├── grid_report()                is this a print or a derived number
                    ├── micro.infer_sides()          who crossed the spread
                    ├── book.OrderBook()             deltas → live book → quotes
                    ├── consolidate()                many venues → one best bid/offer
                    ├── evaluate()                   your fills vs the market
                    ├── align()                      N instruments → one time grid
                    ├── returns() / rolling_*()      features, trailing windows only
                    ├── purged_splits()              folds whose labels do not overlap
                    ├── Universe.members_at()        who was actually listed then
                    ├── RevisionSeries.as_of()       which version you had then
                    ├── sharpe_report()              and how much of it is the search
                    └── capacity()                   the size at which the edge runs out
```

## Examples

Three runnable scripts in [`examples/`](examples), standard library only:

| | |
| --- | --- |
| [`demo.py`](examples/demo.py) | one trade in CSV, WebSocket JSON and FIX collapsing to the same event |
| [`is_this_file_real.py`](examples/is_this_file_real.py) | prints against mids, VWAPs and an adjusted history, on the tick grid |
| [`nothing_looks_forward.py`](examples/nothing_looks_forward.py) | edit the tail of a series; a trailing z-score does not move, a full-sample one moves everywhere |

## Tests

```bash
pip install pytest
pytest -q
```

The suite includes a cross-venue equivalence test proving CSV, WebSocket and
FIX representations of one trade collapse to an identical event, and a
causality property applied across the feature layer: change the tail of an
input, and every output before the change must be byte-identical.

CI also runs `mypy`, and it is clean. The package ships a PEP 561 `py.typed`
marker, so your type checker will use its annotations rather than ignore them.

That marker was missing for most of this project's life while the packaging
metadata claimed otherwise. Removing the false claim and then earning it back
took one release each; what the second one mostly consisted of was writing
down invariants the code already enforced — a trade cannot exist without a
price, a window past the gap guard holds no `None`. See
[CONTRIBUTING.md](CONTRIBUTING.md) for where `cast` is and is not acceptable.

## License

MIT © HarvestGroup360 (AMII LTD). See [LICENSE](LICENSE).

---

Maintained by [HarvestGroup360](https://harvestgroup360.com) as part of our
open quantitative-infrastructure tooling.
