examples/orders_messy.csv

6 problems here will change your numbers, not just annoy you.

6 errors, 14 warnings, 3 notes across 40 rows and 10 columns.

order_id
customer
order_date
region
quantity
order_total
discount_code
zip
status
notes

Columns in file order. Colour marks the worst finding in each.

order_id

key-not-unique

Key column 'order_id' repeats 2 values. Joining on it will multiply rows.

A-1033, A-1036

order_date

conflicting-date-formats

'order_date' holds two date formats mixed together. 12 rows can only be day-first and 9 can only be month-first, so no single reading fits. The column holds two formats mixed together and cannot be repaired without knowing which rows came from where. Every row is individually valid, so nothing will error — the dates will simply be wrong.

07/05/2024, 07/01/2024, 04/04/2024
future-dates

'order_date' has 1 date in the future, latest 2027-08-14.

sentinel-dates

'order_date' uses placeholder dates in 1 row. They will be treated as real dates in any range filter.

1970-01-01

quantity

mixed-types

'quantity' is 2.5% non-numeric (1 row). A typed loader will fall back to text for the whole column, or drop these rows.

twelve

discount_code

missing-values

'discount_code' is 77.5% empty (31 of 40 rows).

notes

mojibake

'notes' contains 2 values with mangled characters — UTF-8 text that was decoded as Latin-1 upstream. Re-export rather than find-and-replace.

Café order, morning drop
whitespace

'notes' has 1 value with stray whitespace: padded with spaces (1). Grouping and joining will treat them as distinct.

'Customer asked for a reprint '

whole file

ragged-rows

1 row does not have 10 fields. Usually an unescaped delimiter or an unclosed quote.

line 21
duplicate-rows

1 redundant row across 1 group of identical rows. Any total taken from this file is inflated.

A-1036, Eastgate Retail, 14/09/2024, north
mixed-line-endings

Mixed line endings: 11 CRLF and 30 LF. Diffs and checksums will be noisy.

customer

label-variants

'customer' spells 2 values more than one way (case or spacing only). Any group-by will split them.

Bluefin Ltd / Bluefin ltd, Acme Corp / Acme Corp / acme corp
whitespace

'customer' has 3 values with stray whitespace: padded with spaces (3). Grouping and joining will treat them as distinct.

' Acme Corp'

region

disguised-null

'region' writes missing values as text in 6 rows, so they will load as strings rather than nulls.

N/A
label-variants

'region' spells 1 value more than one way (case or spacing only). Any group-by will split them.

NORTH / North / north
missing-values

'region' is 15.0% empty (6 of 40 rows).

whitespace

'region' has 5 values with stray whitespace: padded with spaces (5). Grouping and joining will treat them as distinct.

'NORTH '

order_total

number-as-text

'order_total' holds numbers written with currency symbols and thousands separators. It will load as text and silently break any arithmetic.

$465.46, $1,065.11, $2,443.74
numeric-sentinel

'order_total' repeats -999 in 2 rows. That is a placeholder for missing, and it will be summed and averaged as if it were real.

outliers

'order_total' has 1 value far from the median of 2,166.35. Worth a look before you average this column.

999,999
unexpected-negative

'order_total' reads like a quantity but has 2 negative values, lowest -999. Refunds, or a sign error?

zip

leading-zeros

'zip' is all digits and 14 values start with a zero. Reading it as a number destroys them — keep it text.

02134, 07094, 02139

status

constant-column

'status' is 'complete' in every row. It carries no information; check whether a filter upstream collapsed it.