道michi inspect
data/customers.csv
600
rows
14
columns
14.5%
cells missing
0
duplicate rows
14
findings
purchased
target

Findings

severitywherewhat
high country only one distinct value (JP)
high country_copy only one distinct value (JP)
high notes every value is missing
high cabin 89.0% missing (534 of 600)
high outcome_code each of its 2 values maps to exactly one 'purchased' class
warn signup_date values parse as dates but are stored as text
warn country, country_copy identical values in country, country_copy
warn cabin 66 distinct values across 66 rows
warn fare skew 6.87 (right-tailed)
warn age, age_months correlation +1.000
warn salary 13.5% missing (81 of 600)
warn amount_text 100% of values parse as numbers
warn fare 37 values beyond 1.5×IQR (6.2%)
info record_id every value is distinct (looks like an identifier)

Columns

columnkindmissingunique distributionsummary
age numeric — 45 mean 42.702 · range 20–64
salary numeric 13.5% 511 mean 39,357.938 · range 19,798–56,542 · 1 outliers
region categorical — 4 top: north (239), south (188), east (117)
cabin categorical 89.0% 66 top: C13 (1), C37 (1), C38 (1)
notes empty 100.0% 0 —
country categorical — 1 top: JP (600)
country_copy categorical — 1 top: JP (600)
record_id text — 600 length 7–7
fare numeric — 467 mean 210.883 · range 5.29–10,000 · skew +6.87 · 37 outliers
amount_text text — 598 length 5–6
signup_date categorical — 11 top: 2024-01-15 (55), 2024-02-15 (55), 2024-03-15 (55)
age_months numeric — 45 mean 512.42 · range 240–768
outcome_code categorical — 2 top: class-0 (363), class-1 (237)
purchased numeric — 2 mean 0.395 · range 0–1

What these findings mean

Column has a single value · constant-column

Every row shares the same value, so the column cannot help distinguish rows and contributes nothing to a model.

Options

  • drop the column
  • check whether a filter upstream collapsed the variation
Caution: If the column varies in production but not in this extract, the extract may be unrepresentative.
Column is entirely missing · empty-column

Every value in this column is missing, so it carries no information about any row.

Options

  • drop the column
  • check whether an upstream export or join silently dropped the values
Caution: An all-missing column often signals a broken pipeline rather than an unimportant feature.
Mostly missing column · high-missing

More than half of this column's values are absent. Imputing a majority of a column invents most of its content, and the imputed value itself becomes the dominant signal.

Options

  • drop the column
  • keep it and impute (mean, median, mode, or a constant)
  • replace it with a boolean "was present" indicator
  • keep it only for the subset of rows where it is present
Caution: Missingness is often informative — rows lacking a value may differ systematically from rows that have one.
Possible target leakage · leakage-suspect

This column predicts the target almost perfectly. That usually means it was recorded at the same time as the outcome or after it, and will not be available when the model is actually used.

Options

  • check when the value becomes known relative to the prediction moment
  • drop the column if it postdates the outcome
  • keep it if it genuinely precedes the outcome and is available at inference
Caution: Leakage is the most common cause of a model that scores superbly offline and fails completely in production. michi can only raise the suspicion; confirming it needs knowledge of how the data was produced.
Dates stored as text · datetime-stored-as-text

The values parse as dates but are stored as strings, so sorting is lexicographic and no time arithmetic is possible.

Options

  • parse to a datetime type
  • derive components (year, month, day of week, hour)
  • compute durations relative to a reference event
Caution: With time-ordered data, split chronologically rather than randomly, or the model trains on the future to predict the past.
Identical columns · duplicate-columns

Two or more columns contain exactly the same values, so they carry the same information twice.

Options

  • keep one and drop the rest
  • check whether a join duplicated a field
Caution: Duplicated features distort feature-importance rankings by splitting the credit for a single signal.
Many distinct categories · high-cardinality

The column holds a large number of distinct categories relative to its row count. One-hot encoding it would create a very wide, very sparse matrix.

Options

  • group rare categories into an "other" bucket
  • use target or frequency encoding instead of one-hot
  • use a model that handles categoricals natively
  • drop the column
Caution: Target encoding computed over the whole dataset leaks the label; fit it inside the cross-validation loop.
Skewed distribution · high-skew

The column's values are concentrated on one side with a long tail on the other. Models that assume roughly symmetric inputs — linear and distance based ones — are most affected; tree ensembles are largely indifferent.

Options

  • apply a log or square-root transform (log1p handles zeros)
  • apply a quantile or power transform
  • bin the values into ranges
  • leave it unchanged, especially for tree-based models
Caution: Log transforms are undefined at zero and negative values; shift first or use log1p.
Nearly redundant columns · highly-correlated

The two columns move together almost perfectly. They contribute overlapping information, which destabilises linear-model coefficients and splits feature importance between them.

Options

  • keep one of the pair
  • combine them (a ratio, a difference, a principal component)
  • keep both, if the model is regularised or tree-based
Caution: Extremely high correlation with the target — as opposed to another feature — is usually leakage, not a great feature.
Missing values · missing

Some values are absent. Most estimators cannot train on missing values, so they must be filled or the affected rows removed.

Options

  • impute with the median (robust to skew) or the mean
  • impute with the most frequent value, for categorical columns
  • impute with an explicit sentinel such as "unknown"
  • drop the affected rows
  • drop the column
  • add a boolean indicator recording where the value was missing
Caution: Compute imputation values from the training fold only. Imputing from the whole dataset leaks test-set information into training.
Numbers stored as text · numeric-stored-as-text

The column's values parse as numbers but are stored as strings, usually because of thousands separators, currency symbols, or a stray non-numeric marker. Left as text, they are treated as unordered categories.

Options

  • convert to a numeric type, coercing unparseable entries to missing
  • strip the offending characters first, then convert
  • inspect the values that fail to parse before deciding
Caution: Coercing silently turns every unparseable value into a missing one; count them before and after.
Extreme values · outliers

Some values sit far outside the interquartile range (beyond 1.5×IQR from the quartiles). They may be recording errors, or they may be the real, important tail of the distribution.

Options

  • investigate the extreme rows before changing anything
  • clip values to a percentile range (winsorise)
  • remove the affected rows
  • use robust scaling instead of standardisation
  • leave them in place if they are genuine
Caution: Removing outliers because a metric improves is how real signal gets deleted. Decide on the basis of what the values mean.
Every value is distinct · identifier-like

The column has a different value in every row, which is the signature of an identifier (a key, an order number, a hash) rather than a feature.

Options

  • drop it from the feature set
  • keep it aside as a join or grouping key for splitting
  • derive features from it if it encodes structure (a prefix, a date)
Caution: Tree models can memorise identifiers and score well in cross-validation while learning nothing that generalises.