道michi inspect
data/customers.csv
120
rows
13
columns
15.3%
cells missing
0
duplicate rows
15
findings
purchased
target

Findings

severitywherewhat
high country only one distinct value (JP)
high country_copy only one distinct value (JP)
high notes every value is missing
high cabin 87.5% missing (105 of 120)
high signup_date each of its 12 values maps to exactly one 'purchased' class
warn purchased smallest class 8.3% vs largest 91.7% across 2 classes
warn signup_date values parse as dates but are stored as text
warn country, country_copy identical values in country, country_copy
warn fare skew 6.16 (right-tailed)
warn age, age_months correlation +1.000
warn amount_text, salary correlation +1.000
warn salary 11.7% missing (14 of 120)
info purchased skew 3.05 (right-tailed)
info record_id every value is distinct (looks like an identifier)
info fare 3 values beyond 1.5×IQR (2.5%)

Columns

columnkindmissingunique distributionsummary
age numeric — 45 mean 41.625 · range 20–64
salary numeric 11.7% 106 mean 38,169.594 · range 30,137–46,303
cabin categorical 87.5% 15 top: C0 (1), C8 (1), C16 (1)
notes empty 100.0% 0 —
country categorical — 1 top: JP (120)
record_id text — 120 length 7–7
fare numeric — 11 mean 259.175 · range 5–10,000 · skew +6.16 · 3 outliers
amount_text numeric — 120 mean 1,059.5 · range 1,000–1,119
signup_date categorical — 12 top: 2024-01-15 (10), 2024-02-15 (10), 2024-03-15 (10)
age_months numeric — 45 mean 499.5 · range 240–768
country_copy categorical — 1 top: JP (120)
outcome_code categorical — 2 top: class-0 (60), class-1 (60)
purchased numeric — 2 mean 0.083 · range 0–1 · skew +3.05

What these findings mean

Column has a single value · constant-column

Every row shares the same value, so the column cannot help distinguish rows and contributes nothing to a model.

Options

  • drop the column
  • check whether a filter upstream collapsed the variation
Caution: If the column varies in production but not in this extract, the extract may be unrepresentative.
Column is entirely missing · empty-column

Every value in this column is missing, so it carries no information about any row.

Options

  • drop the column
  • check whether an upstream export or join silently dropped the values
Caution: An all-missing column often signals a broken pipeline rather than an unimportant feature.
Mostly missing column · high-missing

More than half of this column's values are absent. Imputing a majority of a column invents most of its content, and the imputed value itself becomes the dominant signal.

Options

  • drop the column
  • keep it and impute (mean, median, mode, or a constant)
  • replace it with a boolean "was present" indicator
  • keep it only for the subset of rows where it is present
Caution: Missingness is often informative — rows lacking a value may differ systematically from rows that have one.
Possible target leakage · leakage-suspect

This column predicts the target almost perfectly. That usually means it was recorded at the same time as the outcome or after it, and will not be available when the model is actually used.

Options

  • check when the value becomes known relative to the prediction moment
  • drop the column if it postdates the outcome
  • keep it if it genuinely precedes the outcome and is available at inference
Caution: Leakage is the most common cause of a model that scores superbly offline and fails completely in production. michi can only raise the suspicion; confirming it needs knowledge of how the data was produced.
Imbalanced classes · class-imbalance

The target's classes are unevenly represented. Accuracy becomes misleading: a model that always predicts the majority class can look strong while being useless for the minority class.

Options

  • keep the imbalance and judge with precision, recall, F1, or PR-AUC
  • use stratified splits so every fold keeps the class ratio
  • apply class weights in the estimator
  • resample (oversample the minority, undersample the majority)
  • adjust the decision threshold after training
Caution: Resample inside the cross-validation loop, never before splitting; oversampling first copies minority rows into both train and test.
Dates stored as text · datetime-stored-as-text

The values parse as dates but are stored as strings, so sorting is lexicographic and no time arithmetic is possible.

Options

  • parse to a datetime type
  • derive components (year, month, day of week, hour)
  • compute durations relative to a reference event
Caution: With time-ordered data, split chronologically rather than randomly, or the model trains on the future to predict the past.
Identical columns · duplicate-columns

Two or more columns contain exactly the same values, so they carry the same information twice.

Options

  • keep one and drop the rest
  • check whether a join duplicated a field
Caution: Duplicated features distort feature-importance rankings by splitting the credit for a single signal.
Skewed distribution · high-skew

The column's values are concentrated on one side with a long tail on the other. Models that assume roughly symmetric inputs — linear and distance based ones — are most affected; tree ensembles are largely indifferent.

Options

  • apply a log or square-root transform (log1p handles zeros)
  • apply a quantile or power transform
  • bin the values into ranges
  • leave it unchanged, especially for tree-based models
Caution: Log transforms are undefined at zero and negative values; shift first or use log1p.
Nearly redundant columns · highly-correlated

The two columns move together almost perfectly. They contribute overlapping information, which destabilises linear-model coefficients and splits feature importance between them.

Options

  • keep one of the pair
  • combine them (a ratio, a difference, a principal component)
  • keep both, if the model is regularised or tree-based
Caution: Extremely high correlation with the target — as opposed to another feature — is usually leakage, not a great feature.
Missing values · missing

Some values are absent. Most estimators cannot train on missing values, so they must be filled or the affected rows removed.

Options

  • impute with the median (robust to skew) or the mean
  • impute with the most frequent value, for categorical columns
  • impute with an explicit sentinel such as "unknown"
  • drop the affected rows
  • drop the column
  • add a boolean indicator recording where the value was missing
Caution: Compute imputation values from the training fold only. Imputing from the whole dataset leaks test-set information into training.
Every value is distinct · identifier-like

The column has a different value in every row, which is the signature of an identifier (a key, an order number, a hash) rather than a feature.

Options

  • drop it from the feature set
  • keep it aside as a join or grouping key for splitting
  • derive features from it if it encodes structure (a prefix, a date)
Caution: Tree models can memorise identifiers and score well in cross-validation while learning nothing that generalises.
Extreme values · outliers

Some values sit far outside the interquartile range (beyond 1.5×IQR from the quartiles). They may be recording errors, or they may be the real, important tail of the distribution.

Options

  • investigate the extreme rows before changing anything
  • clip values to a percentile range (winsorise)
  • remove the affected rows
  • use robust scaling instead of standardisation
  • leave them in place if they are genuine
Caution: Removing outliers because a metric improves is how real signal gets deleted. Decide on the basis of what the values mean.