Advanced Workflow with DataKit¶

This notebook demonstrates advanced usage of DataKit: custom imputation strategies, ordinal encoding, curve fitting, multiple comparisons correction, and full parameter control.

We assume you are already familiar with the basics from the beginner workflow.

In [ ]:

import datakit as dk
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# set random seed for reproducibility
dk.set_seed(42)

import datakit as dk import pandas as pd import numpy as np import matplotlib.pyplot as plt # set random seed for reproducibility dk.set_seed(42)

1. Custom imputation strategies¶

Instead of the default 'smart' strategy, we can provide a dictionary to control per‑column behaviour.

In [ ]:

# create a DataFrame with various missing patterns
np.random.seed(42)
n = 100
df = pd.DataFrame({
    'price': np.random.normal(100, 20, n),
    'rating': np.random.uniform(1, 5, n),
    'category': np.random.choice(['A', 'B', 'C', 'D'], n),
    'notes': [f'note_{i}' for i in range(n)],
    'date': pd.date_range('2024-01-01', periods=n)
})

# inject missing values
df.loc[10:20, 'price'] = np.nan
df.loc[5:15, 'rating'] = np.nan
df.loc[30:40, 'category'] = np.nan
df.loc[50:60, 'notes'] = np.nan
df.loc[70:75, 'date'] = pd.NaT

# custom strategy: drop 'notes', fill price with mean, rating with 3.0, category with 'unknown', date with bfill
custom_strategy = {
    'notes': 'drop',
    'price': 'mean',
    'rating': 3.0,
    'category': 'mode',
    'date': 'bfill'
}

df_clean = dk.auto_impute(df, strategy=custom_strategy, threshold=0.5, verbose=True)
print("\nRemaining nulls:", df_clean.isnull().sum().sum())

# create a DataFrame with various missing patterns np.random.seed(42) n = 100 df = pd.DataFrame({ 'price': np.random.normal(100, 20, n), 'rating': np.random.uniform(1, 5, n), 'category': np.random.choice(['A', 'B', 'C', 'D'], n), 'notes': [f'note_{i}' for i in range(n)], 'date': pd.date_range('2024-01-01', periods=n) }) # inject missing values df.loc[10:20, 'price'] = np.nan df.loc[5:15, 'rating'] = np.nan df.loc[30:40, 'category'] = np.nan df.loc[50:60, 'notes'] = np.nan df.loc[70:75, 'date'] = pd.NaT # custom strategy: drop 'notes', fill price with mean, rating with 3.0, category with 'unknown', date with bfill custom_strategy = { 'notes': 'drop', 'price': 'mean', 'rating': 3.0, 'category': 'mode', 'date': 'bfill' } df_clean = dk.auto_impute(df, strategy=custom_strategy, threshold=0.5, verbose=True) print("\nRemaining nulls:", df_clean.isnull().sum().sum())

2. Ordinal encoding¶

For ordered categories, we can provide an explicit mapping.

In [ ]:

df_ord = pd.DataFrame({
    'size': ['small', 'medium', 'large', 'medium', 'small', 'large'],
    'severity': ['low', 'medium', 'high', 'low', 'high', 'medium']
})

ordinal_map = {
    'size': ['small', 'medium', 'large'],
    'severity': ['low', 'medium', 'high']
}

df_encoded = dk.encode_categoricals(df_ord, method='ordinal', ordinal_map=ordinal_map, verbose=True)
df_encoded

df_ord = pd.DataFrame({ 'size': ['small', 'medium', 'large', 'medium', 'small', 'large'], 'severity': ['low', 'medium', 'high', 'low', 'high', 'medium'] }) ordinal_map = { 'size': ['small', 'medium', 'large'], 'severity': ['low', 'medium', 'high'] } df_encoded = dk.encode_categoricals(df_ord, method='ordinal', ordinal_map=ordinal_map, verbose=True) df_encoded

3. Curve fitting with custom model¶

Fit a custom function to data, plot fit and residuals, and get parameter estimates.

In [ ]:

# define a logistic growth model
def logistic(x, L, k, x0):
    return L / (1 + np.exp(-k * (x - x0)))

# generate synthetic data
x = np.linspace(0, 10, 50)
true_params = [100, 1.5, 5]
y_true = logistic(x, *true_params)
y = y_true + np.random.normal(0, 5, size=len(x))

# fit and plot
result = dk.fit_and_plot(logistic, x, y, p0=[80, 1, 4], plot_residuals=True, verbose=True, show=True)

print(f"Estimated parameters: {result['popt']}")
print(f"Standard errors: {result['perr']}")
print(f"R-squared: {result['r_squared']:.4f}")

# define a logistic growth model def logistic(x, L, k, x0): return L / (1 + np.exp(-k * (x - x0))) # generate synthetic data x = np.linspace(0, 10, 50) true_params = [100, 1.5, 5] y_true = logistic(x, *true_params) y = y_true + np.random.normal(0, 5, size=len(x)) # fit and plot result = dk.fit_and_plot(logistic, x, y, p0=[80, 1, 4], plot_residuals=True, verbose=True, show=True) print(f"Estimated parameters: {result['popt']}") print(f"Standard errors: {result['perr']}") print(f"R-squared: {result['r_squared']:.4f}")

4. Multiple comparisons correction¶

When performing many hypothesis tests, correct p‑values to control false discovery rate.

In [ ]:

# simulate p‑values from 20 tests
np.random.seed(42)
pvals = np.random.uniform(0, 1, 20)
# make a few significant
pvals[:3] = [0.001, 0.01, 0.03]

corrected = dk.correct_pvalues(pvals, method='fdr_bh', alpha=0.05, verbose=True)
print(f"\nRejected {corrected['n_rejected']} out of {corrected['n_total']} hypotheses")

# simulate p‑values from 20 tests np.random.seed(42) pvals = np.random.uniform(0, 1, 20) # make a few significant pvals[:3] = [0.001, 0.01, 0.03] corrected = dk.correct_pvalues(pvals, method='fdr_bh', alpha=0.05, verbose=True) print(f"\nRejected {corrected['n_rejected']} out of {corrected['n_total']} hypotheses")

5. Full control over plotting functions¶

Every plotting function exposes all underlying parameters.

In [ ]:

# correlation heatmap with custom figure size, colormap, and annotation format
numeric_df = pd.DataFrame({
    'a': np.random.randn(100),
    'b': np.random.randn(100) * 2,
    'c': np.random.randn(100) + 1,
    'd': np.random.randn(100) * 0.5
})
# add some correlation
numeric_df['e'] = numeric_df['a'] * 0.8 + np.random.randn(100) * 0.3

dk.plot_corr(numeric_df, method='spearman', figsize=(8, 6), cmap='viridis', annot=True, fmt='.3f', threshold=0.6, show=True)

# correlation heatmap with custom figure size, colormap, and annotation format numeric_df = pd.DataFrame({ 'a': np.random.randn(100), 'b': np.random.randn(100) * 2, 'c': np.random.randn(100) + 1, 'd': np.random.randn(100) * 0.5 }) # add some correlation numeric_df['e'] = numeric_df['a'] * 0.8 + np.random.randn(100) * 0.3 dk.plot_corr(numeric_df, method='spearman', figsize=(8, 6), cmap='viridis', annot=True, fmt='.3f', threshold=0.6, show=True)

In [ ]:

# distribution plots with KDE off and save to a directory
import tempfile
import os
with tempfile.TemporaryDirectory() as tmpdir:
    dk.plot_distributions(numeric_df, kde=False, include_boxplot=True, n_cols=2, save_dir=tmpdir, show=False)
    print("Saved plots to:", tmpdir)
    print("Files:", os.listdir(tmpdir))

# distribution plots with KDE off and save to a directory import tempfile import os with tempfile.TemporaryDirectory() as tmpdir: dk.plot_distributions(numeric_df, kde=False, include_boxplot=True, n_cols=2, save_dir=tmpdir, show=False) print("Saved plots to:", tmpdir) print("Files:", os.listdir(tmpdir))

6. Quickplot context manager with custom style and saving¶

The quickplot context manager replaces the usual fig, ax = plt.subplots() boilerplate and automatically saves and closes.

In [ ]:

x = np.linspace(0, 10, 100)
y = np.sin(x) + np.random.normal(0, 0.1, 100)

with dk.quickplot(figsize=(12, 5), title='Sine wave with noise', save='sine_plot.png', dpi=200, style='seaborn-v0_8-darkgrid') as ax:
    ax.scatter(x, y, alpha=0.5, label='data', color='steelblue')
    ax.plot(x, np.sin(x), 'r-', linewidth=2, label='true sine')
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.legend()
    ax.grid(True, alpha=0.3)

print("Plot saved as sine_plot.png")

x = np.linspace(0, 10, 100) y = np.sin(x) + np.random.normal(0, 0.1, 100) with dk.quickplot(figsize=(12, 5), title='Sine wave with noise', save='sine_plot.png', dpi=200, style='seaborn-v0_8-darkgrid') as ax: ax.scatter(x, y, alpha=0.5, label='data', color='steelblue') ax.plot(x, np.sin(x), 'r-', linewidth=2, label='true sine') ax.set_xlabel('x') ax.set_ylabel('y') ax.legend() ax.grid(True, alpha=0.3) print("Plot saved as sine_plot.png")

7. Using ColorRegistry for consistent colours across plots¶

When you have the same categories in multiple plots, ColorRegistry ensures they get the same colour.

In [ ]:

registry = dk.ColorRegistry(palette='Set2')

categories = ['Apple', 'Banana', 'Cherry']

# first plot
fig1, ax1 = plt.subplots()
values1 = [10, 20, 15]
ax1.bar(categories, values1, color=[registry.get(c) for c in categories])
ax1.set_title('Plot 1')
plt.show()

# second plot (same categories get same colours)
fig2, ax2 = plt.subplots()
values2 = [5, 25, 10]
ax2.bar(categories, values2, color=[registry.get(c) for c in categories])
ax2.set_title('Plot 2')
plt.show()

registry.reset()  # start over if needed

registry = dk.ColorRegistry(palette='Set2') categories = ['Apple', 'Banana', 'Cherry'] # first plot fig1, ax1 = plt.subplots() values1 = [10, 20, 15] ax1.bar(categories, values1, color=[registry.get(c) for c in categories]) ax1.set_title('Plot 1') plt.show() # second plot (same categories get same colours) fig2, ax2 = plt.subplots() values2 = [5, 25, 10] ax2.bar(categories, values2, color=[registry.get(c) for c in categories]) ax2.set_title('Plot 2') plt.show() registry.reset() # start over if needed

8. Using the configuration file¶

DataKit can read a .datakit.toml file to set default parameters for theme, imputation, etc.

In [ ]:

# create a temporary config file
config_content = """
[theme]
style = "darkgrid"
palette = "colorblind"
font_scale = 1.3
dpi = 150

[imputation]
threshold = 0.3
strategy = "smart"
"""

with open('.datakit.toml', 'w') as f:
    f.write(config_content)

# load config and use it
config = dk.load_config()
print("Loaded config:", config)

# set_theme will use the config values if from_config=True
dk.set_theme(from_config=True, config_path='.datakit.toml')

# later, auto_impute could also read from config if implemented
# but this example just shows the pattern

# create a temporary config file config_content = """ [theme] style = "darkgrid" palette = "colorblind" font_scale = 1.3 dpi = 150 [imputation] threshold = 0.3 strategy = "smart" """ with open('.datakit.toml', 'w') as f: f.write(config_content) # load config and use it config = dk.load_config() print("Loaded config:", config) # set_theme will use the config values if from_config=True dk.set_theme(from_config=True, config_path='.datakit.toml') # later, auto_impute could also read from config if implemented # but this example just shows the pattern

9. Full override – passing all parameters explicitly¶

Advanced users can override every single default parameter.

In [ ]:

# custom imputation with per‑column strategies and low threshold
df_test = pd.DataFrame({'x': [1, 2, np.nan, 4], 'y': ['a', np.nan, 'c', 'd']})
df_fixed = dk.auto_impute(df_test, strategy={'x': 'mean', 'y': 'drop'}, threshold=0.2, inplace=False, verbose=True)

# custom normalization with robust method and clipping
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
norm_arr = dk.normalize(arr, method='robust', axis=0, feature_range=(-1, 1), clip=True)
print("Normalized array (robust, clipped):\n", norm_arr)

# shape assertion with auto‑fix
wrong_shape = np.array([1, 2, 3])
corrected = dk.assert_shape(wrong_shape, expected=(3, 1), fix='expand')
print(f"Shape after fix: {corrected.shape}")

# custom imputation with per‑column strategies and low threshold df_test = pd.DataFrame({'x': [1, 2, np.nan, 4], 'y': ['a', np.nan, 'c', 'd']}) df_fixed = dk.auto_impute(df_test, strategy={'x': 'mean', 'y': 'drop'}, threshold=0.2, inplace=False, verbose=True) # custom normalization with robust method and clipping arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) norm_arr = dk.normalize(arr, method='robust', axis=0, feature_range=(-1, 1), clip=True) print("Normalized array (robust, clipped):\n", norm_arr) # shape assertion with auto‑fix wrong_shape = np.array([1, 2, 3]) corrected = dk.assert_shape(wrong_shape, expected=(3, 1), fix='expand') print(f"Shape after fix: {corrected.shape}")

10. Summary¶

In this advanced notebook we covered:

• Custom imputation strategies per column
• Ordinal encoding with explicit mappings
• Nonlinear curve fitting with residuals and R²
• Multiple comparisons correction (FDR, Bonferroni)
• Full parameter control over plots
• Quickplot context manager for streamlined plotting
• ColorRegistry for consistent colours
• Configuration file usage
• Complete override of all defaults

DataKit scales from beginners to experts – the same functions work at every level.