# ML2 Study Material

# ML2_ESA_Feb25_Solved_Notebook.ipynb

# Machine Learning - II ESA February 2025: Solved Previous Question Paper

**Program:** M.Tech Data Science and Artificial Intelligence  
**Course:** UE20CS931 - Machine Learning - II  
**Exam:** February 2025 End Semester Assessment  
**Maximum Marks:** 100

This notebook solves the full question paper in an exam-focused style. It includes the original question before each answer, clear theory notes, formulas, Python code, plots, and inferences based on the actual dataset output.

## Syllabus alignment used while writing answers

The course outline emphasizes supervised classification, logistic regression and odds/probability, evaluation metrics, imbalanced data handling, Naive Bayes, decision trees with entropy/Gini, model evaluation, random forests, feature importance, boosting, stacking, and voting. The solutions below therefore focus on definitions, assumptions, step-wise algorithms, formulas, practical interpretation, and model-comparison reasoning.

```python
# Confirm that the notebook is running in the requested Python environment.
# This output is useful during exam preparation because it proves that all
# packages and code were executed using the intended interpreter.
import sys
print("Python executable:", sys.executable)
print("Python version:", sys.version.split()[0])
```

## Section A - Theory Questions (20 marks)

### Question 1(a) - 4 marks

**Full question:**  
Define precision, recall, and F1-score in the context of a classification problem. Explain their significance and how they are used to evaluate a model's performance.

### Answer

In a classification problem, the model predicts class labels. For binary classification, the prediction outcomes are summarized using a confusion matrix:

| Term | Meaning |
|---|---|
| True Positive (TP) | Actual positive and predicted positive |
| True Negative (TN) | Actual negative and predicted negative |
| False Positive (FP) | Actual negative but predicted positive |
| False Negative (FN) | Actual positive but predicted negative |

#### Precision

$$
\text{Precision} = \frac{TP}{TP + FP}
$$

Precision answers: **Out of all records predicted as positive, how many were actually positive?**

High precision is important when false positives are costly. Example: if a company predicts that an employee will leave, a false alarm may lead to unnecessary retention cost.

#### Recall

$$
\text{Recall} = \frac{TP}{TP + FN}
$$

Recall answers: **Out of all actual positive records, how many did the model correctly find?**

High recall is important when missing a positive case is costly. Example: if the business wants to identify employees likely to leave, low recall means many risky employees are missed.

#### F1-score

$$
F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}
$$

F1-score is the harmonic mean of precision and recall. It is useful when the dataset is imbalanced or when both false positives and false negatives matter.

#### Significance in model evaluation

- Precision focuses on the correctness of positive predictions.
- Recall focuses on coverage of the actual positive class.
- F1-score balances precision and recall in one number.
- Accuracy alone can be misleading on imbalanced datasets.
- Threshold adjustment changes precision and recall, so F1-score helps choose a balanced threshold.

**Exam point:** For classification problems with class imbalance, always report precision, recall, F1-score, and confusion matrix instead of relying only on accuracy.

### Question 1(b) - 4 marks

**Full question:**  
Explain briefly about Gaussian naive Bayes classification algorithm. State its assumptions.

### Answer

Gaussian Naive Bayes is a probabilistic classification algorithm based on Bayes theorem. It is mainly used when the input features are continuous and are assumed to follow a Gaussian or normal distribution within each class.

#### Bayes theorem

$$
P(C_k \mid X) = \frac{P(X \mid C_k)P(C_k)}{P(X)}
$$

Where:

- $C_k$ is a class label.
- $X = (x_1, x_2, ..., x_n)$ is the feature vector.
- $P(C_k)$ is the prior probability of class $C_k$.
- $P(X \mid C_k)$ is the likelihood of the data given the class.
- $P(C_k \mid X)$ is the posterior probability.

Since $P(X)$ is the same for all classes, the predicted class is:

$$
\hat{C} = \arg\max_{C_k} P(C_k)\prod_{i=1}^{n} P(x_i \mid C_k)
$$

#### Gaussian likelihood

For a continuous feature $x_i$, Gaussian Naive Bayes assumes:

$$
P(x_i \mid C_k) =
\frac{1}{\sqrt{2\pi\sigma_k^2}}
\exp\left(-\frac{(x_i-\mu_k)^2}{2\sigma_k^2}\right)
$$

Here, $\mu_k$ and $\sigma_k^2$ are the mean and variance of that feature for class $C_k$.

#### Algorithm steps

1. Separate the training data by class.
2. For each class, compute prior probability.
3. For each feature and class, compute mean and variance.
4. For a new observation, compute posterior probability for every class.
5. Predict the class with maximum posterior probability.

#### Assumptions

- Features are conditionally independent given the class.
- Continuous features follow a Gaussian distribution within each class.
- Each feature contributes independently to the final class probability.
- Training and test data come from the same distribution.

#### Advantages

- Simple and fast.
- Works well with small datasets.
- Handles high-dimensional data efficiently.
- Performs surprisingly well even when independence is not perfectly true.

#### Limitations

- The independence assumption is often unrealistic.
- It may perform poorly when features are highly correlated.
- Gaussian assumption may be wrong for skewed continuous variables.

**Exam point:** The word "naive" refers to the conditional independence assumption, not to the use of Bayes theorem itself.

### Question 1(c) - 4 marks

**Full question:**  
Explain briefly about stacking and voting classifier.

### Answer

Stacking and voting are ensemble learning methods. Ensemble methods combine multiple models to improve prediction performance, reduce variance, and increase robustness.

#### Voting classifier

A voting classifier combines predictions from multiple base models and gives the final class based on voting.

There are two main types:

1. **Hard voting:** Each model predicts a class. The final class is the majority vote.

   Example: If Logistic Regression predicts 1, Decision Tree predicts 0, and Random Forest predicts 1, the final prediction is 1.

2. **Soft voting:** Each model predicts class probabilities. The probabilities are averaged, and the class with highest average probability is selected.

   Soft voting usually performs better when models produce well-calibrated probabilities.

#### Stacking classifier

Stacking uses multiple base learners and a second-level model called a meta-learner.

Steps:

1. Train different base models, such as Logistic Regression, Decision Tree, and Random Forest.
2. Use their predictions or probabilities as new features.
3. Train a meta-model, such as Logistic Regression, on these new features.
4. The meta-model learns how to combine the base models.

#### Difference between voting and stacking

| Aspect | Voting | Stacking |
|---|---|---|
| Combination method | Majority vote or average probability | Learns combination using meta-model |
| Complexity | Simple | More complex |
| Risk of overfitting | Lower | Higher if not cross-validated properly |
| Performance | Good when base models are strong | Can be better if meta-model learns useful patterns |

#### Advantages

- Combines strengths of different algorithms.
- Reduces dependency on one model.
- Often improves generalization.

#### Limitations

- More computationally expensive.
- Less interpretable than a single model.
- Stacking must be implemented carefully to avoid data leakage.

**Exam point:** Voting combines decisions directly, while stacking trains another model to learn how to combine decisions.

### Question 1(d) - 5 marks

**Full question:**  
Explain the concept of bagging in machine learning. How does it improve the performance of a classification model? Explain the training and testing process of random forest algorithm.

### Answer

Bagging stands for **Bootstrap Aggregating**. It is an ensemble technique that trains multiple models on different bootstrap samples of the training data and combines their predictions.

#### Concept of bagging

Given a training dataset, bagging creates several new datasets by sampling with replacement. Each sample is called a bootstrap sample. A separate base model is trained on each bootstrap sample.

For classification, final prediction is usually made by majority voting.

#### How bagging improves classification performance

- It reduces variance by averaging many unstable models.
- It reduces overfitting compared to a single decision tree.
- It improves generalization on unseen data.
- It makes predictions more stable because each model sees a slightly different dataset.
- It can estimate out-of-bag error using samples not selected in each bootstrap sample.

#### Random Forest

Random Forest is a bagging-based ensemble of decision trees. It adds one more layer of randomness: at each split, only a random subset of features is considered.

#### Training process of Random Forest

1. Draw many bootstrap samples from the training data.
2. Train one decision tree on each bootstrap sample.
3. At every split in each tree, randomly select a subset of features.
4. Choose the best split from that subset using Gini index or entropy.
5. Grow many trees, usually without pruning or with controlled depth.
6. Store all trees as the trained forest.

#### Testing process of Random Forest

1. Pass a test observation through every tree.
2. Each tree gives a class prediction.
3. The final class is decided by majority voting.
4. Class probabilities can be estimated as the fraction of trees voting for each class.

#### Important hyperparameters

- `n_estimators`: number of trees.
- `max_depth`: maximum depth of each tree.
- `max_features`: number of features considered at each split.
- `min_samples_leaf`: minimum samples required in a leaf node.
- `class_weight`: useful for imbalanced classification.

#### Advantages

- High predictive performance.
- Handles nonlinear relationships.
- Handles mixed numerical and categorical features after encoding.
- Provides feature importance.

#### Limitations

- Less interpretable than a single decision tree.
- Can be slower for large datasets.
- Feature importance may be biased toward high-cardinality features.

**Exam point:** Bagging reduces variance; Random Forest improves bagging by decorrelating trees through random feature selection.

### Question 1(e) - 3 marks

**Full question:**  
What is cross-validation in machine learning?

### Answer

Cross-validation is a model evaluation technique used to estimate how well a model will generalize to unseen data.

#### K-fold cross-validation

In K-fold cross-validation:

1. The dataset is divided into $K$ equal parts called folds.
2. The model is trained on $K-1$ folds.
3. The remaining fold is used for validation.
4. This process is repeated $K$ times.
5. The final score is the average of the $K$ validation scores.

#### Why cross-validation is useful

- Gives a more reliable estimate than a single train-test split.
- Reduces dependency on one random split.
- Helps tune hyperparameters.
- Helps detect overfitting and underfitting.

#### Stratified cross-validation

For classification, stratified K-fold is preferred because it preserves the class distribution in every fold. This is especially important for imbalanced datasets.

#### Limitations

- More computationally expensive than a single split.
- Must be done carefully to avoid data leakage, especially during preprocessing.

**Exam point:** Preprocessing steps such as imputation, scaling, and encoding should be fitted only on the training fold, not on the full dataset.

## Section B - Dataset Understanding and Preprocessing (40 marks)

The dataset is stored as `dataset1.csv` in the same folder as this notebook. The target variable is `LeaveOrNot`, where:

- `0` means the employee did not leave.
- `1` means the employee left.

### Question 2 - Dataset description

**Full question:**  
This dataset contains information about employees in a company, including their educational backgrounds, work history, demographics, and employment-related factors. It has been anonymized to protect privacy while still providing valuable insights into the workforce. (Refer jupyter notebook for detail of the dataset)

```python
# Import the required libraries.
# pandas and numpy are used for data handling.
# matplotlib and seaborn are used for plots.
# sklearn is used for preprocessing, model building, and evaluation.

import warnings
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.tree import DecisionTreeClassifier

warnings.filterwarnings("ignore")

# Keep plots readable and consistent throughout the notebook.
sns.set_theme(style="whitegrid", context="notebook")
plt.rcParams["figure.figsize"] = (8, 5)
plt.rcParams["axes.titlesize"] = 12

# Fixed random state makes the results reproducible.
RANDOM_STATE = 42
TARGET = "LeaveOrNot"
```

### Question 2(a) - 8 marks

**Full question:**  
Read the dataset and print the following:

- Describe the structure of a given dataset in terms of its shape, the number of numerical and categorical variables, and provide descriptive statistics for both types of variables. Additionally, mention any key observations based on the descriptive statistics. (5 marks)
- Summarize the categorical variables in a dataset by identifying the number of unique categories and the proportion of observations in each category. Highlight any noticeable patterns or anomalies. (3 marks)

### Approach

To score full marks, we should show:

1. Dataset shape.
2. First few rows.
3. Data types.
4. Numerical and categorical column separation.
5. Descriptive statistics for numerical columns.
6. Category counts and proportions for categorical columns.
7. Clear observations from the output.

```python
# Read the dataset from the same folder as this notebook.
data_path = Path("dataset1.csv")
df = pd.read_csv(data_path)

# Display basic structure.
print("Dataset shape:", df.shape)
print("\nColumn names:")
print(df.columns.tolist())

# Show the first five rows to understand actual values.
display(df.head())

# Show data types and non-null counts.
print("\nData information:")
df.info()
```

```python
# Separate numerical and categorical columns.
# The target is numeric, but for EDA we keep it visible because it is the class label.
numeric_cols = df.select_dtypes(include=np.number).columns.tolist()
categorical_cols = df.select_dtypes(exclude=np.number).columns.tolist()

print("Number of numerical columns:", len(numeric_cols))
print("Numerical columns:", numeric_cols)
print("\nNumber of categorical columns:", len(categorical_cols))
print("Categorical columns:", categorical_cols)
```

```python
# Descriptive statistics for numerical variables.
# Transposing makes the table easier to read in notebooks.
numeric_summary = df[numeric_cols].describe().T
display(numeric_summary)
```

```python
# Descriptive statistics for categorical variables.
# This gives count, number of unique values, most frequent value, and its frequency.
categorical_summary = df[categorical_cols].describe().T
display(categorical_summary)

# For each categorical column, show counts and proportions.
for col in categorical_cols:
    print(f"\nCategorical summary for {col}:")
    temp = (
        df[col]
        .value_counts(dropna=False)
        .rename_axis(col)
        .reset_index(name="count")
    )
    temp["proportion"] = (temp["count"] / len(df)).round(4)
    display(temp)
```

### Inference for Question 2(a)

Based on the actual output:

- The dataset has **4653 rows and 11 columns**.
- There are **7 numerical columns**: `JoiningYear`, `PaymentTier`, `Age`, `ExperienceInCurrentDomain`, `LeaveOrNot`, `WorkHoursPerWeek`, and `AnnualBonus`.
- There are **4 categorical columns**: `Education`, `City`, `Gender`, and `EverBenched`.
- `Age` has missing values because its count is lower than the total row count.
- `EverBenched` also has a small number of missing values.
- `Education` is dominated by `Bachelors` employees.
- `City` has three categories, with `Bangalore` having the highest representation.
- `Gender` is not perfectly balanced; males are the larger group.
- The target variable `LeaveOrNot` is binary, so this is a binary classification problem.

### Question 2(b) - 8 marks

**Full question:**  
Examine missing values, outliers by plotting. Examine is the Target variable evenly balanced.

### Approach

We will:

1. Count missing values column-wise.
2. Visualize missing values.
3. Detect outliers using boxplots and the IQR rule.
4. Check target class counts and proportions.
5. Interpret whether the class distribution is balanced.

```python
# Count missing values in every column.
missing_count = df.isna().sum()
missing_percent = (missing_count / len(df) * 100).round(2)
missing_table = pd.DataFrame({
    "missing_count": missing_count,
    "missing_percent": missing_percent,
}).sort_values("missing_count", ascending=False)

display(missing_table)
```

```python
# Plot missing value counts.
plt.figure(figsize=(9, 4))
missing_count.sort_values(ascending=False).plot(kind="bar", color="#4C78A8")
plt.title("Missing Values by Column")
plt.ylabel("Number of missing values")
plt.xlabel("Column")
plt.xticks(rotation=45, ha="right")
plt.tight_layout()
plt.show()
```

```python
# Outlier detection using the IQR rule.
# We calculate outliers for numeric columns except the target.
outlier_rows = []

for col in [c for c in numeric_cols if c != TARGET]:
    q1 = df[col].quantile(0.25)
    q3 = df[col].quantile(0.75)
    iqr = q3 - q1
    lower = q1 - 1.5 * iqr
    upper = q3 + 1.5 * iqr
    outlier_mask = (df[col] < lower) | (df[col] > upper)
    outlier_rows.append({
        "feature": col,
        "lower_bound": round(lower, 3),
        "upper_bound": round(upper, 3),
        "outlier_count": int(outlier_mask.sum()),
        "outlier_percent": round(outlier_mask.mean() * 100, 2),
    })

outlier_table = pd.DataFrame(outlier_rows)
display(outlier_table)
```

```python
# Boxplots help visually inspect the spread and possible outliers.
# PaymentTier is ordinal and has only three levels, so an IQR flag on it
# should be interpreted carefully as a discrete-category artifact.
plot_numeric = [c for c in numeric_cols if c != TARGET]

fig, axes = plt.subplots(2, 3, figsize=(14, 8))
axes = axes.ravel()

for ax, col in zip(axes, plot_numeric):
    sns.boxplot(y=df[col], ax=ax, color="#72B7B2")
    ax.set_title(f"Boxplot of {col}")
    ax.set_xlabel("")

plt.tight_layout()
plt.show()
```

```python
# Check whether the target variable is balanced.
target_counts = df[TARGET].value_counts().sort_index()
target_proportions = df[TARGET].value_counts(normalize=True).sort_index()

target_balance = pd.DataFrame({
    "class_label": target_counts.index,
    "count": target_counts.values,
    "proportion": target_proportions.values.round(4),
})
display(target_balance)

plt.figure(figsize=(6, 4))
sns.countplot(data=df, x=TARGET, palette=["#59A14F", "#E15759"])
plt.title("Target Class Distribution")
plt.xlabel("LeaveOrNot")
plt.ylabel("Count")
plt.tight_layout()
plt.show()
```

### Inference for Question 2(b)

Based on the actual output:

- `Age` has **233 missing values**, which is about 5% of the dataset.
- `EverBenched` has **47 missing values**, which is about 1% of the dataset.
- The continuous numeric variables `Age`, `WorkHoursPerWeek`, and `AnnualBonus` do not show severe IQR-based outliers.
- `PaymentTier` is flagged by the IQR method because most employees are in tier 3. Since it is an ordinal categorical-like variable with valid values 1, 2, and 3, these should not be treated as numerical outliers.
- The target variable is **not evenly balanced**. Around **65.61%** of employees did not leave and **34.39%** left.
- This is a moderate class imbalance, so F1-score, precision, recall, ROC-AUC, and class-weight handling are more meaningful than accuracy alone.

### Question 2(c) - 6 marks

**Full question:**  
Perform appropriate encoding on the categorical attributes.

### Approach

The categorical attributes are `Education`, `City`, `Gender`, and `EverBenched`.

For machine learning models:

- Logistic Regression needs numeric input and benefits from scaling numerical features.
- Decision Tree and Random Forest also need numeric input, but do not require scaling.
- One-hot encoding is appropriate here because the categorical variables have only a few categories.

```python
# Before encoding, fill missing categorical values for demonstration.
# The complete cleaning pipeline is done in Question 2(d).
df_encoding_demo = df.copy()
df_encoding_demo["EverBenched"] = df_encoding_demo["EverBenched"].fillna("Missing")

# One-hot encode categorical columns.
# drop_first=False keeps all categories, which is safer for tree models and easy to interpret.
encoded_demo = pd.get_dummies(
    df_encoding_demo,
    columns=categorical_cols,
    drop_first=False,
    dtype=int,
)

print("Original shape:", df.shape)
print("Encoded shape:", encoded_demo.shape)
print("\nFirst 15 encoded columns:")
print(encoded_demo.columns[:15].tolist())

display(encoded_demo.head())
```

### Inference for Question 2(c)

One-hot encoding converts each category into a binary indicator column. This avoids imposing a false numerical order on categories such as `City` or `Gender`.

For example:

- `City_Bangalore`, `City_New Delhi`, and `City_Pune` represent city membership.
- `Gender_Female` and `Gender_Male` represent gender categories.
- `EverBenched_No`, `EverBenched_Yes`, and `EverBenched_Missing` represent benching status including missing values for demonstration.

In the final modeling pipeline, the encoder is fitted only on the training data to avoid data leakage.

### Question 2(d) - 8 marks

**Full question:**  
Perform necessary actions to "fix" defects in the data:

- Impute the missing values using central values (Mean, median, mode)
- Detect the outliers and decide if any treatment is required.
- Detect the unnecessary features and remove.

### Approach

Data defects are handled as follows:

1. Missing numeric value in `Age`: impute using median because median is robust to skew and outliers.
2. Missing categorical value in `EverBenched`: impute using mode because it is categorical.
3. Outliers: use IQR detection. Continuous variables do not show severe outliers, but the code includes safe capping for continuous columns if needed.
4. Unnecessary features: check duplicate rows, constant columns, and identifier-like columns.

```python
# Create a cleaned copy so the original dataframe remains unchanged.
df_clean = df.copy()

# 1. Remove exact duplicate rows if any exist.
duplicate_count = df_clean.duplicated().sum()
df_clean = df_clean.drop_duplicates()

# 2. Impute numeric missing values.
# Age is numeric; median is robust and suitable for this dataset.
age_median = df_clean["Age"].median()
df_clean["Age"] = df_clean["Age"].fillna(age_median)

# 3. Impute categorical missing values.
# EverBenched is categorical; mode is the most frequent category.
ever_benched_mode = df_clean["EverBenched"].mode()[0]
df_clean["EverBenched"] = df_clean["EverBenched"].fillna(ever_benched_mode)

print("Duplicate rows removed:", duplicate_count)
print("Age median used for imputation:", age_median)
print("EverBenched mode used for imputation:", ever_benched_mode)
print("\nMissing values after imputation:")
display(df_clean.isna().sum().to_frame("missing_after_cleaning"))
```

```python
# Outlier treatment decision.
# Only continuous columns are capped. Discrete ordinal variables such as PaymentTier
# are not capped because values 1, 2, and 3 are valid categories.
continuous_cols = ["Age", "WorkHoursPerWeek", "AnnualBonus"]
cap_report = []

for col in continuous_cols:
    q1 = df_clean[col].quantile(0.25)
    q3 = df_clean[col].quantile(0.75)
    iqr = q3 - q1
    lower = q1 - 1.5 * iqr
    upper = q3 + 1.5 * iqr
    before_outliers = ((df_clean[col] < lower) | (df_clean[col] > upper)).sum()

    # Clip values only if they fall outside the calculated bounds.
    df_clean[col] = df_clean[col].clip(lower=lower, upper=upper)

    after_outliers = ((df_clean[col] < lower) | (df_clean[col] > upper)).sum()
    cap_report.append({
        "feature": col,
        "lower_bound": round(lower, 3),
        "upper_bound": round(upper, 3),
        "outliers_before_capping": int(before_outliers),
        "outliers_after_capping": int(after_outliers),
    })

display(pd.DataFrame(cap_report))
```

```python
# Detect unnecessary features.
# We check for constant columns and identifier-like columns.
feature_cols = [c for c in df_clean.columns if c != TARGET]

constant_cols = [c for c in feature_cols if df_clean[c].nunique(dropna=False) <= 1]

# Identifier-like columns usually have a unique value for almost every row.
# Such columns often do not generalize well. Here we use a 95% uniqueness threshold.
identifier_like_cols = [
    c for c in feature_cols
    if df_clean[c].nunique(dropna=False) / len(df_clean) > 0.95
]

print("Constant columns:", constant_cols)
print("Identifier-like columns:", identifier_like_cols)

# In this dataset, high-uniqueness numeric columns are continuous measurements,
# not employee IDs. Therefore, no feature is removed only due to uniqueness.
features_to_remove = constant_cols
df_clean = df_clean.drop(columns=features_to_remove)

print("Features removed:", features_to_remove)
print("Cleaned data shape:", df_clean.shape)
```

### Inference for Question 2(d)

Based on the actual output:

- No duplicate rows are present.
- `Age` missing values are imputed using the median value **28.0**.
- `EverBenched` missing values are imputed using the mode value **No**.
- Continuous variables do not require aggressive outlier treatment; the capping step is included as a safe exam-standard preprocessing step.
- `PaymentTier` should not be capped even though the IQR method flags values, because it is an ordinal variable with valid categories.
- No constant columns or true identifier columns are found. Therefore, no meaningful feature is removed.

### Question 2(e) - 6 marks

**Full question:**  
Examine the correlation and summarize the relationship between variables. Use appropriate plots to justify the same.

### Approach

Correlation requires numerical variables. For categorical variables, we first create one-hot encoded columns, then compute correlation with the target. We will show:

1. Correlation heatmap for numeric variables.
2. Top encoded-feature correlations with `LeaveOrNot`.
3. Interpretation of relationship direction and strength.

```python
# Correlation among numeric variables in the cleaned dataset.
numeric_corr = df_clean.select_dtypes(include=np.number).corr()

plt.figure(figsize=(9, 6))
sns.heatmap(numeric_corr, annot=True, cmap="RdBu_r", center=0, fmt=".2f")
plt.title("Correlation Heatmap for Numeric Variables")
plt.tight_layout()
plt.show()

display(numeric_corr[TARGET].sort_values(key=lambda s: s.abs(), ascending=False).to_frame("correlation_with_target"))
```

```python
# Encode categorical variables to examine their relationship with the target.
df_corr_encoded = pd.get_dummies(df_clean, columns=categorical_cols, drop_first=True, dtype=int)

target_corr = (
    df_corr_encoded
    .corr(numeric_only=True)[TARGET]
    .drop(TARGET)
    .sort_values(key=lambda s: s.abs(), ascending=False)
)

display(target_corr.head(12).to_frame("correlation_with_LeaveOrNot"))

plt.figure(figsize=(8, 5))
target_corr.head(12).sort_values().plot(kind="barh", color="#F28E2B")
plt.title("Top Feature Correlations with LeaveOrNot")
plt.xlabel("Correlation")
plt.tight_layout()
plt.show()
```

### Inference for Question 2(e)

Based on the actual output:

- `Gender_Male` has a negative correlation with leaving, which means the encoded male category is associated with a lower probability of `LeaveOrNot = 1`.
- `City_Pune` has a positive relationship with leaving.
- `PaymentTier` has a negative relationship with leaving; higher payment tier is associated with lower leave tendency.
- `JoiningYear` has a positive relationship with leaving, suggesting that joining cohort may matter.
- `Education_Masters` also shows positive association with leaving.
- `WorkHoursPerWeek` and `AnnualBonus` show very weak linear correlations with the target in this dataset.

Correlation does not prove causation. It only indicates the direction and strength of linear association.

### Question 2(f) - 4 marks

**Full question:**  
Split dataset into train and test (70:30).

### Approach

We will use a stratified train-test split so that the class distribution of `LeaveOrNot` is preserved in both train and test sets.

```python
# Separate input features and target.
X = df_clean.drop(columns=[TARGET])
y = df_clean[TARGET]

# Stratified split preserves the class ratio in train and test.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.30,
    random_state=RANDOM_STATE,
    stratify=y,
)

print("Training feature shape:", X_train.shape)
print("Testing feature shape:", X_test.shape)
print("Training target shape:", y_train.shape)
print("Testing target shape:", y_test.shape)

split_balance = pd.DataFrame({
    "train_proportion": y_train.value_counts(normalize=True).sort_index(),
    "test_proportion": y_test.value_counts(normalize=True).sort_index(),
})
display(split_balance)
```

### Inference for Question 2(f)

The data is split into 70% training and 30% testing. Stratification keeps the class distribution nearly identical in both sets, which is important because the target class is moderately imbalanced.

## Section C - Model Building, Threshold Analysis, and Business Summary (40 marks)

### Common preprocessing and evaluation functions

The following helper code creates reusable preprocessing pipelines and evaluation functions.

Important exam point: imputation, scaling, and encoding are placed inside `Pipeline` and `ColumnTransformer`. This prevents data leakage because preprocessing is learned from the training data only.

```python
# Identify model input columns after cleaning.
model_numeric_cols = X.select_dtypes(include=np.number).columns.tolist()
model_categorical_cols = X.select_dtypes(exclude=np.number).columns.tolist()

print("Numeric model columns:", model_numeric_cols)
print("Categorical model columns:", model_categorical_cols)


def make_one_hot_encoder():
    # Create a OneHotEncoder that works across sklearn versions.
    try:
        return OneHotEncoder(handle_unknown="ignore", sparse_output=False)
    except TypeError:
        return OneHotEncoder(handle_unknown="ignore", sparse=False)


# Logistic Regression benefits from scaling numerical columns.
logistic_preprocess = ColumnTransformer(
    transformers=[
        ("num", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]), model_numeric_cols),
        ("cat", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("onehot", make_one_hot_encoder()),
        ]), model_categorical_cols),
    ]
)

# Tree-based models do not need scaling.
tree_preprocess = ColumnTransformer(
    transformers=[
        ("num", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="median")),
        ]), model_numeric_cols),
        ("cat", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("onehot", make_one_hot_encoder()),
        ]), model_categorical_cols),
    ]
)


def evaluate_predictions(y_true, y_pred, y_prob=None):
    # Return common classification metrics as a dictionary.
    metrics = {
        "Accuracy": accuracy_score(y_true, y_pred),
        "Precision": precision_score(y_true, y_pred, zero_division=0),
        "Recall": recall_score(y_true, y_pred, zero_division=0),
        "F1-score": f1_score(y_true, y_pred, zero_division=0),
    }
    if y_prob is not None:
        metrics["ROC-AUC"] = roc_auc_score(y_true, y_prob)
    return metrics


def evaluate_model(model, X_test, y_test, threshold=None):
    # Evaluate a fitted classifier, optionally using a custom threshold.
    if threshold is not None:
        probabilities = model.predict_proba(X_test)[:, 1]
        predictions = (probabilities >= threshold).astype(int)
    else:
        predictions = model.predict(X_test)
        probabilities = model.predict_proba(X_test)[:, 1]
    return evaluate_predictions(y_test, predictions, probabilities)
```

### Question 3(a) - 16 marks

**Full question:**  
Fit a Logistic Regression model and compare the results with respect to different threshold values. Please write your key observations based on F1 score for your model.

### Approach

Logistic Regression outputs probabilities. A threshold converts probabilities into class labels.

- Default threshold is usually 0.50.
- Lower threshold increases recall but may reduce precision.
- Higher threshold increases precision but may reduce recall.
- F1-score is used to find the best balance between precision and recall.

```python
# Build and train Logistic Regression.
# class_weight="balanced" gives more importance to the minority class.
logistic_model = Pipeline(steps=[
    ("preprocess", logistic_preprocess),
    ("model", LogisticRegression(
        max_iter=2000,
        class_weight="balanced",
        random_state=RANDOM_STATE,
    )),
])

logistic_model.fit(X_train, y_train)

# Predict probabilities for the positive class, LeaveOrNot = 1.
logistic_prob = logistic_model.predict_proba(X_test)[:, 1]

# Evaluate multiple thresholds from 0.10 to 0.90.
threshold_results = []
for threshold in np.arange(0.10, 0.91, 0.05):
    logistic_pred = (logistic_prob >= threshold).astype(int)
    row = evaluate_predictions(y_test, logistic_pred, logistic_prob)
    row["Threshold"] = round(float(threshold), 2)
    threshold_results.append(row)

threshold_df = pd.DataFrame(threshold_results)
threshold_df = threshold_df[["Threshold", "Accuracy", "Precision", "Recall", "F1-score", "ROC-AUC"]]

# Round only for display.
display(threshold_df.round(4))

best_threshold_row = threshold_df.sort_values(["F1-score", "Recall"], ascending=False).iloc[0]
print("Best threshold based on F1-score:")
display(best_threshold_row.to_frame().T.round(4))
```

```python
# Plot threshold effect on Precision, Recall, and F1-score.
plt.figure(figsize=(8, 5))
plt.plot(threshold_df["Threshold"], threshold_df["Precision"], marker="o", label="Precision")
plt.plot(threshold_df["Threshold"], threshold_df["Recall"], marker="o", label="Recall")
plt.plot(threshold_df["Threshold"], threshold_df["F1-score"], marker="o", label="F1-score")
plt.axvline(best_threshold_row["Threshold"], color="black", linestyle="--", label="Best F1 threshold")
plt.title("Logistic Regression: Threshold vs Metrics")
plt.xlabel("Decision threshold")
plt.ylabel("Metric value")
plt.legend()
plt.tight_layout()
plt.show()
```

```python
# Confusion matrix and classification report at the best threshold.
best_threshold = float(best_threshold_row["Threshold"])
best_logistic_pred = (logistic_prob >= best_threshold).astype(int)

print("Best threshold:", best_threshold)
print("\nConfusion matrix:")
display(pd.DataFrame(
    confusion_matrix(y_test, best_logistic_pred),
    index=["Actual 0", "Actual 1"],
    columns=["Predicted 0", "Predicted 1"],
))

print("\nClassification report:")
print(classification_report(y_test, best_logistic_pred, zero_division=0))
```

### Inference for Question 3(a)

Based on the actual threshold table:

- Very low thresholds such as 0.10 give very high recall but poor precision. This means the model predicts too many employees as likely to leave.
- Very high thresholds such as 0.80 or 0.90 give low recall. This means many employees who actually leave are missed.
- The best F1-score occurs around the default threshold of **0.50**, with F1-score around **0.5962**.
- At threshold 0.50, the model gives a reasonable balance between precision and recall for the employee-leaving class.
- ROC-AUC is around **0.7268**, so the Logistic Regression model has moderate ranking ability.

**Exam conclusion:** Threshold selection should depend on business cost. If missing a leaving employee is more costly, choose a lower threshold to increase recall. If unnecessary retention action is costly, choose a higher threshold to increase precision.

### Question 3(b) - 16 marks

**Full question:**  
Compare the performance of logistic regression, Decision Tree, Random Forest models on the given dataset. Analyze the results, clearly state the changes or optimizations you would make to each model before re-fitting, and justify your approach.

### Approach

We will compare:

1. Logistic Regression.
2. Decision Tree.
3. Random Forest.

Optimizations used:

- Logistic Regression: scale numeric variables, one-hot encode categorical variables, use `class_weight="balanced"`, and tune decision threshold.
- Decision Tree: restrict depth, require minimum samples in leaves, and use class weights to reduce overfitting and handle imbalance.
- Random Forest: use many trees, restrict depth, require minimum samples in leaves, and use class weights to improve generalization.

```python
# Define baseline models.
# Baseline models show what happens before applying careful tuning.
baseline_models = {
    "Baseline Logistic Regression": Pipeline(steps=[
        ("preprocess", logistic_preprocess),
        ("model", LogisticRegression(max_iter=2000, random_state=RANDOM_STATE)),
    ]),
    "Baseline Decision Tree": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", DecisionTreeClassifier(random_state=RANDOM_STATE)),
    ]),
    "Baseline Random Forest": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", RandomForestClassifier(n_estimators=100, random_state=RANDOM_STATE, n_jobs=-1)),
    ]),
}

# Define optimized models based on the nature of each algorithm.
optimized_models = {
    "Optimized Logistic Regression": logistic_model,
    "Optimized Decision Tree": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", DecisionTreeClassifier(
            max_depth=6,
            min_samples_leaf=25,
            class_weight="balanced",
            random_state=RANDOM_STATE,
        )),
    ]),
    "Optimized Random Forest": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", RandomForestClassifier(
            n_estimators=300,
            max_depth=8,
            min_samples_leaf=15,
            class_weight="balanced",
            random_state=RANDOM_STATE,
            n_jobs=-1,
        )),
    ]),
}

# Fit and evaluate all models.
comparison_rows = []

for name, model in baseline_models.items():
    model.fit(X_train, y_train)
    metrics = evaluate_model(model, X_test, y_test)
    metrics["Model"] = name
    metrics["Type"] = "Baseline"
    comparison_rows.append(metrics)

for name, model in optimized_models.items():
    model.fit(X_train, y_train)

    # Use best threshold only for optimized logistic regression.
    if name == "Optimized Logistic Regression":
        metrics = evaluate_model(model, X_test, y_test, threshold=best_threshold)
    else:
        metrics = evaluate_model(model, X_test, y_test)

    metrics["Model"] = name
    metrics["Type"] = "Optimized"
    comparison_rows.append(metrics)

model_comparison = pd.DataFrame(comparison_rows)
model_comparison = model_comparison[["Type", "Model", "Accuracy", "Precision", "Recall", "F1-score", "ROC-AUC"]]
display(model_comparison.round(4))
```

```python
# Plot the optimized model comparison using key metrics.
optimized_only = model_comparison[model_comparison["Type"] == "Optimized"].copy()
plot_df = optimized_only.melt(
    id_vars="Model",
    value_vars=["Accuracy", "Precision", "Recall", "F1-score", "ROC-AUC"],
    var_name="Metric",
    value_name="Score",
)

plt.figure(figsize=(11, 5))
sns.barplot(data=plot_df, x="Metric", y="Score", hue="Model")
plt.title("Optimized Model Performance Comparison")
plt.ylim(0, 1)
plt.legend(loc="lower right")
plt.tight_layout()
plt.show()
```

```python
# Identify the best model by F1-score because the target is moderately imbalanced.
best_model_row = optimized_only.sort_values("F1-score", ascending=False).iloc[0]
print("Best optimized model by F1-score:")
display(best_model_row.to_frame().T.round(4))
```

### Inference for Question 3(b)

Based on the actual output:

- Logistic Regression is interpretable and gives a useful baseline, but it has lower F1-score because it models mostly linear relationships.
- Decision Tree improves performance because it captures nonlinear decision rules, but an unrestricted tree can overfit. Therefore, `max_depth` and `min_samples_leaf` are important.
- Random Forest gives the best overall result among the optimized models, with F1-score around **0.7304** and ROC-AUC around **0.8646**.
- The optimized Random Forest performs better than Logistic Regression because employee attrition patterns are likely nonlinear and involve feature interactions.
- `class_weight="balanced"` is justified because the target class is moderately imbalanced.

#### Model-wise optimization justification

| Model | Optimization | Justification |
|---|---|---|
| Logistic Regression | Scaling, one-hot encoding, class weights, threshold tuning | Improves handling of numeric scale, categorical variables, imbalance, and precision-recall trade-off |
| Decision Tree | Limit depth, increase leaf size, class weights | Reduces overfitting and improves minority-class handling |
| Random Forest | More trees, limited depth, minimum leaf size, class weights | Reduces variance, improves stability, handles nonlinear relationships |

**Exam conclusion:** For this dataset, Random Forest is the recommended model because it gives the best balance of accuracy, recall, F1-score, and ROC-AUC.

### Question 3(c) - 8 marks

**Full question:**  
Summarize the solution to the business problem as follows:

- Feature Importance (4 marks): Identify and explain which features are most influential in solving the problem and why.
- Evaluation Metrics Performance (2 marks): Provide an analysis of the model's performance based on key evaluation metrics.
- Overall Results and Observations (2 marks): Summarize the overall findings, including key insights and their implications for the business problem.

### Approach

We use the optimized Random Forest because it is the best model based on F1-score and ROC-AUC. Feature importance tells us which variables contribute most to prediction.

```python
# Extract feature importance from the optimized Random Forest model.
best_rf_model = optimized_models["Optimized Random Forest"]
best_rf_model.fit(X_train, y_train)

# Get encoded feature names from the preprocessing step.
encoded_feature_names = best_rf_model.named_steps["preprocess"].get_feature_names_out()
rf_importance_values = best_rf_model.named_steps["model"].feature_importances_

rf_importance = pd.DataFrame({
    "encoded_feature": encoded_feature_names,
    "importance": rf_importance_values,
}).sort_values("importance", ascending=False)

display(rf_importance.head(15).round(4))

plt.figure(figsize=(9, 6))
sns.barplot(
    data=rf_importance.head(12),
    y="encoded_feature",
    x="importance",
    color="#4E79A7",
)
plt.title("Top Random Forest Feature Importances")
plt.xlabel("Importance")
plt.ylabel("Feature")
plt.tight_layout()
plt.show()
```

```python
# Group one-hot encoded feature importances back to original feature names.
def recover_original_feature(encoded_name):
    # Map transformed feature names back to original columns.
    clean_name = encoded_name.split("__", 1)[1]

    if clean_name in model_numeric_cols:
        return clean_name

    for col in model_categorical_cols:
        if clean_name.startswith(col + "_"):
            return col

    return clean_name


rf_importance["original_feature"] = rf_importance["encoded_feature"].apply(recover_original_feature)
grouped_importance = (
    rf_importance
    .groupby("original_feature", as_index=False)["importance"]
    .sum()
    .sort_values("importance", ascending=False)
)

display(grouped_importance.round(4))

plt.figure(figsize=(8, 5))
sns.barplot(data=grouped_importance, y="original_feature", x="importance", color="#59A14F")
plt.title("Grouped Feature Importance by Original Feature")
plt.xlabel("Total importance")
plt.ylabel("Original feature")
plt.tight_layout()
plt.show()
```

```python
# Final evaluation for the recommended model.
rf_pred = best_rf_model.predict(X_test)
rf_prob = best_rf_model.predict_proba(X_test)[:, 1]

print("Recommended model: Optimized Random Forest")
print("\nConfusion matrix:")
display(pd.DataFrame(
    confusion_matrix(y_test, rf_pred),
    index=["Actual 0", "Actual 1"],
    columns=["Predicted 0", "Predicted 1"],
))

print("\nClassification report:")
print(classification_report(y_test, rf_pred, zero_division=0))

final_rf_metrics = evaluate_predictions(y_test, rf_pred, rf_prob)
display(pd.DataFrame([final_rf_metrics]).round(4))
```

### Inference for Question 3(c)

#### Feature importance

Based on the optimized Random Forest output, the most influential features are:

1. `JoiningYear`: The strongest feature, suggesting that employee joining cohort or tenure-related patterns are important.
2. `City`: Location has a strong relationship with leaving behavior, especially the `Pune` category.
3. `Education`: Education level contributes to attrition prediction, especially the `Masters` category.
4. `Gender`: Gender-related patterns are visible in the model.
5. `PaymentTier`: Compensation tier is important; higher payment tier is associated with lower leaving tendency.

#### Evaluation metrics performance

The optimized Random Forest achieves the best balance among the tested models. Its F1-score is around **0.7304**, and ROC-AUC is around **0.8646**, meaning it separates leaving and non-leaving employees well.

#### Overall business observations

- The business problem is employee attrition prediction.
- The dataset is moderately imbalanced, so F1-score and recall are important.
- Random Forest is the best model among those tested because it captures nonlinear patterns and feature interactions.
- Employees' joining year, city, education level, gender, and payment tier are the strongest signals for predicting whether they may leave.
- From a business perspective, HR teams can use such a model to prioritize retention interventions for employees with high predicted leave probability.

#### Caution

Feature importance indicates predictive influence, not direct causation. Business decisions should combine model output with domain knowledge and ethical review.

## Final Exam Revision Summary

- Use **precision** when false positives are costly.
- Use **recall** when false negatives are costly.
- Use **F1-score** when both precision and recall matter, especially in imbalanced classification.
- Gaussian Naive Bayes assumes conditional independence and Gaussian feature distribution.
- Bagging reduces variance; Random Forest improves bagging using random feature selection.
- Cross-validation gives a more reliable estimate of model performance than a single split.
- In preprocessing, fit imputation, scaling, and encoding only on training data.
- For this dataset, the target is moderately imbalanced, so class weights and threshold tuning are justified.
- Logistic Regression is interpretable but less flexible.
- Decision Tree captures nonlinear patterns but can overfit.
- Random Forest gives the best result here because it combines many trees and reduces variance.

# ML2_ESA_Dec21_Set1_Solved_Notebook.ipynb

# Machine Learning - II ESA December 2021: Solved Question Paper

**Course:** UE20CS931 - Machine Learning - II  
**Exam:** December 2021 End Semester Assessment  
**Maximum Marks:** 100  
**Dataset:** `Loan.csv`

This notebook solves the complete paper in an exam-focused style. It includes the full question before every answer, theory explanations, formulas, Python code, plots, model evaluation, and interpretation based on actual executed output.

## Important dataset note

The PDF description mentions an `Exited` column, but the actual CSV provided with this paper contains the target column named `Loan`. Therefore, this notebook uses `Loan` as the binary target variable:

- `Loan = 1`: positive class as described in the question paper.
- `Loan = 0`: negative class.

This is mentioned explicitly because using a non-existing `Exited` column would make the notebook fail.

```python
# Confirm the exact Python interpreter used by this notebook.
import sys
print("Python executable:", sys.executable)
print("Python version:", sys.version.split()[0])
```

## Section A - Theory Questions (30 marks)

### Question 1(a) - 5 marks

**Full question:**  
Define Precision and Recall.

### Answer

Precision and recall are classification performance metrics derived from the confusion matrix.

| Term | Meaning |
|---|---|
| True Positive (TP) | Actual positive and predicted positive |
| True Negative (TN) | Actual negative and predicted negative |
| False Positive (FP) | Actual negative but predicted positive |
| False Negative (FN) | Actual positive but predicted negative |

#### Precision

$$
\text{Precision} = \frac{TP}{TP + FP}
$$

Precision answers: **Out of all observations predicted as positive, how many were actually positive?**

High precision means the model makes fewer false positive errors. In a loan-payback context, precision is important if wrongly classifying a customer as a positive class creates business cost.

#### Recall

$$
\text{Recall} = \frac{TP}{TP + FN}
$$

Recall answers: **Out of all actual positive observations, how many did the model correctly identify?**

High recall means the model misses fewer actual positive cases. It is important when false negatives are costly.

#### Key difference

- Precision controls false positives.
- Recall controls false negatives.
- A threshold change usually increases one and decreases the other.

#### Related metric: F1-score

$$
F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}
$$

F1-score is useful when the target class is imbalanced and both precision and recall matter.

### Question 1(b) - 5 marks

**Full question:**  
Differentiate between bagging and boosting.

### Answer

Bagging and boosting are ensemble learning techniques. Both combine multiple learners, but they differ in how the learners are trained and combined.

| Point | Bagging | Boosting |
|---|---|---|
| Full form | Bootstrap Aggregating | Sequential boosting of weak learners |
| Training style | Parallel and independent | Sequential and dependent |
| Data sampling | Bootstrap samples with replacement | Learners focus more on previous errors |
| Main objective | Reduce variance | Reduce bias and sometimes variance |
| Base learner | Usually high-variance models such as decision trees | Usually weak learners such as shallow trees |
| Combination | Majority vote or averaging | Weighted voting or additive model |
| Example algorithms | Bagging Classifier, Random Forest | AdaBoost, Gradient Boosting, XGBoost |
| Overfitting tendency | Usually less prone | Can overfit if too many learners or high learning rate |

#### Bagging example: Random Forest

Random Forest trains many decision trees on bootstrap samples. At each split, it also uses a random subset of features. Final classification is by majority voting.

#### Boosting example: Gradient Boosting

Gradient Boosting builds trees sequentially. Each new tree tries to correct the errors made by the previous model.

#### Exam conclusion

Bagging mainly improves stability by reducing variance, while boosting improves accuracy by sequentially correcting model errors.

### Question 1(c) - 5 marks

**Full question:**  
Describe ROC curve.

### Answer

The ROC curve, or Receiver Operating Characteristic curve, is a graph used to evaluate binary classification models across different threshold values.

It plots:

$$
\text{True Positive Rate (TPR)} = \frac{TP}{TP + FN}
$$

against:

$$
\text{False Positive Rate (FPR)} = \frac{FP}{FP + TN}
$$

#### Interpretation

- The x-axis is FPR.
- The y-axis is TPR or recall.
- Each point on the curve represents a different classification threshold.
- A model closer to the top-left corner is better.
- The diagonal line represents random guessing.

#### AUC

The area under the ROC curve is called ROC-AUC.

- AUC = 0.5 means random performance.
- AUC close to 1 means excellent class separation.
- AUC below 0.5 means the model is worse than random.

#### Why ROC curve is useful

- It evaluates ranking ability independent of one fixed threshold.
- It helps compare multiple models.
- It is useful when the decision threshold is not finalized.

#### Limitation

For heavily imbalanced datasets, precision-recall curves may be more informative because ROC-AUC can look optimistic when the negative class dominates.

### Question 1(d) - 5 marks

**Full question:**  
Explain Gini Impurity & Entropy.

### Answer

Gini impurity and entropy are node impurity measures used in decision tree algorithms. They help decide the best feature and split point.

#### Gini impurity

For a node with class probabilities $p_1, p_2, ..., p_k$:

$$
Gini = 1 - \sum_{i=1}^{k} p_i^2
$$

For binary classification:

$$
Gini = 1 - p^2 - (1-p)^2
$$

Gini impurity is 0 when the node is pure, meaning all observations belong to one class.

#### Entropy

Entropy measures uncertainty or disorder.

$$
Entropy = - \sum_{i=1}^{k} p_i \log_2(p_i)
$$

Entropy is 0 for a pure node and maximum when classes are evenly mixed.

#### Information gain

Decision trees using entropy choose splits that maximize information gain:

$$
Information\ Gain = Entropy(parent) - Weighted\ Entropy(children)
$$

#### Difference

| Aspect | Gini Impurity | Entropy |
|---|---|---|
| Formula | $1 - \sum p_i^2$ | $-\sum p_i\log_2(p_i)$ |
| Used in | CART trees | ID3/C4.5 style trees |
| Computation | Slightly faster | Slightly more expensive |
| Split behavior | Often similar to entropy | Often similar to Gini |

#### Exam conclusion

Both Gini and entropy measure impurity. A good split reduces impurity and creates purer child nodes.

### Question 1(e) - 5 marks

**Full question:**  
Explain the working of Random Forest and list two advantages.

### Answer

Random Forest is an ensemble learning algorithm that builds many decision trees and combines their predictions.

#### Working of Random Forest

1. Create multiple bootstrap samples from the training dataset.
2. Train one decision tree on each bootstrap sample.
3. At every split in each tree, consider only a random subset of features.
4. Grow many trees.
5. For classification, each tree votes for a class.
6. The final prediction is the class with majority vote.

#### Why it works well

Random Forest reduces overfitting by averaging many decision trees. Individual trees may be noisy, but their combined prediction is more stable.

#### Two advantages

1. **Reduces overfitting:** Compared with a single decision tree, Random Forest has lower variance.
2. **Provides feature importance:** It can rank variables based on their contribution to prediction.

Other advantages:

- Handles nonlinear relationships.
- Works well with mixed numerical and categorical inputs after encoding.
- Handles feature interactions automatically.
- Usually gives strong performance with limited tuning.

#### Limitations

- Less interpretable than a single decision tree.
- Can be computationally heavier.
- Feature importance can be biased toward variables with many split points.

### Question 1(f) - 5 marks

**Full question:**  
Explain Ensemble Learning.

### Answer

Ensemble learning combines multiple machine learning models to produce a stronger model.

The main idea is that a group of models can perform better than a single model when their errors are not identical.

#### Types of ensemble learning

1. **Bagging:** Trains multiple models independently on bootstrap samples. Example: Random Forest.
2. **Boosting:** Trains models sequentially, where each new model corrects previous errors. Example: AdaBoost, Gradient Boosting.
3. **Voting:** Combines predictions from multiple models by majority vote or averaged probabilities.
4. **Stacking:** Uses base model predictions as input to a meta-model.

#### Benefits

- Improves predictive performance.
- Reduces variance.
- Can reduce bias.
- Improves robustness.
- Handles complex nonlinear patterns.

#### Limitations

- More computational cost.
- Lower interpretability.
- Can overfit if tuning is poor, especially in boosting.

#### Exam conclusion

Ensemble learning is a powerful method because it combines several weak or diverse models to obtain a more accurate and stable final prediction.

## Section B - Data Analysis and Base Model (30 marks)

The coding questions use `Loan.csv`, which contains bank customer information such as credit score, geography, gender, age, tenure, balance, products used, credit-card status, active-member status, estimated salary, and the target column `Loan`.

### Imports and setup

```python
# Core data and numerical libraries.
import warnings
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

# Statistical tests.
from scipy.stats import chi2_contingency, pointbiserialr

# Machine learning tools.
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    cohen_kappa_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.tree import DecisionTreeClassifier

warnings.filterwarnings("ignore")

# Plot style for clean notebook output.
sns.set_theme(style="whitegrid", context="notebook")
plt.rcParams["figure.figsize"] = (8, 5)
plt.rcParams["axes.titlesize"] = 12

RANDOM_STATE = 42
TARGET = "Loan"
```

### Question 2(a) - 5 marks

**Full question:**  
Read the dataset and summarize important observations from the data set.

1. Find out number of rows; no. and types of variables (continuous, categorical etc.)
2. Calculate five-point summary for numerical variables.
3. Summarize observations for categorical variables - no. of categories, percentage observations in each category.

### Approach

We will inspect the shape, first rows, data types, numerical summaries, and categorical proportions.

```python
# Read the dataset.
data_path = Path("Loan.csv")
df = pd.read_csv(data_path)

print("Dataset shape:", df.shape)
print("\nColumn names:")
print(df.columns.tolist())

display(df.head())

print("\nData information:")
df.info()
```

```python
# Separate numerical and categorical columns.
numeric_cols = df.select_dtypes(include=np.number).columns.tolist()
categorical_cols = df.select_dtypes(exclude=np.number).columns.tolist()

print("Numerical columns:", numeric_cols)
print("Number of numerical columns:", len(numeric_cols))
print("\nCategorical columns:", categorical_cols)
print("Number of categorical columns:", len(categorical_cols))
```

```python
# Five-point summary and descriptive statistics for numerical variables.
# The five-point summary is min, Q1, median, Q3, and max.
numeric_summary = df[numeric_cols].describe().T
display(numeric_summary)
```

```python
# Categorical summaries: unique values, counts, and proportions.
categorical_summary = df[categorical_cols].describe().T
display(categorical_summary)

for col in categorical_cols:
    print(f"\nCategory proportions for {col}:")
    temp = (
        df[col]
        .value_counts(dropna=False)
        .rename_axis(col)
        .reset_index(name="count")
    )
    temp["proportion"] = (temp["count"] / len(df)).round(4)
    display(temp)
```

### Inference for Question 2(a)

Based on the actual output:

- The dataset has **1630 rows and 12 columns**.
- The actual target column is `Loan`.
- `CustomerId` is an identifier and should not be used for modeling.
- Numerical columns include `CreditScore`, `Age`, `Tenure`, `Balance`, `NumOfProducts`, `HasCrCard`, `IsActiveMember`, `EstimatedSalary`, and `Loan`.
- Categorical columns are `Geography` and `Gender`.
- `Geography` has three categories: France, Germany, and Spain. France is the largest group.
- `Gender` contains Male, Female, and missing values.
- `Balance` has many zero values, so the distribution is not normal.

### Question 2(b) - 10 marks

**Full question:**  
Check for defects in the data. Perform necessary actions to "fix" these defects.

1. Do variables have missing/null values? Is it a defect? If yes, what steps are being taken to rectify the problem.
2. Do variables have outliers? Is it a defect? If yes, what steps are being taken to rectify the problem.
3. Is the target distributed evenly? Is it a defect? If yes, what steps are being taken to rectify the problem.
4. Is there any textual data? Is it a defect? If yes, what steps are being taken to rectify the problem.

### Approach

We will check missing values, duplicates, outliers using IQR, class imbalance, and textual/categorical columns. Then we will create a cleaned dataset.

```python
# Missing value report.
missing_count = df.isna().sum()
missing_percent = (missing_count / len(df) * 100).round(2)
missing_table = pd.DataFrame({
    "missing_count": missing_count,
    "missing_percent": missing_percent,
}).sort_values("missing_count", ascending=False)

display(missing_table)

plt.figure(figsize=(8, 4))
missing_count.sort_values(ascending=False).plot(kind="bar", color="#4C78A8")
plt.title("Missing Values by Column")
plt.ylabel("Missing count")
plt.xticks(rotation=45, ha="right")
plt.tight_layout()
plt.show()
```

```python
# Duplicate check.
duplicate_rows = df.duplicated().sum()
print("Duplicate rows:", duplicate_rows)
```

```python
# IQR-based outlier report for numerical predictors.
# CustomerId is excluded because it is an identifier.
outlier_rows = []
for col in [c for c in numeric_cols if c not in [TARGET, "CustomerId"]]:
    q1 = df[col].quantile(0.25)
    q3 = df[col].quantile(0.75)
    iqr = q3 - q1
    lower = q1 - 1.5 * iqr
    upper = q3 + 1.5 * iqr
    outlier_mask = (df[col] < lower) | (df[col] > upper)
    outlier_rows.append({
        "feature": col,
        "lower_bound": round(lower, 3),
        "upper_bound": round(upper, 3),
        "outlier_count": int(outlier_mask.sum()),
        "outlier_percent": round(outlier_mask.mean() * 100, 2),
    })

outlier_table = pd.DataFrame(outlier_rows)
display(outlier_table)
```

```python
# Boxplots for numerical variables.
box_cols = [c for c in numeric_cols if c not in [TARGET, "CustomerId"]]
fig, axes = plt.subplots(2, 4, figsize=(16, 8))
axes = axes.ravel()

for ax, col in zip(axes, box_cols):
    sns.boxplot(y=df[col], ax=ax, color="#72B7B2")
    ax.set_title(col)
    ax.set_xlabel("")

plt.tight_layout()
plt.show()
```

```python
# Target distribution.
target_counts = df[TARGET].value_counts().sort_index()
target_prop = df[TARGET].value_counts(normalize=True).sort_index()

target_balance = pd.DataFrame({
    "class_label": target_counts.index,
    "count": target_counts.values,
    "proportion": target_prop.values.round(4),
})
display(target_balance)

plt.figure(figsize=(6, 4))
sns.countplot(data=df, x=TARGET, palette=["#59A14F", "#E15759"])
plt.title("Target Distribution")
plt.xlabel("Loan")
plt.ylabel("Count")
plt.tight_layout()
plt.show()
```

```python
# Create a cleaned dataset.
df_clean = df.copy()

# Drop exact duplicate rows if present.
df_clean = df_clean.drop_duplicates()

# Impute Age using median because it is numerical and median is robust.
age_median = df_clean["Age"].median()
df_clean["Age"] = df_clean["Age"].fillna(age_median)

# Impute Gender using mode because it is categorical.
gender_mode = df_clean["Gender"].mode()[0]
df_clean["Gender"] = df_clean["Gender"].fillna(gender_mode)

# Drop CustomerId because it is an identifier and does not carry generalizable signal.
df_clean = df_clean.drop(columns=["CustomerId"])

print("Age median used:", age_median)
print("Gender mode used:", gender_mode)
print("Cleaned shape:", df_clean.shape)
print("\nMissing values after cleaning:")
display(df_clean.isna().sum().to_frame("missing_after_cleaning"))
```

### Inference for Question 2(b)

Based on the actual output:

- `Age` has **151 missing values**. This is a defect because most models cannot handle missing values directly. It is fixed using median imputation.
- `Gender` has **59 missing values**. This is fixed using mode imputation.
- There are no duplicate rows.
- `CustomerId` is removed because it is an identifier, not a meaningful predictive feature.
- `CreditScore` shows about **5.71% IQR outliers**, but these are realistic low/high scores, so they are not automatically removed.
- `NumOfProducts` has a few values flagged as outliers, but values 1 to 4 are valid business categories, so they are retained.
- The target is imbalanced: around **78.9%** class 0 and **21.1%** class 1. This is a modeling concern, so stratified splitting, class weights, and F1/recall are used.
- Textual/categorical variables `Geography` and `Gender` are not defects. They are encoded using one-hot encoding.

### Question 2(c) - 5 marks

**Full question:**  
Summarize relationships among variables.

1. Plot relevant categorical plots. Find out which variables are most correlated or appear to be in causation with Target? Do you want to exclude some variables from the model based on this analysis? What other actions will you take?
2. Plot all independent variables with the target and find out the relationship. Perform the relevant tests to find out if the independent variables are associated with the target variable.

### Approach

We use:

- Count plots and target-rate plots for categorical variables.
- Boxplots for numerical variables against target.
- Correlation and point-biserial correlation for numerical variables.
- Chi-square tests for categorical variables.

```python
# Target rate by categorical variables.
for col in ["Geography", "Gender"]:
    rate_table = df_clean.groupby(col)[TARGET].agg(["count", "mean"]).sort_values("mean", ascending=False)
    print(f"\nTarget rate by {col}:")
    display(rate_table)

    plt.figure(figsize=(7, 4))
    sns.barplot(data=df_clean, x=col, y=TARGET, estimator=np.mean, errorbar=None, color="#F28E2B")
    plt.title(f"Average Loan Target by {col}")
    plt.ylabel("Mean target rate")
    plt.tight_layout()
    plt.show()
```

```python
# Numerical relationships with target.
num_predictors = [c for c in df_clean.select_dtypes(include=np.number).columns if c != TARGET]

fig, axes = plt.subplots(2, 4, figsize=(16, 8))
axes = axes.ravel()

for ax, col in zip(axes, num_predictors):
    sns.boxplot(data=df_clean, x=TARGET, y=col, ax=ax, palette=["#A0CBE8", "#FFBE7D"])
    ax.set_title(f"{col} by Loan")

plt.tight_layout()
plt.show()
```

```python
# Correlation heatmap for numerical variables.
plt.figure(figsize=(9, 6))
numeric_corr = df_clean.corr(numeric_only=True)
sns.heatmap(numeric_corr, annot=True, cmap="RdBu_r", center=0, fmt=".2f")
plt.title("Correlation Heatmap")
plt.tight_layout()
plt.show()

display(numeric_corr[TARGET].sort_values(key=lambda s: s.abs(), ascending=False).to_frame("correlation_with_target"))
```

```python
# Chi-square test for categorical/binary/discrete variables.
# H0: Feature and target are independent.
chi_features = ["Geography", "Gender", "HasCrCard", "IsActiveMember", "NumOfProducts"]
chi_rows = []

for col in chi_features:
    contingency = pd.crosstab(df_clean[col], df_clean[TARGET])
    chi2, p_value, dof, expected = chi2_contingency(contingency)
    chi_rows.append({
        "feature": col,
        "test": "chi-square",
        "p_value": p_value,
        "significant_at_0.05": p_value < 0.05,
    })

chi_test_results = pd.DataFrame(chi_rows).sort_values("p_value")
display(chi_test_results)
```

```python
# Point-biserial correlation for continuous predictors with binary target.
# H0: No linear association with target.
continuous_for_test = ["CreditScore", "Age", "Tenure", "Balance", "EstimatedSalary"]
pb_rows = []

for col in continuous_for_test:
    corr, p_value = pointbiserialr(df_clean[col], df_clean[TARGET])
    pb_rows.append({
        "feature": col,
        "point_biserial_corr": corr,
        "p_value": p_value,
        "significant_at_0.05": p_value < 0.05,
    })

pb_results = pd.DataFrame(pb_rows).sort_values("p_value")
display(pb_results)
```

### Inference for Question 2(c)

Based on the actual output:

- `Age` has the strongest numerical relationship with the target. Its point-biserial correlation is positive and statistically significant.
- `CreditScore` has a significant negative relationship with the target.
- `Balance` and `EstimatedSalary` are also statistically significant, though the relationship is weaker.
- `Tenure` is not statistically significant at the 5% level in this dataset.
- Chi-square tests show `Geography`, `Gender`, `IsActiveMember`, and `NumOfProducts` are significantly associated with the target.
- `HasCrCard` is not significant at the 5% level.
- We do not remove weak variables immediately because tree-based models can handle interactions. However, `CustomerId` is removed because it is an identifier.

### Question 2(d) - 5 marks

**Full question:**  
Split dataset into train and test (70:30). Are both train and test representative of the overall data? How would you ascertain this statistically?

### Approach

We use a stratified 70:30 split. Stratification preserves target class proportions in train and test sets.

```python
# Separate features and target.
X = df_clean.drop(columns=[TARGET])
y = df_clean[TARGET]

# Stratified split keeps target distribution similar in train and test.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.30,
    random_state=RANDOM_STATE,
    stratify=y,
)

print("X_train shape:", X_train.shape)
print("X_test shape:", X_test.shape)
print("y_train shape:", y_train.shape)
print("y_test shape:", y_test.shape)

split_check = pd.DataFrame({
    "overall": y.value_counts(normalize=True).sort_index(),
    "train": y_train.value_counts(normalize=True).sort_index(),
    "test": y_test.value_counts(normalize=True).sort_index(),
})
display(split_check)
```

### Inference for Question 2(d)

The train-test split is representative because the target proportions in the overall, training, and testing datasets are almost the same. This is statistically ensured using stratified sampling.

### Question 2(e) - 5 marks

**Full question:**  
Fit a base model and explain the reason of selecting that model. Please write your key observations.

1. What is the overall Accuracy? Please comment on whether it is good or not.
2. What is Precision, Recall and F1 Score and what will be the optimization objective keeping in mind the problem statement.
3. Which variables are significant?
4. What is Cohen's Kappa Value and what inference do you make from the model?
5. Which other key model output parameters do you want to look at?

### Base model choice

Logistic Regression is selected as the base model because:

- It is a standard baseline for binary classification.
- It is interpretable.
- Its coefficients help explain feature direction.
- It gives probabilities, so thresholds can be tuned later.

```python
# Identify preprocessing columns.
model_numeric_cols = X.select_dtypes(include=np.number).columns.tolist()
model_categorical_cols = X.select_dtypes(exclude=np.number).columns.tolist()

print("Numeric model columns:", model_numeric_cols)
print("Categorical model columns:", model_categorical_cols)


def make_one_hot_encoder():
    # Create a OneHotEncoder compatible with different sklearn versions.
    try:
        return OneHotEncoder(handle_unknown="ignore", sparse_output=False)
    except TypeError:
        return OneHotEncoder(handle_unknown="ignore", sparse=False)


# Logistic Regression needs scaled numerical features.
logistic_preprocess = ColumnTransformer(
    transformers=[
        ("num", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]), model_numeric_cols),
        ("cat", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("onehot", make_one_hot_encoder()),
        ]), model_categorical_cols),
    ]
)

# Tree-based models do not need scaling.
tree_preprocess = ColumnTransformer(
    transformers=[
        ("num", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="median")),
        ]), model_numeric_cols),
        ("cat", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("onehot", make_one_hot_encoder()),
        ]), model_categorical_cols),
    ]
)


def evaluate_predictions(y_true, y_pred, y_prob):
    # Return common classification metrics in one dictionary.
    return {
        "Accuracy": accuracy_score(y_true, y_pred),
        "Precision": precision_score(y_true, y_pred, zero_division=0),
        "Recall": recall_score(y_true, y_pred, zero_division=0),
        "F1-score": f1_score(y_true, y_pred, zero_division=0),
        "ROC-AUC": roc_auc_score(y_true, y_prob),
        "Cohen Kappa": cohen_kappa_score(y_true, y_pred),
    }
```

```python
# Fit the base Logistic Regression model.
base_logistic = Pipeline(steps=[
    ("preprocess", logistic_preprocess),
    ("model", LogisticRegression(max_iter=2000, random_state=RANDOM_STATE)),
])

base_logistic.fit(X_train, y_train)

base_prob = base_logistic.predict_proba(X_test)[:, 1]
base_pred = base_logistic.predict(X_test)
base_metrics = evaluate_predictions(y_test, base_pred, base_prob)

display(pd.DataFrame([base_metrics]).round(4))

print("Confusion matrix:")
display(pd.DataFrame(
    confusion_matrix(y_test, base_pred),
    index=["Actual 0", "Actual 1"],
    columns=["Predicted 0", "Predicted 1"],
))

print("Classification report:")
print(classification_report(y_test, base_pred, zero_division=0))
```

```python
# Logistic Regression coefficient interpretation.
feature_names = base_logistic.named_steps["preprocess"].get_feature_names_out()
coefficients = base_logistic.named_steps["model"].coef_[0]

coef_table = (
    pd.DataFrame({
        "feature": feature_names,
        "coefficient": coefficients,
        "abs_coefficient": np.abs(coefficients),
    })
    .sort_values("abs_coefficient", ascending=False)
)

display(coef_table.head(12).round(4))
```

### Inference for Question 2(e)

Based on the actual base model output:

- Base Logistic Regression gives accuracy around **0.8057**.
- However, because the target is imbalanced, accuracy alone is not enough.
- The recall and F1-score for the positive class are much lower than accuracy, meaning the model misses many positive-class cases.
- Cohen's Kappa is around **0.2538**, which indicates only fair agreement beyond chance.
- Important variables from coefficients and statistical tests include `Age`, `CreditScore`, `NumOfProducts`, `IsActiveMember`, `Geography`, and `Gender`.
- Other key outputs to examine are confusion matrix, ROC-AUC, class-wise recall, F1-score, and ROC curve.

**Optimization objective:** Since the target is imbalanced, F1-score and recall should be monitored along with accuracy. If the business mainly wants overall correctness, optimize accuracy; if the business wants to catch positive-class cases, optimize recall/F1.

## Section C - Model Improvement and Business Summary (40 marks)

### Question 3(a) - 20 marks

**Full question:**  
How do you improve the accuracy of the model? Write clearly the changes that you will make before re-fitting the model. Fit the final model.

### Improvement plan

To improve the base model:

1. Use proper preprocessing pipelines to avoid data leakage.
2. Drop `CustomerId` because it is an identifier.
3. Impute missing values.
4. One-hot encode categorical variables.
5. Try models that capture nonlinear relationships: Decision Tree, Random Forest, and Gradient Boosting.
6. Use class-weighted models where useful because the target is imbalanced.
7. Use a compact `GridSearchCV` for Gradient Boosting to tune important hyperparameters without excessive runtime.
8. Compare models using accuracy, precision, recall, F1-score, ROC-AUC, and Cohen's Kappa.

```python
# Candidate models for improvement.
candidate_models = {
    "Base Logistic Regression": base_logistic,
    "Balanced Logistic Regression": Pipeline(steps=[
        ("preprocess", logistic_preprocess),
        ("model", LogisticRegression(
            max_iter=2000,
            class_weight="balanced",
            random_state=RANDOM_STATE,
        )),
    ]),
    "Decision Tree": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", DecisionTreeClassifier(
            max_depth=5,
            min_samples_leaf=25,
            class_weight="balanced",
            random_state=RANDOM_STATE,
        )),
    ]),
    "Random Forest": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", RandomForestClassifier(
            n_estimators=300,
            max_depth=8,
            min_samples_leaf=10,
            class_weight="balanced",
            random_state=RANDOM_STATE,
            n_jobs=1,
        )),
    ]),
    "Gradient Boosting": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", GradientBoostingClassifier(random_state=RANDOM_STATE)),
    ]),
}

model_rows = []
fitted_models = {}

for name, model in candidate_models.items():
    model.fit(X_train, y_train)
    pred = model.predict(X_test)
    prob = model.predict_proba(X_test)[:, 1]
    row = evaluate_predictions(y_test, pred, prob)
    row["Model"] = name
    model_rows.append(row)
    fitted_models[name] = model

model_comparison = pd.DataFrame(model_rows)
model_comparison = model_comparison[
    ["Model", "Accuracy", "Precision", "Recall", "F1-score", "ROC-AUC", "Cohen Kappa"]
]
display(model_comparison.sort_values("Accuracy", ascending=False).round(4))
```

```python
# Compact GridSearchCV for Gradient Boosting.
# The grid is intentionally small because the exam instruction warns that
# GridSearchCV can impact system performance.
gb_pipeline = Pipeline(steps=[
    ("preprocess", tree_preprocess),
    ("model", GradientBoostingClassifier(random_state=RANDOM_STATE)),
])

gb_param_grid = {
    "model__n_estimators": [100, 150],
    "model__learning_rate": [0.05, 0.10],
    "model__max_depth": [2, 3],
    "model__min_samples_leaf": [1, 10],
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)

gb_grid = GridSearchCV(
    estimator=gb_pipeline,
    param_grid=gb_param_grid,
    scoring="accuracy",
    cv=cv,
    n_jobs=1,
)

gb_grid.fit(X_train, y_train)

print("Best Gradient Boosting parameters:")
print(gb_grid.best_params_)
print("Best cross-validation accuracy:", round(gb_grid.best_score_, 4))

tuned_gb = gb_grid.best_estimator_
tuned_pred = tuned_gb.predict(X_test)
tuned_prob = tuned_gb.predict_proba(X_test)[:, 1]
tuned_metrics = evaluate_predictions(y_test, tuned_pred, tuned_prob)

display(pd.DataFrame([{"Model": "Tuned Gradient Boosting", **tuned_metrics}]).round(4))
```

```python
# Choose final model.
# For the paper's wording, we select the model with best test accuracy.
# We still report F1 and recall because the target is imbalanced.
all_model_results = pd.concat([
    model_comparison,
    pd.DataFrame([{"Model": "Tuned Gradient Boosting", **tuned_metrics}]),
], ignore_index=True)

final_model_name = all_model_results.sort_values("Accuracy", ascending=False).iloc[0]["Model"]
print("Final model selected by accuracy:", final_model_name)

if final_model_name == "Tuned Gradient Boosting":
    final_model = tuned_gb
else:
    final_model = fitted_models[final_model_name]

final_pred = final_model.predict(X_test)
final_prob = final_model.predict_proba(X_test)[:, 1]
final_metrics = evaluate_predictions(y_test, final_pred, final_prob)

display(pd.DataFrame([{"Final Model": final_model_name, **final_metrics}]).round(4))

print("Final model confusion matrix:")
display(pd.DataFrame(
    confusion_matrix(y_test, final_pred),
    index=["Actual 0", "Actual 1"],
    columns=["Predicted 0", "Predicted 1"],
))

print("Final model classification report:")
print(classification_report(y_test, final_pred, zero_division=0))
```

```python
# Plot ROC curves for the base and final models.
base_fpr, base_tpr, _ = roc_curve(y_test, base_prob)
final_fpr, final_tpr, _ = roc_curve(y_test, final_prob)

plt.figure(figsize=(7, 5))
plt.plot(base_fpr, base_tpr, label=f"Base Logistic ROC-AUC = {roc_auc_score(y_test, base_prob):.3f}")
plt.plot(final_fpr, final_tpr, label=f"{final_model_name} ROC-AUC = {roc_auc_score(y_test, final_prob):.3f}")
plt.plot([0, 1], [0, 1], linestyle="--", color="gray", label="Random classifier")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve Comparison")
plt.legend()
plt.tight_layout()
plt.show()
```

### Inference for Question 3(a)

Based on the actual model comparison:

- Base Logistic Regression gives accuracy around **0.8057**.
- Gradient Boosting gives the best overall accuracy, around **0.8548**, and a strong ROC-AUC around **0.8452**.
- Random Forest gives stronger recall/F1 for the positive class, but lower overall accuracy.
- The compact Gradient Boosting grid search selects a simpler tuned model, but the default Gradient Boosting model performs best on the held-out test set by accuracy.
- Therefore, for the paper's wording of improving accuracy, the final selected model is **Gradient Boosting**.

Important improvement actions:

- Removed `CustomerId`.
- Imputed missing values.
- Encoded categorical variables.
- Used nonlinear ensemble models.
- Compared multiple metrics instead of accuracy alone.

**Exam-quality conclusion:** Accuracy improved over the base model, but because the target is imbalanced, the model should also be judged using recall, F1-score, ROC-AUC, and Cohen's Kappa.

### Question 3(b) - 20 marks

**Full question:**  
Summarize as follows:

1. Summarize the overall fit of the model and list down the measures to prove that it is a good model.
2. Write down a business interpretation/explanation of the model - which variables are affecting the target the most and explain the relationship. Feel free to use charts or graphs to explain.
3. What changes from the base model had the most effect on model performance?
4. What are the key risks to your results and interpretation?

### Approach

We summarize model fit using evaluation metrics and explain the model using feature importance.

```python
# Feature importance for the final model.
# Gradient Boosting and Random Forest expose feature_importances_.
final_feature_names = final_model.named_steps["preprocess"].get_feature_names_out()
final_importance_values = final_model.named_steps["model"].feature_importances_

feature_importance = (
    pd.DataFrame({
        "encoded_feature": final_feature_names,
        "importance": final_importance_values,
    })
    .sort_values("importance", ascending=False)
)

display(feature_importance.head(15).round(4))

plt.figure(figsize=(9, 6))
sns.barplot(
    data=feature_importance.head(12),
    y="encoded_feature",
    x="importance",
    color="#4E79A7",
)
plt.title("Top Feature Importances")
plt.xlabel("Importance")
plt.ylabel("Encoded feature")
plt.tight_layout()
plt.show()
```

```python
# Group one-hot encoded importances back to original feature names.
def recover_original_feature(encoded_feature):
    # Remove ColumnTransformer prefix such as "num__" or "cat__".
    clean_name = encoded_feature.split("__", 1)[1]

    if clean_name in model_numeric_cols:
        return clean_name

    for col in model_categorical_cols:
        if clean_name.startswith(col + "_"):
            return col

    return clean_name


feature_importance["original_feature"] = feature_importance["encoded_feature"].apply(recover_original_feature)
grouped_importance = (
    feature_importance
    .groupby("original_feature", as_index=False)["importance"]
    .sum()
    .sort_values("importance", ascending=False)
)

display(grouped_importance.round(4))

plt.figure(figsize=(8, 5))
sns.barplot(data=grouped_importance, y="original_feature", x="importance", color="#59A14F")
plt.title("Grouped Feature Importance")
plt.xlabel("Total importance")
plt.ylabel("Original feature")
plt.tight_layout()
plt.show()
```

```python
# Show important relationships as business-friendly target-rate plots.
important_categorical = ["NumOfProducts", "IsActiveMember", "Geography", "Gender"]

fig, axes = plt.subplots(2, 2, figsize=(13, 8))
axes = axes.ravel()

for ax, col in zip(axes, important_categorical):
    sns.barplot(data=df_clean, x=col, y=TARGET, estimator=np.mean, errorbar=None, ax=ax, color="#F28E2B")
    ax.set_title(f"Mean target rate by {col}")
    ax.set_ylabel("Mean target rate")

plt.tight_layout()
plt.show()
```

### Final business summary

#### 1. Overall fit of the model

The final Gradient Boosting model is a good accuracy-focused model because:

- Accuracy is around **0.8548**, which is higher than the base Logistic Regression accuracy of around **0.8057**.
- ROC-AUC is around **0.8452**, showing good separation between the two classes.
- Cohen's Kappa improves compared with the base model, showing better agreement beyond chance.
- The confusion matrix and classification report give class-wise evidence of model performance.

However, the positive class is a minority class, so recall and F1-score must also be monitored.

#### 2. Business interpretation

The most influential variables are:

- `NumOfProducts`: Strongest predictor in the final model. Product usage pattern is strongly related to the target.
- `Age`: Older and younger customers show different target behavior.
- `CreditScore`: Lower credit score is associated with higher risk patterns in the target.
- `IsActiveMember`: Active membership status is strongly related to the target.
- `Balance`: Account balance contributes to model decisions.
- `Geography`: Germany/France/Spain differences affect the target rate.
- `Gender`: Shows statistically significant association with the target.

#### 3. Changes that most improved performance

The biggest improvements came from:

- Moving from a linear model to nonlinear ensemble models.
- Using Gradient Boosting to capture interactions between variables.
- Dropping the identifier column `CustomerId`.
- Properly imputing missing values and encoding categorical variables.
- Evaluating multiple models instead of relying on a single baseline.

#### 4. Key risks

- The target is imbalanced, so high accuracy may hide weak positive-class recall.
- Feature importance does not prove causation.
- The dataset is small, so results may vary with different train-test splits.
- Missing-value imputation may introduce bias.
- The meaning of `Loan` in the CSV differs from the `Exited` wording in the PDF, so business interpretation must use the actual available target.
- A model used in banking should be checked for fairness, bias, and regulatory compliance before deployment.

#### Exam conclusion

The final model improves overall accuracy and ROC-AUC over the base model. For a real business deployment, the decision threshold should be chosen based on business cost: whether false positives or false negatives are more expensive.

## Final Revision Notes

- Precision reduces false-positive concern.
- Recall reduces false-negative concern.
- F1-score balances precision and recall.
- ROC-AUC measures ranking ability across thresholds.
- Gini and entropy are impurity measures for decision trees.
- Bagging reduces variance; boosting sequentially corrects errors.
- Random Forest is bagging with random feature selection.
- Gradient Boosting often improves accuracy by modeling residual errors.
- In imbalanced classification, never rely only on accuracy.
- Always avoid data leakage by fitting preprocessing steps only on training data.

# ML2_ESA_Sep21_Set2_Solved_Notebook.ipynb

# Machine Learning - II ESA September 2021: Solved Question Paper

**Course:** UE20CS908 - Machine Learning - II  
**Exam:** September 2021 End Semester Assessment  
**Maximum Marks:** 100  
**Dataset:** `Churn_Modelling(1).csv`

This notebook solves the complete paper in an exam-focused style. It includes the full question before every answer, theory explanations, formulas, Python code, plots, model evaluation, threshold analysis, and interpretations based on actual executed output.

## Problem context

The dataset contains details of bank customers. The target variable is `Exited`, where:

- `Exited = 1`: customer left the bank.
- `Exited = 0`: customer continued with the bank.

This is a binary classification problem with class imbalance, so accuracy must be interpreted together with precision, recall, F1-score, ROC-AUC, and Cohen's Kappa.

```python
# Confirm the exact Python interpreter used by this notebook.
import sys
print("Python executable:", sys.executable)
print("Python version:", sys.version.split()[0])
```

## Section A - Theory Questions (30 marks)

### Question 1(a) - 5 marks

**Full question:**  
Explain K Nearest Neighbor Algorithm with an example.

### Answer

K Nearest Neighbor (KNN) is a supervised machine learning algorithm used for classification and regression. It is called a lazy learning algorithm because it does not learn an explicit model during training. Instead, it stores the training data and makes predictions by comparing a new observation with nearby training observations.

#### Working of KNN for classification

1. Choose the value of $K$, the number of neighbors.
2. Calculate the distance between the new data point and all training points.
3. Select the $K$ nearest training points.
4. Count the class labels of those nearest neighbors.
5. Assign the majority class to the new point.

#### Common distance measure

For numerical features, Euclidean distance is commonly used:

$$
d(x, y) = \sqrt{\sum_{i=1}^{n}(x_i - y_i)^2}
$$

#### Example

Suppose we want to classify a customer as `Exited = 1` or `Exited = 0`. For a new customer, we compare their features such as age, balance, credit score, and activity status with existing customers. If $K = 5$ and among the 5 nearest customers, 3 exited and 2 did not exit, KNN predicts `Exited = 1`.

#### Choosing K

- Small K: sensitive to noise and may overfit.
- Large K: smoother boundary but may underfit.
- Odd K is often preferred in binary classification to avoid ties.

#### Advantages

- Simple and intuitive.
- No training phase.
- Can model nonlinear decision boundaries.

#### Limitations

- Prediction can be slow on large datasets.
- Sensitive to feature scaling.
- Sensitive to irrelevant features.
- Performance depends strongly on K and distance metric.

**Exam point:** Always scale features before KNN because distance-based algorithms are affected by feature magnitude.

### Question 1(b) - 5 marks

**Full question:**  
i) Explain F1 Score metric with formula.  
ii) Calculate Precision metric based on below confusion matrix.

| | Predicted 0 | Predicted 1 |
|---|---:|---:|
| Actual 0 | 56850 | 4 |
| Actual 1 | 30 | 78 |

### Answer

#### F1-score

F1-score is the harmonic mean of precision and recall.

$$
F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}
$$

Where:

$$
Precision = \frac{TP}{TP + FP}
$$

$$
Recall = \frac{TP}{TP + FN}
$$

F1-score is useful when the dataset is imbalanced and both false positives and false negatives matter.

#### From the given confusion matrix

Taking class `1` as the positive class:

- True Positive (TP) = 78
- False Positive (FP) = 4
- False Negative (FN) = 30
- True Negative (TN) = 56850

#### Precision calculation

$$
Precision = \frac{TP}{TP + FP}
$$

$$
Precision = \frac{78}{78 + 4} = \frac{78}{82} = 0.9512
$$

Therefore, precision is:

$$
\boxed{95.12\%}
$$

#### Extra interpretation

Out of all cases predicted as class 1, about 95.12% were actually class 1. This is a high precision value, meaning false positives are very low.

### Question 1(c) - 5 marks

**Full question:**  
Describe Ada boost and Gradient Boosting with differences and advantages.

### Answer

AdaBoost and Gradient Boosting are boosting ensemble algorithms. Boosting builds models sequentially, where each new model tries to correct errors made by previous models.

#### AdaBoost

AdaBoost stands for Adaptive Boosting.

Working:

1. Train a weak learner, usually a decision stump.
2. Increase weights of misclassified observations.
3. Train the next weak learner on the reweighted data.
4. Combine learners using weighted voting.

AdaBoost focuses more on observations that were incorrectly classified in earlier rounds.

#### Gradient Boosting

Gradient Boosting builds models sequentially by fitting each new model to the residual errors of the previous model.

For classification, it minimizes a differentiable loss function such as log loss.

Working:

1. Start with an initial prediction.
2. Compute residuals or negative gradients of the loss.
3. Fit a weak learner to those residuals.
4. Add the weak learner to the model with a learning rate.
5. Repeat until the required number of learners is reached.

#### Differences

| Aspect | AdaBoost | Gradient Boosting |
|---|---|---|
| Error correction | Reweights misclassified samples | Fits next learner to negative gradients |
| Loss function | Exponential loss | Flexible differentiable loss |
| Base learner | Often decision stump | Usually shallow decision trees |
| Sensitivity | Sensitive to noisy labels/outliers | Also sensitive, but tunable by learning rate/depth |
| Flexibility | Less flexible | More flexible |

#### Advantages

- Both improve weak learners into a strong learner.
- Often produce high predictive accuracy.
- Capture nonlinear relationships.
- Gradient Boosting supports flexible loss functions and hyperparameter tuning.

#### Limitations

- Can overfit if too many trees are used.
- More computationally expensive than a single model.
- Less interpretable than simple models.

### Question 1(d) - 5 marks

**Full question:**  
Explain Stacking Classifier and its advantages.

### Answer

Stacking, or stacked generalization, is an ensemble method that combines multiple base models using a second-level model called a meta-model.

#### Working

1. Train several base models, such as Logistic Regression, Decision Tree, Random Forest, and KNN.
2. Generate predictions or probabilities from these base models.
3. Use the base-model outputs as new features.
4. Train a meta-model on these new features.
5. The meta-model learns how to combine the base model predictions.

#### Example

For bank churn prediction:

- Logistic Regression may capture linear relationships.
- Random Forest may capture nonlinear interactions.
- Gradient Boosting may capture complex sequential error corrections.
- A meta-model combines their predictions to produce the final churn prediction.

#### Advantages

- Combines strengths of different algorithms.
- Can improve prediction performance.
- Reduces reliance on one model.
- Learns the best way to combine model outputs.

#### Important caution

Stacking must use cross-validation or out-of-fold predictions to train the meta-model. If the same training predictions are reused directly, data leakage can occur and performance will look artificially high.

### Question 1(e) - 5 marks

**Full question:**  
Describe working of Decision tree with an example.

### Answer

A decision tree is a supervised learning algorithm that splits data into smaller groups using feature-based rules. It can be used for classification and regression.

#### Working

1. Start with the full dataset at the root node.
2. Select the best feature and split point using impurity measures such as Gini impurity or entropy.
3. Split the data into child nodes.
4. Repeat splitting until a stopping condition is met.
5. Assign a class label to each leaf node.

#### Gini impurity

$$
Gini = 1 - \sum p_i^2
$$

#### Entropy

$$
Entropy = -\sum p_i \log_2(p_i)
$$

#### Example

For bank churn prediction, a tree may learn rules such as:

- If `Age > 45` and `IsActiveMember = 0`, churn risk is high.
- If `NumOfProducts = 2` and customer is active, churn risk is low.
- If `Geography = Germany` and balance is high, churn risk may be higher.

#### Advantages

- Easy to understand and visualize.
- Handles nonlinear relationships.
- Requires little preprocessing.
- Can model feature interactions.

#### Limitations

- Can overfit easily.
- Small data changes can create a different tree.
- A single tree may have high variance.

**Exam point:** Pruning, limiting depth, and setting minimum samples per leaf help reduce overfitting.

### Question 1(f) - 5 marks

**Full question:**  
Compare Decision tree and Random Forest.

### Answer

| Aspect | Decision Tree | Random Forest |
|---|---|---|
| Model type | Single tree | Ensemble of many trees |
| Training data | Uses full training data | Uses bootstrap samples |
| Feature selection | Considers all features at split | Considers random subset of features |
| Variance | High variance | Lower variance |
| Overfitting | More prone to overfitting | Less prone due to averaging |
| Interpretability | High | Lower than single tree |
| Accuracy | Moderate | Usually higher |
| Feature importance | Available | More stable importance |
| Prediction | One tree prediction | Majority vote across trees |

#### Decision Tree

A decision tree is easy to interpret but can overfit if it grows too deep.

#### Random Forest

Random Forest reduces overfitting by building many decision trees on bootstrap samples and combining their predictions using majority voting.

#### Exam conclusion

Decision Tree is simple and interpretable, while Random Forest is more accurate and robust because it combines many trees and reduces variance.

## Section B - EDA and Preprocessing (30 marks)

### Imports and setup

```python
import warnings
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from scipy.stats import chi2_contingency, pointbiserialr, zscore

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    cohen_kappa_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.tree import DecisionTreeClassifier

warnings.filterwarnings("ignore")
sns.set_theme(style="whitegrid", context="notebook")
plt.rcParams["figure.figsize"] = (8, 5)
plt.rcParams["axes.titlesize"] = 12

RANDOM_STATE = 42
TARGET = "Exited"
```

### Question 2(i) - 5 marks

**Full question:**  
Read the dataset and print the following:

- Shape of the data.
- Number of numerical and categorical variable.
- Descriptive stats of numerical data.
- Descriptive stats of categorical data.

### Approach

We inspect the shape, first records, data types, numerical summary, and categorical summary. For `Surname`, only the top categories are shown because it is a high-cardinality identifier-like text column.

```python
# Read the dataset.
data_path = Path("Churn_Modelling(1).csv")
df = pd.read_csv(data_path)

print("Dataset shape:", df.shape)
print("\nColumn names:")
print(df.columns.tolist())

display(df.head())

print("\nData information:")
df.info()
```

```python
# Separate numerical and categorical columns.
numeric_cols = df.select_dtypes(include=np.number).columns.tolist()
categorical_cols = df.select_dtypes(exclude=np.number).columns.tolist()

print("Number of numerical columns:", len(numeric_cols))
print("Numerical columns:", numeric_cols)
print("\nNumber of categorical columns:", len(categorical_cols))
print("Categorical columns:", categorical_cols)
```

```python
# Descriptive statistics for numerical data.
numeric_summary = df[numeric_cols].describe().T
display(numeric_summary)
```

```python
# Descriptive statistics for categorical data.
categorical_summary = df[categorical_cols].describe().T
display(categorical_summary)

# Show compact category proportions.
for col in categorical_cols:
    print(f"\nCategory proportions for {col}:")
    temp = (
        df[col]
        .value_counts(dropna=False)
        .rename_axis(col)
        .reset_index(name="count")
    )
    temp["proportion"] = (temp["count"] / len(df)).round(4)

    if col == "Surname":
        print("Showing top 10 because Surname has high cardinality.")
        display(temp.head(10))
    else:
        display(temp)
```

### Inference for Question 2(i)

Based on the actual output:

- The dataset has **10000 rows and 14 columns**.
- There are **11 numerical columns**, including identifiers and the target.
- There are **3 categorical columns**: `Surname`, `Geography`, and `Gender`.
- `Surname` has very high cardinality and behaves like an identifier-like text field, so it should not be used directly for modeling.
- `Geography` has three categories: France, Germany, and Spain. France is the largest group.
- `Gender` has two categories: Male and Female.
- The target `Exited` is binary.

### Question 2(ii) - 5 marks

**Full question:**  
Perform appropriate encoding on `Geography` and `Gender` column.

### Approach

`Geography` and `Gender` are nominal categorical variables. One-hot encoding is appropriate because it does not impose false numerical order.

```python
# Demonstration of one-hot encoding for Geography and Gender.
encoded_demo = pd.get_dummies(
    df,
    columns=["Geography", "Gender"],
    drop_first=False,
    dtype=int,
)

print("Original shape:", df.shape)
print("Shape after encoding Geography and Gender:", encoded_demo.shape)
print("\nNew encoded columns:")
print([c for c in encoded_demo.columns if c.startswith("Geography_") or c.startswith("Gender_")])

display(encoded_demo.head())
```

### Inference for Question 2(ii)

One-hot encoding creates binary indicator variables such as `Geography_France`, `Geography_Germany`, `Geography_Spain`, `Gender_Female`, and `Gender_Male`.

In the final modeling pipeline, encoding is fitted only on the training data to prevent data leakage.

### Question 2(iii) - 5 marks

**Full question:**  
Examine outliers by plotting and also by using z score. Examine is the Target variable evenly balanced.

### Approach

We use boxplots, IQR outlier counts, z-score counts with threshold $|z| > 3$, and a target class distribution plot.

```python
# IQR outlier summary for numeric predictors.
outlier_rows = []
numeric_predictors_for_outliers = [c for c in numeric_cols if c not in [TARGET, "RowNumber", "CustomerId"]]

for col in numeric_predictors_for_outliers:
    q1 = df[col].quantile(0.25)
    q3 = df[col].quantile(0.75)
    iqr = q3 - q1
    lower = q1 - 1.5 * iqr
    upper = q3 + 1.5 * iqr
    mask = (df[col] < lower) | (df[col] > upper)
    outlier_rows.append({
        "feature": col,
        "lower_bound": round(lower, 3),
        "upper_bound": round(upper, 3),
        "iqr_outlier_count": int(mask.sum()),
        "iqr_outlier_percent": round(mask.mean() * 100, 2),
    })

outlier_iqr_table = pd.DataFrame(outlier_rows)
display(outlier_iqr_table)
```

```python
# Z-score based outlier summary.
zscore_rows = []

for col in numeric_predictors_for_outliers:
    z_values = np.abs(zscore(df[col].astype(float), nan_policy="omit"))
    zscore_rows.append({
        "feature": col,
        "count_abs_z_gt_3": int(np.nansum(z_values > 3)),
        "percent_abs_z_gt_3": round(np.nanmean(z_values > 3) * 100, 2),
    })

zscore_outlier_table = pd.DataFrame(zscore_rows)
display(zscore_outlier_table)
```

```python
# Boxplots for visual outlier inspection.
fig, axes = plt.subplots(2, 4, figsize=(16, 8))
axes = axes.ravel()

for ax, col in zip(axes, numeric_predictors_for_outliers):
    sns.boxplot(y=df[col], ax=ax, color="#72B7B2")
    ax.set_title(col)
    ax.set_xlabel("")

plt.tight_layout()
plt.show()
```

```python
# Target balance.
target_counts = df[TARGET].value_counts().sort_index()
target_proportions = df[TARGET].value_counts(normalize=True).sort_index()

target_balance = pd.DataFrame({
    "class_label": target_counts.index,
    "count": target_counts.values,
    "proportion": target_proportions.values.round(4),
})
display(target_balance)

plt.figure(figsize=(6, 4))
sns.countplot(data=df, x=TARGET, palette=["#59A14F", "#E15759"])
plt.title("Target Class Distribution")
plt.xlabel("Exited")
plt.ylabel("Count")
plt.tight_layout()
plt.show()
```

### Inference for Question 2(iii)

Based on the actual output:

- IQR detects outliers in `CreditScore`, `Age`, and `NumOfProducts`.
- Z-score detects strongest outliers mainly in `Age`, `NumOfProducts`, and a few `CreditScore` values.
- These values are realistic business values, so they are not automatically removed.
- `NumOfProducts = 4` is rare but valid, so it is retained.
- The target is **not evenly balanced**: about **79.63%** customers did not exit and **20.37%** exited.
- Because of imbalance, accuracy alone can be misleading. We must use recall, F1-score, ROC-AUC, and Cohen's Kappa.

### Question 2(iv) - 6 marks

**Full question:**  
Check for defects in the data like missing values, removing unnecessary features/columns. Perform necessary actions to "fix" these defects.

### Approach

We check missing values, duplicates, unnecessary columns, and prepare a clean modeling dataset.

```python
# Missing values and duplicates.
missing_table = pd.DataFrame({
    "missing_count": df.isna().sum(),
    "missing_percent": (df.isna().sum() / len(df) * 100).round(2),
}).sort_values("missing_count", ascending=False)

display(missing_table)
print("Duplicate rows:", df.duplicated().sum())
```

```python
# Clean the dataset.
df_clean = df.copy()

# Remove identifier/unnecessary columns.
# RowNumber and CustomerId are IDs. Surname is high-cardinality text and not useful directly.
columns_to_drop = ["RowNumber", "CustomerId", "Surname"]
df_clean = df_clean.drop(columns=columns_to_drop)

print("Dropped columns:", columns_to_drop)
print("Cleaned dataset shape:", df_clean.shape)
print("\nMissing values after cleaning:")
display(df_clean.isna().sum().to_frame("missing_after_cleaning"))
```

### Inference for Question 2(iv)

Based on the actual output:

- There are **no missing values** in the dataset.
- There are **no duplicate rows**.
- `RowNumber` and `CustomerId` are removed because they are identifiers.
- `Surname` is removed because it is high-cardinality text and can cause overfitting if one-hot encoded directly.
- No outlier row removal is performed because the detected values are plausible customer records.

### Question 2(v) - 6 marks

**Full question:**  
Examine the correlation and summarize the relationship between variables. Use appropriate plots to justify the same.

### Approach

We use:

- Correlation heatmap for numerical variables.
- Point-biserial correlation for continuous variables with binary target.
- Chi-square tests for categorical/discrete variables.
- Target-rate plots for business interpretation.

```python
# Correlation heatmap for numerical variables in cleaned data.
plt.figure(figsize=(9, 6))
numeric_corr = df_clean.corr(numeric_only=True)
sns.heatmap(numeric_corr, annot=True, cmap="RdBu_r", center=0, fmt=".2f")
plt.title("Correlation Heatmap")
plt.tight_layout()
plt.show()

display(numeric_corr[TARGET].sort_values(key=lambda s: s.abs(), ascending=False).to_frame("correlation_with_target"))
```

```python
# Statistical tests.
chi_features = ["Geography", "Gender", "HasCrCard", "IsActiveMember", "NumOfProducts"]
chi_rows = []

for col in chi_features:
    contingency = pd.crosstab(df_clean[col], df_clean[TARGET])
    chi2, p_value, dof, expected = chi2_contingency(contingency)
    chi_rows.append({
        "feature": col,
        "test": "chi-square",
        "p_value": p_value,
        "significant_at_0.05": p_value < 0.05,
    })

chi_results = pd.DataFrame(chi_rows).sort_values("p_value")
display(chi_results)

continuous_features = ["CreditScore", "Age", "Tenure", "Balance", "EstimatedSalary"]
pb_rows = []

for col in continuous_features:
    corr, p_value = pointbiserialr(df_clean[col], df_clean[TARGET])
    pb_rows.append({
        "feature": col,
        "point_biserial_corr": corr,
        "p_value": p_value,
        "significant_at_0.05": p_value < 0.05,
    })

pb_results = pd.DataFrame(pb_rows).sort_values("p_value")
display(pb_results)
```

```python
# Target-rate plots for important categorical/discrete variables.
plot_features = ["Geography", "Gender", "NumOfProducts", "IsActiveMember"]

fig, axes = plt.subplots(2, 2, figsize=(13, 8))
axes = axes.ravel()

for ax, col in zip(axes, plot_features):
    sns.barplot(data=df_clean, x=col, y=TARGET, estimator=np.mean, errorbar=None, ax=ax, color="#F28E2B")
    ax.set_title(f"Mean exit rate by {col}")
    ax.set_ylabel("Mean exit rate")

plt.tight_layout()
plt.show()
```

### Inference for Question 2(v)

Based on the actual output:

- `Age` has the strongest positive numerical correlation with `Exited`.
- `Balance` also has a positive relationship with churn.
- `CreditScore` has a weak but statistically significant negative relationship with churn.
- `Tenure` and `EstimatedSalary` do not show significant linear association at the 5% level.
- Chi-square tests show `Geography`, `Gender`, `IsActiveMember`, and `NumOfProducts` are significantly associated with churn.
- `HasCrCard` is not statistically significant at the 5% level.
- Customers from Germany, inactive customers, and customers with certain product-count patterns show higher exit rates.

### Question 2(vi) - 3 marks

**Full question:**  
Split dataset into train and test (70:30).

### Approach

We use stratified sampling so the imbalanced target distribution is preserved in both train and test sets.

```python
# Split features and target.
X = df_clean.drop(columns=[TARGET])
y = df_clean[TARGET]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.30,
    random_state=RANDOM_STATE,
    stratify=y,
)

print("X_train shape:", X_train.shape)
print("X_test shape:", X_test.shape)
print("y_train shape:", y_train.shape)
print("y_test shape:", y_test.shape)

split_balance = pd.DataFrame({
    "overall": y.value_counts(normalize=True).sort_index(),
    "train": y_train.value_counts(normalize=True).sort_index(),
    "test": y_test.value_counts(normalize=True).sort_index(),
})
display(split_balance)
```

### Inference for Question 2(vi)

The train-test split is 70:30. The class proportions in train and test are almost identical to the overall dataset because stratified splitting was used.

## Section C - Model Building, Threshold Analysis, and Summary (40 marks)

### Common preprocessing and metric functions

```python
model_numeric_cols = X.select_dtypes(include=np.number).columns.tolist()
model_categorical_cols = X.select_dtypes(exclude=np.number).columns.tolist()

print("Numeric model columns:", model_numeric_cols)
print("Categorical model columns:", model_categorical_cols)


def make_one_hot_encoder():
    # Create encoder compatible with different sklearn versions.
    try:
        return OneHotEncoder(handle_unknown="ignore", sparse_output=False)
    except TypeError:
        return OneHotEncoder(handle_unknown="ignore", sparse=False)


logistic_preprocess = ColumnTransformer(
    transformers=[
        ("num", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="median")),
            ("scaler", StandardScaler()),
        ]), model_numeric_cols),
        ("cat", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("onehot", make_one_hot_encoder()),
        ]), model_categorical_cols),
    ]
)

tree_preprocess = ColumnTransformer(
    transformers=[
        ("num", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="median")),
        ]), model_numeric_cols),
        ("cat", Pipeline(steps=[
            ("imputer", SimpleImputer(strategy="most_frequent")),
            ("onehot", make_one_hot_encoder()),
        ]), model_categorical_cols),
    ]
)


def evaluate_predictions(y_true, y_pred, y_prob):
    # Return all important metrics for imbalanced classification.
    return {
        "Accuracy": accuracy_score(y_true, y_pred),
        "Precision": precision_score(y_true, y_pred, zero_division=0),
        "Recall": recall_score(y_true, y_pred, zero_division=0),
        "F1-score": f1_score(y_true, y_pred, zero_division=0),
        "ROC-AUC": roc_auc_score(y_true, y_prob),
        "Cohen Kappa": cohen_kappa_score(y_true, y_pred),
    }
```

### Question 3(i) - 15 marks

**Full question:**  
Fit a base model and explain the reason of selecting that model. Please write your key observations. Calculate Cohen Kappa value with the best model achieved.

### Base model choice

Logistic Regression is selected as the base model because it is simple, interpretable, standard for binary classification, and gives probability outputs.

```python
# Base Logistic Regression model.
base_logistic = Pipeline(steps=[
    ("preprocess", logistic_preprocess),
    ("model", LogisticRegression(max_iter=2000, random_state=RANDOM_STATE)),
])

base_logistic.fit(X_train, y_train)

base_prob = base_logistic.predict_proba(X_test)[:, 1]
base_pred = base_logistic.predict(X_test)
base_metrics = evaluate_predictions(y_test, base_pred, base_prob)

display(pd.DataFrame([base_metrics]).round(4))

print("Base model confusion matrix:")
display(pd.DataFrame(
    confusion_matrix(y_test, base_pred),
    index=["Actual 0", "Actual 1"],
    columns=["Predicted 0", "Predicted 1"],
))

print("Base model classification report:")
print(classification_report(y_test, base_pred, zero_division=0))
```

### Inference for Question 3(i)

Based on the actual base Logistic Regression output:

- Accuracy is around **0.8127**, but this is partly due to the majority class.
- Positive-class recall is low, around **0.1964**, meaning many churned customers are missed.
- F1-score is around **0.2993**, so the base model is weak for the minority class.
- ROC-AUC is around **0.7880**, which shows moderate ranking ability.
- Cohen's Kappa is around **0.2240**, indicating only fair agreement beyond chance.

Therefore, the base model is useful as a benchmark, but it needs improvement for churn detection.

### Question 3(ii) - 15 marks

**Full question:**  
How do you improve the accuracy of the model? Write clearly the changes that you will make before re-fitting the model. Fit the final model.

### Improvement strategy

We improve the model by:

1. Using nonlinear models such as Decision Tree, Random Forest, and Gradient Boosting.
2. Using class-weighted models where appropriate because the target is imbalanced.
3. Using a compact GridSearchCV for Gradient Boosting.
4. Comparing models using accuracy, precision, recall, F1-score, ROC-AUC, and Cohen's Kappa.

```python
# Candidate models.
candidate_models = {
    "Base Logistic Regression": base_logistic,
    "Balanced Logistic Regression": Pipeline(steps=[
        ("preprocess", logistic_preprocess),
        ("model", LogisticRegression(
            max_iter=2000,
            class_weight="balanced",
            random_state=RANDOM_STATE,
        )),
    ]),
    "Decision Tree": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", DecisionTreeClassifier(
            max_depth=5,
            min_samples_leaf=30,
            class_weight="balanced",
            random_state=RANDOM_STATE,
        )),
    ]),
    "Random Forest": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", RandomForestClassifier(
            n_estimators=300,
            max_depth=10,
            min_samples_leaf=10,
            class_weight="balanced",
            random_state=RANDOM_STATE,
            n_jobs=1,
        )),
    ]),
    "Gradient Boosting": Pipeline(steps=[
        ("preprocess", tree_preprocess),
        ("model", GradientBoostingClassifier(random_state=RANDOM_STATE)),
    ]),
}

model_rows = []
fitted_models = {}

for name, model in candidate_models.items():
    model.fit(X_train, y_train)
    pred = model.predict(X_test)
    prob = model.predict_proba(X_test)[:, 1]
    row = evaluate_predictions(y_test, pred, prob)
    row["Model"] = name
    model_rows.append(row)
    fitted_models[name] = model

model_comparison = pd.DataFrame(model_rows)
model_comparison = model_comparison[
    ["Model", "Accuracy", "Precision", "Recall", "F1-score", "ROC-AUC", "Cohen Kappa"]
]
display(model_comparison.sort_values("Accuracy", ascending=False).round(4))
```

```python
# Compact GridSearchCV for Gradient Boosting.
# The grid is small because the paper explicitly warns that GridSearchCV
# can impact system performance.
gb_pipeline = Pipeline(steps=[
    ("preprocess", tree_preprocess),
    ("model", GradientBoostingClassifier(random_state=RANDOM_STATE)),
])

gb_param_grid = {
    "model__n_estimators": [80, 100, 150],
    "model__learning_rate": [0.05, 0.10],
    "model__max_depth": [2, 3],
    "model__min_samples_leaf": [1, 20],
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)

gb_grid = GridSearchCV(
    estimator=gb_pipeline,
    param_grid=gb_param_grid,
    scoring="accuracy",
    cv=cv,
    n_jobs=1,
)

gb_grid.fit(X_train, y_train)

print("Best Gradient Boosting parameters:")
print(gb_grid.best_params_)
print("Best cross-validation accuracy:", round(gb_grid.best_score_, 4))

tuned_gb = gb_grid.best_estimator_
tuned_pred = tuned_gb.predict(X_test)
tuned_prob = tuned_gb.predict_proba(X_test)[:, 1]
tuned_metrics = evaluate_predictions(y_test, tuned_pred, tuned_prob)

display(pd.DataFrame([{"Model": "Tuned Gradient Boosting", **tuned_metrics}]).round(4))
```

```python
# Select final model by held-out test accuracy because the question asks
# specifically about improving accuracy.
all_model_results = pd.concat([
    model_comparison,
    pd.DataFrame([{"Model": "Tuned Gradient Boosting", **tuned_metrics}]),
], ignore_index=True)

final_model_name = all_model_results.sort_values("Accuracy", ascending=False).iloc[0]["Model"]
print("Final model selected by accuracy:", final_model_name)

if final_model_name == "Tuned Gradient Boosting":
    final_model = tuned_gb
else:
    final_model = fitted_models[final_model_name]

final_pred = final_model.predict(X_test)
final_prob = final_model.predict_proba(X_test)[:, 1]
final_metrics = evaluate_predictions(y_test, final_pred, final_prob)

display(pd.DataFrame([{"Final Model": final_model_name, **final_metrics}]).round(4))

print("Final model confusion matrix:")
display(pd.DataFrame(
    confusion_matrix(y_test, final_pred),
    index=["Actual 0", "Actual 1"],
    columns=["Predicted 0", "Predicted 1"],
))

print("Final model classification report:")
print(classification_report(y_test, final_pred, zero_division=0))
```

### Inference for Question 3(ii)

Based on the actual output:

- Base Logistic Regression accuracy is around **0.8127**.
- Gradient Boosting gives the best held-out accuracy, around **0.8730**.
- The tuned Gradient Boosting model gives similar performance, with high ROC-AUC, but the default Gradient Boosting model has the best held-out accuracy in this run.
- Random Forest gives the best positive-class F1-score among the compared untuned models, but lower accuracy than Gradient Boosting.
- Cohen's Kappa improves substantially from the base model, showing better agreement beyond chance.

**Exam conclusion:** Gradient Boosting is selected as the final accuracy-focused model, while Random Forest is a strong alternative when minority-class recall/F1 is prioritized.

### Question 3(ii continued) - Threshold analysis, classification report, and probability threshold effect

**Full question from starter notebook:**  
Print the classification report of the model built in the above step and check that increasing or reducing the probability threshold will lead to model's better performance.

### Approach

We evaluate different probability thresholds. Lower threshold usually increases recall, while higher threshold usually increases precision.

```python
# Threshold analysis on the final model.
threshold_rows = []

for threshold in np.arange(0.10, 0.91, 0.05):
    pred_at_threshold = (final_prob >= threshold).astype(int)
    row = evaluate_predictions(y_test, pred_at_threshold, final_prob)
    row["Threshold"] = round(float(threshold), 2)
    threshold_rows.append(row)

threshold_df = pd.DataFrame(threshold_rows)
threshold_df = threshold_df[["Threshold", "Accuracy", "Precision", "Recall", "F1-score", "ROC-AUC", "Cohen Kappa"]]
display(threshold_df.round(4))

best_threshold_row = threshold_df.sort_values(["F1-score", "Recall"], ascending=False).iloc[0]
print("Best threshold based on F1-score:")
display(best_threshold_row.to_frame().T.round(4))
```

```python
# Plot precision, recall, and F1 against threshold.
plt.figure(figsize=(8, 5))
plt.plot(threshold_df["Threshold"], threshold_df["Precision"], marker="o", label="Precision")
plt.plot(threshold_df["Threshold"], threshold_df["Recall"], marker="o", label="Recall")
plt.plot(threshold_df["Threshold"], threshold_df["F1-score"], marker="o", label="F1-score")
plt.axvline(best_threshold_row["Threshold"], color="black", linestyle="--", label="Best F1 threshold")
plt.title("Threshold vs Classification Metrics")
plt.xlabel("Probability threshold")
plt.ylabel("Metric value")
plt.legend()
plt.tight_layout()
plt.show()
```

```python
# ROC curve comparison: base model vs final model.
base_fpr, base_tpr, _ = roc_curve(y_test, base_prob)
final_fpr, final_tpr, _ = roc_curve(y_test, final_prob)

plt.figure(figsize=(7, 5))
plt.plot(base_fpr, base_tpr, label=f"Base Logistic ROC-AUC = {roc_auc_score(y_test, base_prob):.3f}")
plt.plot(final_fpr, final_tpr, label=f"{final_model_name} ROC-AUC = {roc_auc_score(y_test, final_prob):.3f}")
plt.plot([0, 1], [0, 1], linestyle="--", color="gray", label="Random classifier")
plt.title("ROC Curve Comparison")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.legend()
plt.tight_layout()
plt.show()
```

### Threshold inference

The default threshold of 0.50 is not always optimal for an imbalanced churn problem.

- Reducing the threshold increases recall, so the model catches more churned customers.
- Increasing the threshold increases precision, so fewer non-churn customers are wrongly flagged.
- The best threshold depends on business cost.
- If the bank wants to prevent churn aggressively, a lower threshold may be preferred.
- If retention campaigns are expensive, a higher threshold may be preferred.

### Question 3(iii) - 10 marks

**Full question:**  
Summarize as follows:

1. With respect to features.
2. Evaluation metrics.
3. Overall Results and Observations.

```python
# Feature importance from the final model.
final_feature_names = final_model.named_steps["preprocess"].get_feature_names_out()
final_importance_values = final_model.named_steps["model"].feature_importances_

feature_importance = (
    pd.DataFrame({
        "encoded_feature": final_feature_names,
        "importance": final_importance_values,
    })
    .sort_values("importance", ascending=False)
)

display(feature_importance.head(15).round(4))

plt.figure(figsize=(9, 6))
sns.barplot(
    data=feature_importance.head(12),
    y="encoded_feature",
    x="importance",
    color="#4E79A7",
)
plt.title("Top Feature Importances")
plt.xlabel("Importance")
plt.ylabel("Encoded feature")
plt.tight_layout()
plt.show()
```

```python
# Group encoded feature importances back to original features.
def recover_original_feature(encoded_feature):
    # Remove prefixes such as "num__" and "cat__".
    clean_name = encoded_feature.split("__", 1)[1]

    if clean_name in model_numeric_cols:
        return clean_name

    for col in model_categorical_cols:
        if clean_name.startswith(col + "_"):
            return col

    return clean_name


feature_importance["original_feature"] = feature_importance["encoded_feature"].apply(recover_original_feature)
grouped_importance = (
    feature_importance
    .groupby("original_feature", as_index=False)["importance"]
    .sum()
    .sort_values("importance", ascending=False)
)

display(grouped_importance.round(4))

plt.figure(figsize=(8, 5))
sns.barplot(data=grouped_importance, y="original_feature", x="importance", color="#59A14F")
plt.title("Grouped Feature Importance")
plt.xlabel("Total importance")
plt.ylabel("Original feature")
plt.tight_layout()
plt.show()
```

### Final summary

#### 1. With respect to features

The most important features in the final model are:

- `Age`: Older customers show higher churn tendency.
- `NumOfProducts`: Product-count pattern is highly influential.
- `IsActiveMember`: Inactive customers have higher churn risk.
- `Balance`: Balance contributes meaningfully to prediction.
- `Geography`: Especially Germany, which shows higher churn tendency.
- `CreditScore`: Has a weaker but still useful contribution.

`RowNumber`, `CustomerId`, and `Surname` were removed because they are identifier-like and can create overfitting.

#### 2. Evaluation metrics

The final Gradient Boosting model improves accuracy from the base model's approximately **0.8127** to about **0.8730**. ROC-AUC also improves from about **0.7880** to about **0.8820**, showing better class separation. Cohen's Kappa improves as well, meaning the model agrees with actual labels more than would be expected by chance.

Because the target is imbalanced, recall and F1-score must be checked along with accuracy.

#### 3. Overall results and observations

- The dataset is an imbalanced binary churn classification problem.
- Base Logistic Regression is interpretable but weak for identifying churners.
- Gradient Boosting gives the best accuracy.
- Random Forest gives strong minority-class recall/F1 and is a strong alternative when the business wants to catch more churners.
- Threshold tuning is important because the best threshold depends on business cost.
- Feature importance suggests that age, product usage, activity status, balance, and geography are key churn drivers.

#### Business interpretation

The bank should pay special attention to older customers, inactive customers, customers with risky product-count patterns, and customers from geographies with higher churn rates. The model can be used to prioritize retention campaigns, but final deployment should include fairness checks, cost-sensitive threshold selection, and validation on recent data.

## Final Revision Notes

- KNN is distance-based and needs feature scaling.
- Precision = TP / (TP + FP).
- F1-score balances precision and recall.
- AdaBoost reweights mistakes; Gradient Boosting fits residual/gradient errors.
- Stacking uses base models plus a meta-model.
- Decision Trees are interpretable but overfit easily.
- Random Forest reduces tree variance using bagging and random feature selection.
- In churn prediction, accuracy alone is misleading because the churn class is the minority.
- Cohen's Kappa measures agreement beyond chance.
- Threshold tuning changes the precision-recall trade-off.

# ML2_All_Section_A_Theory_Quick_Revision.pdf

MACHINE LEARNING - II
All Section A Theory
Quick Revision Questions & Answers
Every theory question from three ESA papers, followed by a syllabus-based
likely-question bank. Answers are deliberately short, pointwise, and exam-ready.
3
17
43
PAST PAPERS
LETTERED QUESTIONS
ADDITIONAL LIKELY
QUESTIONS
Prepared from the ML2 course folder | 02 July 2026


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 2
How to use this revision pack
1
Past papers
Revise Part I first. These are the exact Section A questions found in
the available papers.
2
Likely
questions
Use Part II to cover direct syllabus gaps and common follow-up
questions.
3
Recall practice
Hide the answer, speak 3-4 bullets, then check the formula or key
contrast.
4
Exam writing
For 4-5 marks: definition + formula/steps + one advantage/use + one
caution.
Coverage
•
September 2021 Set 2: KNN, F1/precision calculation, boosting, stacking, Decision Tree, Random
Forest.
•
December 2021 Set 1: metrics, bagging vs boosting, ROC, impurity, Random Forest, ensembles.
•
February 2025: metrics, Gaussian Naive Bayes, stacking/voting, bagging/Random Forest,
cross-validation.
•
Additional bank: classification, Logistic Regression, imbalance, model selection, preprocessing, and
deployment.
Study priority is inferred from past-paper recurrence and the official course outline. It is not a
prediction guarantee.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 3
PART I - ALL PAST-PAPER
SECTION A QUESTIONS
Exact question coverage from every paper
in the destination folder. Answers are
condensed for rapid recall.
September 2021 - Set 2
UE20CS908 | Section A: 30 marks | 6 lettered questions
1(a). Explain K-Nearest Neighbor (KNN) algorithm with an example.
ANSWER
•
KNN is a supervised, distance-based, lazy-learning classifier.
•
Steps: scale features -> choose K -> compute distances -> select K nearest points ->
use majority class.
•
Example: if 3 nearest patients have labels [Diabetic, Diabetic, Healthy], predict Diabetic.
•
Small K may overfit; large K may underfit. Choose K using cross-validation.
1(b)(i). Explain F1-score metric with formula.
ANSWER
•
F1 is the harmonic mean of precision and recall.
•
Formula: F1 = 2 x (Precision x Recall) / (Precision + Recall).
•
It is high only when both precision and recall are high.
•
Useful when classes are imbalanced or FP and FN both matter.
1(b)(ii). Calculate precision for: TN=56850, FP=4, FN=30, TP=78.
ANSWER
•
Precision = TP / (TP + FP).
•
Precision = 78 / (78 + 4) = 78 / 82 = 0.9512.
•
Answer: 95.12%.
•
Meaning: about 95 of every 100 predicted positives are truly positive.
1(c). Describe AdaBoost and Gradient Boosting, with differences and advantages.
ANSWER
•
AdaBoost: sequentially increases weight on misclassified samples; combines weak
learners by weighted vote.
•
Gradient Boosting: each new learner fits residuals/negative gradients to reduce a
chosen loss.
•
Difference: AdaBoost reweights observations; Gradient Boosting directly optimizes a
differentiable loss.
•
Advantages: both convert weak learners into a strong nonlinear model; Gradient
Boosting is more flexible.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 4
1(d). Explain Stacking Classifier and its advantages.
ANSWER
•
Train diverse base models; obtain out-of-fold predictions from each.
•
Use those predictions as inputs to a meta-model, which learns the final combination.
•
Advantages: combines different model strengths and can outperform any single model.
•
Use out-of-fold predictions to prevent data leakage and overfitting.
1(e). Describe the working of a Decision Tree with an example.
ANSWER
•
A tree recursively splits data using feature rules that increase class purity.
•
Choose the best split using Gini decrease or information gain; stop by a rule; assign a
class at each leaf.
•
Example: loan approval -> CreditScore > 700? -> Income > threshold? ->
Approve/Reject.
•
Trees are interpretable but deep trees may overfit; control depth or prune.
1(f). Compare Decision Tree and Random Forest.
ANSWER
•
Decision Tree: one tree, easy to interpret, high variance, and prone to overfitting.
•
Random Forest: many trees built on bootstrap samples with random feature subsets.
•
Prediction: one leaf decision vs majority vote/average probability across trees.
•
Random Forest is more stable and accurate, but less interpretable and more
computationally costly.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 5
December 2021 - Set 1
UE20CS931 | Section A: 30 marks | 6 questions
1(a). Define Precision and Recall.
ANSWER
•
Precision = TP / (TP + FP): correctness of positive predictions.
•
Recall = TP / (TP + FN): proportion of actual positives detected.
•
Prefer precision when false positives are costly; prefer recall when false negatives are
costly.
•
There is usually a precision-recall trade-off when the classification threshold changes.
1(b). Differentiate between Bagging and Boosting.
ANSWER
•
Bagging trains learners independently, usually on bootstrap samples; Boosting trains
them sequentially.
•
Bagging mainly reduces variance; Boosting mainly reduces bias and may also reduce
variance.
•
Bagging gives equal/average votes; Boosting emphasizes previous errors and uses an
additive combination.
•
Examples: Random Forest (bagging); AdaBoost and Gradient Boosting (boosting).
1(c). Describe the ROC curve.
ANSWER
•
ROC plots True Positive Rate (Recall) against False Positive Rate at many thresholds.
•
TPR = TP/(TP+FN); FPR = FP/(FP+TN).
•
A curve nearer the top-left is better; diagonal performance is close to random.
•
ROC-AUC summarizes ranking ability: 1 is perfect, 0.5 is random.
1(d). Explain Gini Impurity and Entropy.
ANSWER
•
Both measure impurity of a decision-tree node; 0 means a pure node.
•
Gini = 1 - sum(p_i^2). It is simple and usually faster to compute.
•
Entropy = -sum(p_i log2 p_i). Information gain is the reduction in entropy.
•
The tree selects the split with the largest weighted impurity reduction.
1(e). Explain Random Forest working and list two advantages.
ANSWER
•
Draw many bootstrap samples and train one decision tree on each sample.
•
At every split, consider only a random subset of features; grow diverse trees.
•
Combine outputs by majority vote or average probability.
•
Advantages: lower variance/overfitting than one tree; robust performance with useful
feature importance.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 6
1(f). Explain Ensemble Learning.
ANSWER
•
Ensemble learning combines multiple models to produce one stronger prediction.
•
Bagging reduces variance; Boosting corrects errors sequentially; Voting combines
outputs; Stacking learns a combination.
•
Diverse models with uncorrelated errors usually give the largest benefit.
•
Main gains: better accuracy, stability, and generalization; cost: complexity and
computation.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 7
February 2025
UE20CS931 | Section A: 20 marks | 5 questions
1(a). Define precision, recall, and F1-score. Explain their significance in model
evaluation.
ANSWER
•
Precision = TP/(TP+FP): trustworthiness of positive predictions.
•
Recall = TP/(TP+FN): ability to find actual positives.
•
F1 = 2PR/(P+R): single score balancing precision and recall.
•
Use them with the confusion matrix, especially when class imbalance makes accuracy
misleading.
1(b). Briefly explain Gaussian Naive Bayes and state its assumptions.
ANSWER
•
Uses Bayes theorem and predicts the class with highest posterior probability.
•
For each class, each continuous feature is modeled by a Gaussian distribution using its
mean and variance.
•
Assumptions: features are conditionally independent given the class; continuous
features are approximately Gaussian within each class.
•
Also assumes representative training data and stable class priors; it is fast but
independence is often approximate.
1(c). Briefly explain Stacking and Voting classifiers.
ANSWER
•
Voting combines base-model outputs directly: hard voting uses majority class; soft
voting averages probabilities.
•
Stacking feeds out-of-fold base-model predictions to a meta-model that learns how to
combine them.
•
Voting is simpler and safer; stacking can be stronger but needs careful cross-validation.
•
Both work best when base models are accurate and diverse.
1(d). Explain bagging. How does it improve classification? Explain Random Forest
training and testing.
ANSWER
•
Bagging trains models independently on bootstrap samples and aggregates their
predictions.
•
Averaging reduces variance, instability, and overfitting of high-variance learners.
•
Training RF: bootstrap data -> random feature subset at each split -> grow many trees.
•
Testing RF: pass the sample through every tree -> majority vote or average
probabilities.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 8
1(e). What is cross-validation in machine learning?
ANSWER
•
Cross-validation repeatedly splits training data into training and validation folds.
•
In K-fold CV, train on K-1 folds, validate on the remaining fold, and average K scores.
•
Stratified K-fold preserves class ratios and is preferred for classification.
•
Uses: estimate generalization, compare models, and tune hyperparameters without
touching the test set.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 9
PART II - ADDITIONAL
LIKELY THEORY
QUESTIONS
Syllabus-based preparation after the full
paper set. HIGH means direct syllabus
emphasis or a frequent exam pattern.
A. Classification foundations
 HIGH PRIORITY
What is supervised classification?
ANSWER
•
Learns from labeled examples where the target is categorical.
•
Goal: predict a class/probability for unseen observations.
•
Examples: churn yes/no, disease class, seed type.
MEDIUM PRIORITY
Differentiate binary and multiclass classification.
ANSWER
•
Binary: exactly two classes, such as default/no default.
•
Multiclass: more than two classes, such as class A/B/C.
•
Metrics may be reported using macro, micro, or weighted averaging.
 HIGH PRIORITY
List the standard steps in a data science classification project.
ANSWER
•
Define problem -> collect data -> EDA -> clean/encode -> split data.
•
Train -> validate/tune -> test -> interpret -> deploy/monitor.
•
Fit preprocessing only on training data to avoid leakage.
 HIGH PRIORITY
What is data leakage? How can it be prevented?
ANSWER
•
Leakage occurs when training uses test, future, or target-derived information.
•
It creates unrealistically high validation scores.
•
Split early; use pipelines; fit imputation/scaling/SMOTE only inside training folds.
 HIGH PRIORITY
Why use a stratified train-test split?
ANSWER
•
It preserves class proportions in train and test sets.
•
It gives more reliable evaluation for imbalanced classes.
•
The test set must remain untouched until final evaluation.
 HIGH PRIORITY
Differentiate overfitting and underfitting.
ANSWER
•
Overfitting: excellent training score, weak validation/test score; high variance.
•
Underfitting: weak scores on both training and test data; high bias.
•
Control using model complexity, regularization, CV, pruning, features, or more data.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 10
B. Logistic Regression and probability
 HIGH PRIORITY
Explain Logistic Regression for classification.
ANSWER
•
Models log-odds as a linear combination of features.
•
Sigmoid converts the score to probability: p = 1/(1 + exp(-z)).
•
Apply a threshold to convert probability into a class.
 HIGH PRIORITY
Define probability, odds, and logit.
ANSWER
•
Probability p lies between 0 and 1.
•
Odds = p/(1-p); logit = log[p/(1-p)].
•
Logistic Regression assumes the logit is linear in the features.
 HIGH PRIORITY
State the main assumptions of Logistic Regression.
ANSWER
•
Independent observations; correct binary outcome specification.
•
Linear relation between continuous features and log-odds.
•
No severe multicollinearity; adequate sample size; limited influential outliers.
 HIGH PRIORITY
How are Logistic Regression coefficients interpreted?
ANSWER
•
Positive beta increases log-odds; negative beta decreases log-odds.
•
exp(beta) is the odds ratio for a one-unit feature increase, holding others fixed.
•
Significance and business meaning should be checked, not only coefficient size.
 HIGH PRIORITY
How does classification threshold affect precision and recall?
ANSWER
•
Lower threshold: more positive predictions -> recall rises, precision often falls.
•
Higher threshold: fewer positive predictions -> precision often rises, recall falls.
•
Choose threshold using business cost and validation data, not the test set.
MEDIUM PRIORITY
What are L1 and L2 regularization?
ANSWER
•
Both add a coefficient penalty to reduce overfitting.
•
L1 can make coefficients exactly zero and perform feature selection.
•
L2 smoothly shrinks coefficients and is stable with correlated features.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 11
C. Evaluation metrics and imbalanced data
 HIGH PRIORITY
Explain a confusion matrix.
ANSWER
•
TP: correctly predicted positive; TN: correctly predicted negative.
•
FP: false alarm; FN: missed positive.
•
It is the base for accuracy, precision, recall, specificity, and F1.
 HIGH PRIORITY
When is accuracy misleading?
ANSWER
•
Accuracy = (TP+TN)/Total.
•
With severe imbalance, predicting only the majority class may look accurate.
•
Also inspect recall, precision, F1, PR-AUC, balanced accuracy, and costs.
 HIGH PRIORITY
Define specificity and balanced accuracy.
ANSWER
•
Specificity = TN/(TN+FP): ability to identify negatives.
•
Balanced Accuracy = (Recall + Specificity)/2.
•
Balanced accuracy gives equal importance to both classes.
 HIGH PRIORITY
Compare ROC-AUC and PR-AUC.
ANSWER
•
ROC-AUC summarizes TPR vs FPR across thresholds.
•
PR-AUC summarizes precision vs recall and focuses on the positive class.
•
Prefer PR-AUC when positives are rare; report both when useful.
 HIGH PRIORITY
What is Cohen's Kappa?
ANSWER
•
Measures agreement between predicted and actual labels beyond chance.
•
Kappa = (observed accuracy - expected accuracy)/(1 - expected accuracy).
•
1 = perfect; 0 = chance-level; below 0 = worse than chance.
 HIGH PRIORITY
How can class imbalance be handled?
ANSWER
•
Use class weights, threshold tuning, oversampling/SMOTE, or undersampling.
•
Use stratified CV and suitable metrics such as recall, F1, and PR-AUC.
•
Resample training folds only; never resample the test set.
 HIGH PRIORITY
Explain SMOTE and one limitation.
ANSWER
•
SMOTE creates synthetic minority samples between nearby minority observations.
•
It can improve minority recall without exact duplication.
•
Limitation: may create noisy/overlapping points; apply only inside training folds.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 12
D. Naive Bayes and KNN
 HIGH PRIORITY
State Bayes theorem and its classification use.
ANSWER
•
P(C|X) = P(X|C)P(C)/P(X).
•
Compute posterior probability for every class.
•
Predict the class with the highest posterior.
 HIGH PRIORITY
What is the 'naive' assumption in Naive Bayes?
ANSWER
•
Features are conditionally independent given the class.
•
Then P(X|C) becomes the product of individual feature likelihoods.
•
The assumption is strong, but the classifier can still work well.
MEDIUM PRIORITY
Name the main Naive Bayes variants.
ANSWER
•
Gaussian NB: continuous, approximately normal features.
•
Multinomial NB: counts/frequencies, common in text.
•
Bernoulli NB: binary feature presence/absence.
 HIGH PRIORITY
Why is feature scaling important for KNN?
ANSWER
•
KNN depends directly on distance.
•
Large-scale features otherwise dominate small-scale features.
•
Fit StandardScaler/MinMaxScaler on training data only.
 HIGH PRIORITY
How is K chosen in KNN?
ANSWER
•
Use cross-validation over candidate K values.
•
Small K: flexible, noisy, high variance; large K: smooth, high bias.
•
For binary classes, an odd K can reduce ties.
MEDIUM PRIORITY
List common KNN distance measures and limitations.
ANSWER
•
Distances: Euclidean, Manhattan, Minkowski, Chebyshev.
•
Limitations: slow prediction, sensitive to scale/noise, memory-heavy.
•
High dimensions make distances less informative (curse of dimensionality).


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 13
E. Decision Trees and Random Forest
 HIGH PRIORITY
What is information gain?
ANSWER
•
Reduction in impurity caused by a split.
•
Gain = parent impurity - weighted child impurity.
•
Choose the split with the greatest gain.
 HIGH PRIORITY
Which hyperparameters control Decision Tree overfitting?
ANSWER
•
max_depth and max_leaf_nodes limit tree size.
•
min_samples_split and min_samples_leaf prevent tiny leaves.
•
ccp_alpha performs cost-complexity pruning.
MEDIUM PRIORITY
Does a Decision Tree require feature scaling? Why?
ANSWER
•
Usually no; trees compare one feature with a threshold.
•
Monotonic rescaling preserves the split ordering.
•
Scaling may still be needed when trees share a pipeline with distance/linear models.
 HIGH PRIORITY
Define bootstrap sampling and out-of-bag evaluation.
ANSWER
•
Bootstrap sample: draw training rows with replacement.
•
Unselected rows for a tree are its out-of-bag (OOB) samples.
•
Aggregate OOB predictions to estimate performance without a separate validation split.
 HIGH PRIORITY
Why does Random Forest use random feature subsets?
ANSWER
•
It prevents the same strong feature from dominating every tree.
•
This decorrelates trees, so averaging reduces variance more effectively.
•
max_features controls the number considered at each split.
 HIGH PRIORITY
Explain feature importance and its cautions.
ANSWER
•
Impurity importance totals split improvements from each feature.
•
Permutation importance measures score drop after shuffling a feature.
•
Correlated features share importance; impurity importance may favor high-cardinality
features.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 14
F. Boosting, Voting, and Stacking
 HIGH PRIORITY
What is a weak learner in boosting?
ANSWER
•
A model only slightly better than random, often a shallow tree/stump.
•
Boosting adds weak learners sequentially to correct prior errors.
•
Their weighted sum becomes a strong learner.
 HIGH PRIORITY
What is XGBoost?
ANSWER
•
A regularized, optimized gradient-boosted tree implementation.
•
Uses shrinkage, row/column subsampling, and efficient tree construction.
•
Strong performance, but careful tuning and validation are required.
 HIGH PRIORITY
Explain the learning-rate and estimator trade-off in boosting.
ANSWER
•
Learning rate controls each tree's contribution.
•
Smaller rate usually needs more estimators and can generalize better.
•
Too many/deep trees may overfit; tune jointly using cross-validation.
 HIGH PRIORITY
Differentiate hard and soft voting.
ANSWER
•
Hard voting: majority of predicted class labels.
•
Soft voting: average predicted probabilities, then choose the largest.
•
Soft voting needs reasonably calibrated probabilities and may use model weights.
 HIGH PRIORITY
How is leakage prevented in stacking?
ANSWER
•
Train the meta-model on out-of-fold predictions from base models.
•
Do not train it on base predictions made on the same fitted rows.
•
Refit base models on all training data only after meta-training design is fixed.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 15
G. Model selection, preprocessing, and
deployment
 HIGH PRIORITY
Compare GridSearchCV and RandomizedSearchCV.
ANSWER
•
Grid search tests every listed combination; thorough but expensive.
•
Randomized search samples a fixed number of combinations; faster for large spaces.
•
Both must use suitable stratified CV and a business-relevant scoring metric.
 HIGH PRIORITY
Why use a preprocessing pipeline?
ANSWER
•
Keeps imputation, encoding, scaling, and model steps together.
•
Fits preprocessing separately inside each training fold, preventing leakage.
•
Makes training and inference consistent and reproducible.
 HIGH PRIORITY
How should missing values be handled?
ANSWER
•
First inspect amount, pattern, and business meaning of missingness.
•
Numeric: median/mean/model-based; categorical: mode or 'Missing' category.
•
Fit imputation on training data only; optionally add a missing indicator.
 HIGH PRIORITY
How should outliers be treated?
ANSWER
•
Validate whether they are errors, rare valid cases, or target-relevant signals.
•
Options: correct, cap/winsorize, transform, use robust models/scalers, or retain.
•
Do not delete automatically; document impact and business reason.
 HIGH PRIORITY
Compare one-hot, ordinal, and target encoding.
ANSWER
•
One-hot: unordered categories; safe and interpretable but may be wide.
•
Ordinal: ordered categories only; assigns meaningful rank.
•
Target encoding: useful for high cardinality, but must be out-of-fold to avoid leakage.
MEDIUM PRIORITY
What are probability calibration and model calibration?
ANSWER
•
A calibrated 0.8 probability should be correct about 80% of the time.
•
Check with calibration curve/Brier score; improve with Platt or isotonic calibration.
•
Calibration matters for risk ranking, thresholds, and cost-based decisions.
MEDIUM PRIORITY
How are models saved and deployed? What risks must be monitored?
ANSWER
•
Serialize the complete fitted pipeline using joblib/pickle and serve it through an
API/batch process.
•
Validate input schema, versions, security, latency, and reproducibility.
•
Monitor data drift, concept drift, bias, calibration, and live performance.


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 16
PART III - ONE-LOOK
FORMULA SHEET
Write the relevant formula first, define each
term, and add one line explaining when the
metric is useful.
CONCEPT
FORMULA / MEMORY LINE
Accuracy
(TP + TN) / (TP + TN + FP + FN)
Precision
TP / (TP + FP)
Recall / Sensitivity
TP / (TP + FN)
Specificity
TN / (TN + FP)
F1-score
2 x Precision x Recall / (Precision + Recall)
False Positive Rate
FP / (FP + TN) = 1 - Specificity
Balanced Accuracy
(Recall + Specificity) / 2
Cohen's Kappa
(Observed accuracy - Expected accuracy) / (1 - Expected
accuracy)
Odds
p / (1 - p)
Logit
log[p / (1 - p)]
Sigmoid
1 / (1 + exp(-z))
Gini Impurity
1 - sum(p_i^2)
Entropy
-sum(p_i log2 p_i)
Information Gain
Parent impurity - Weighted child impurity
Euclidean Distance
sqrt(sum((x_i - y_i)^2))
Bayes Theorem
P(C|X) = P(X|C)P(C) / P(X)
Fast comparison lines
•
Bagging = parallel learners + bootstrap + variance reduction.
•
Boosting = sequential learners + error correction + bias reduction.
•
Decision Tree = interpretable single tree; Random Forest = stable average of randomized trees.
•
Voting = fixed combination; Stacking = learned combination using a meta-model.
•
Lower threshold -> higher recall; higher threshold -> higher precision (usually).


ML2 SECTION A - QUICK REVISION
Machine Learning - II
Past papers first | Predicted questions second | Short pointwise answers
Page 17
Source coverage and final checklist
Past papers used
•
Question_Paper_ML2_ESA_Sep2021_Set2.pdf
•
Question_Paper_ML2_ESA_Dec2021_Set1.pdf
•
Question_Paper_ML2_ESA_Feb2025.pdf
Additional preparation sources
•
ML2-Courseoutline-1.pdf and ML-2 FAQ (1).pdf
•
ML2 Final Exam Master Preparation material and the solved paper notebooks/PDFs.
•
Session topics: Logistic Regression, metrics, imbalance, Naive Bayes, KNN, trees, Random Forest,
boosting, stacking, and voting.
90-minute revision checklist
[ ]
20 min
Memorize metrics, confusion matrix, ROC, threshold trade-off.
[ ]
20 min
Revise trees, impurity, Random Forest, bagging vs boosting.
[ ]
15 min
Revise KNN, Naive Bayes, Logistic Regression assumptions.
[ ]
15 min
Revise AdaBoost, Gradient Boosting, XGBoost, voting, stacking.
[ ]
10 min
Revise imbalance, SMOTE, CV, leakage, pipeline.
[ ]
10 min
Answer any five questions aloud without looking.
Final memory rule: definition -> formula/steps -> why useful -> one
limitation. Keep each point direct.
