Metadata-Version: 2.4
Name: sw-mlp
Version: 1.0.0
Summary: A multilayer perceptron implemented with Numpy.
License-Expression: MIT
License-File: LICENSE
Requires-Dist: matplotlib>=3.11.2
Requires-Dist: numpy>=2.5.4
Requires-Dist: pandas==2.2.3
Requires-Python: >=3.12, <3.13
Description-Content-Type: text/markdown

# Multilayer Perceptron

Multilayer perceptron (MLP) is a simple neural network (NN), can solve both regression and classification tasks. The project provides a reusable and modular implementation.

[[Demo Video]](https://www.youtube.com/watch?v=YRcl1f--tcw)

## Requirements

The project is built from scratch with Python, with only Numpy, Pandas, Matplotlib,
without high-level ML libraries.

Use Python **3.12** and **uv** to manage the environment and dependencies.
Install uv before continuing. Run the following commands from the project root:

```bash
git clone https://github.com/swangarch/multilayer_perceptron
cd multilayer_perceptron

# Create .venv and install the locked dependencies and the project
uv sync --locked

# Alternatively, use the Makefile (runs uv sync)
make setup
```

Commands below use `uv run`, so activating the virtual environment is not required.
To activate it manually on macOS/Linux, use `source .venv/bin/activate`.


## Usage

Run these commands from the project root. Makefile targets use `uv run -m mlp`.

```bash
uv run -m mlp --help
```

### For example dataset classification

```bash
# To split the dataset
make split

# To train with training configuration file and training dataset
make train

# To test with test dataset
make test
```

Breast cancer diagnosis dataset, a binary classification based on 30 features of a cell.

### MNIST handwrite digits classification (0–9)

```bash
# To split, train and test the dataset
make digit
```

For the handwritten digit classification task (0–9), the model achieved 97.19% test accuracy using 33,000 training samples of 28 × 28 images in the optimal configuration.

For the handwritten character recognition task with 94 categories, the model reached 91% accuracy based on 175,000 of 28 × 28 images training samples.

### For random generated 1d data regression

```bash
# To train and test generated data, with relu activation
make regre-relu

# To train and test generated data, with sigmoid activation
make regre-sigmoid
```

## Packaging

Build the wheel and source distribution with uv:

```bash
uv build

# Alternatively
make build
```

The output files are written to `dist/`:

- `sw_mlp-1.0.0-py3-none-any.whl`
- `sw_mlp-1.0.0.tar.gz`

To install the wheel into a Python 3.12 virtual environment:

```bash
uv venv --python 3.12
uv pip install dist/sw_mlp-1.0.0-py3-none-any.whl

# macOS/Linux: run the installed package
.venv/bin/python -m mlp --help
```

The wheel contains the Python package. Example configurations and datasets in the
repository's `config/` and `data/` directories must be distributed separately.
Pass their paths when running the installed package. The split command writes to
`data/train.csv` and `data/test.csv`, so the working directory must contain a
`data/` directory.

## Features

- Training methods → supports **Stochastic Gradient Descent (SGD)** and **mini-batch training**.

- **Visualization** → includes real-time animations to track the learning process.

- **Configurable** architecture → adjustable network shape, activation functions, initialization methods, max iterations, learning rate, batch size, etc via a configuration file.

- Data handling → data loading and preprocessing.

- Support multiple activation functions: **ReLU, Sigmoid, Softmax, Leaky ReLU**

- Support multiple loss functions: 
  - **Categorical Cross Entropy (CCE)** with automatic one-hot encoding for categorical labels
  - **Binary Cross Entropy (BCE)**
  - **Mean Squared Error (MSE)**

- Support **Fine-tuning**: continue training from saved weights for further optimization
  
## Methodology

![Demo](visualization/diagram/mlp.jpg)

A **Multilayer Perceptron** is a basic artificial neural network that learns to map input data to the correct output by adjusting internal parameters called weights and biases. It is composed of multiple layers of artificial neurons, where each neuron performs a simple computation — combining inputs, multiplying weights, adding a bias, and passing the result through an activation function to introduce non-linearity. Then using back propagation and gradient descent to find ideal weights and biases.

### Learning Process
The learning process consists of three main stages:

- **Feedforward** — compute predictions by passing data through all layers.
- **Back propagation** — calculate gradients of the loss with respect to the weights, using chain rule.
- **Gradient Descent** — update weights and biases to minimize the loss over time.

Through **repeated iterations** of these steps, the MLP gradually improves its predictions and learns to generalize from the training data.

### Feedfoward
![Demo](visualization/diagram/feedforward.jpg)

In the feedforward step, the input data passes through several layers of neurons. Each neuron performs a weighted sum of its inputs, adds a bias, and then applies an activation function such as ReLU, Sigmoid, or Softmax.

When neurons are grouped into a layer, their computations can be efficiently expressed using **matrix multiplication**. This allows the model to process many neurons at once, using vectorized operations in **NumPy**. Each layer's output becomes the input to the next layer, forming a feedforward chain that passes information through the network.

$$Z=WX+b$$

$$A=f(Z)$$

The final layer produces the output — either a value (for regression) or class probabilities (for classification).

### Back probagation
![Demo](visualization/diagram/back_probagation.jpg)

In backpropagation, the model calculates how far its predictions are from the correct answers using a loss function (like MSE or Cross-Entropy). Then, it computes the gradient of this loss to each weight and bias using the **chain rule**. This process tells the model which direction each weight and bias should move to reduce the overall error.

For the output layer, the gradient simplification is elegant:

**MSE Regression**: The output layer typically uses no activation function (linear output). The gradient of the loss with respect to the output is:

$$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$$

**Cross-Entropy Classification**: The output layer uses Softmax (multi-class) or Sigmoid (binary) activation. Remarkably, when combined with Cross-Entropy loss, the gradient simplifies to the same form:

$$\frac{\partial L}{\partial z} = \hat{y} - y$$



If you're interested in the details, this [**article**](https://shivammehta25.github.io/posts/deriving-categorical-cross-entropy-and-softmax/) by Shivam Mehta explains in depth how this simplification was derived. This mathematical convenience makes backpropagation efficient for both regression and classification tasks.



### Gradient descent
![Demo](visualization/diagram/gradient_descent.jpg)

Gradient descent is the optimization process that updates the weights and biases. It moves the weights and biases in the direction that reduces the loss the most, step by step.

$$
W := W - 𝜂 \frac{\partial J}{\partial W}
$$

$$
b := b - 𝜂 \frac{\partial J}{\partial b}
$$

The learning rate controls how big each step is — too high can make learning unstable, too low can make it too slow.

There are two main modes implemented in this project:

**Stochastic Gradient Descent (SGD)**: updates weights for each training sample, when batch size is set to 1.

**Mini-batch Gradient Descent**: updates weights after a small batch of samples, balancing speed and stability.

### Non linear classification

![Demo](visualization/diagram/classification.jpg)

This diagram shows how an MLP can separate data that is not linearly separable — for instance, when data points form circular or spiral patterns.
With multiple layers and non-linear activations, the network learns complex boundaries that a simple linear model cannot.

## Programs 

In addition to the provided Makefile, the following section explains the usage in detail and describes how you can customize the training process to fit your specific needs.

### Configuration


An json config file to pass network structure and training configuration is required.

```
      {
          "shape": [784, 64, 32, 10],  
          // Network shape, number of neurons in each layer

          "activation_funcs": ["relu", "relu", "softmax"],  
          // Activation functions for each layer

          "weights_init": ["he", "he", "xavier"],  
          // Layer weights initialization methods, e.g. he / xavier / zero

          "loss": "CategoricalCrossEntropy",  
          // Loss function, supporting MeanSquareError and CrossEntropy

          "max_epoch": 1000,  
          // Maximum number of training epochs

          "learning_rate": 0.01,  
          // Learning rate for gradient descent

          "batch_size": 50,  
          // Batch size for mini-batch training， 1 for SGD

          "classification": true,  
          // Whether to calculate accuracy (true for classification tasks)

          "animation": "none",  
          // To show training animation, recommended only for 1D data

          "train_ratio": 0.8,  
          // Ratio to split dataset into training and validation

          "threshold": false,  
          // Threshold for early stopping, false means disabled

          "index": false  
          // Whether the input CSV file contains an index column
      }
```
### Train and predict
We recommand you to use --help option before the first usage.
```
      Usage:

        uv run -m mlp <option> <config.json> <data.csv> [params.json]

      Options:

        -s:  Split dataset into (train.csv) and (test.csv), no params required

        -t:  Train dataset, if params.json is provided, it performs fine tuning.

        -p:  Test on a test dataset, if params.json is provided, it performs prediction based on trained model.

        --help:  Show help messages.

        -More features to come.

      Note:

        There is built in dataset generated by script, use --gen-data1d to replace dataset
```

The program will:

- Defines the network structure (layers, activation functions).  

- Trains using backpropagation and gradient descent.  

- Visualize loss and prediction:

- By default, regression task will visualize the loss, and the data distribution for first dimension.

- In addition, classification task will add an accuracy plot.


### Demo
The example graph on both classification and regression.

![Demo](visualization/nn/demo.png)


### Datasets

Datasets are provided in this repository, one example corresponding to breast cancer diagnosis classification.
It contains 32 columns, where the diagnosis column is the label to predict. The label can take two values:

M → Malignant (cancerous)

B → Benign (non-cancerous)

The remaining 30 numerical features describe various characteristics of a cell nucleus extracted with fine-needle aspiration (FNA), such as radius, texture, perimeter, area, and smoothness. These measurements are widely used for early breast cancer detection.

### To split data:

It is split into two parts: one for training and validation one for testing.

```bash
uv run -m mlp -s config/data.json data/data.csv
```

The objective is to classify whether a tumor is malignant or benign based on the given features.

### To train
```bash
# Train a new model
uv run -m mlp -t config/data.json data/train.csv

# Fine-tune an existing model
uv run -m mlp -t config/data.json data/train.csv params.json
```
After training, the params.json will be saved, which contain weights, biases and network structure.
If params.json is provided, the program will perform fine tuning.

### To test
```bash
uv run -m mlp -p config/data.json data/test.csv params.json
```

The loss function is selected explicitly by the `loss` field in the configuration.
Use `BinaryCrossEntropy` with a sigmoid output for binary classification,
`CategoricalCrossEntropy` with a softmax output for multi-class classification,
or `MeanSquareError` with a linear output (`"none"`) for regression.
The output activation alone does not select the loss function.

The program will make prediction, an accuracy will be calculated and prediction will be saved in predictions.json.

### Other datasets and configurations provided
- 0-9.csv _____ 0-9.json => 0 to 9 hand wirte digits recognition with softmax and CCE loss
- --gen-data1d _____ regre-relu.json => 1d random generated data with RELU activation and MSE loss
- --gen-data1d _____ regre-sigmoid.json => 1d random generated data with Sigmoid activation and MSE loss

### Class 

Below is a simple example of using the `MultiLayerPerceptron` class to train a regression model.

```python
from mlp.neural_network import MultiLayerPerceptron, relu
from mlp.data_process import generate_data_rand

def main():
    try:
        # Define network structure: input → 64 → 32 → output
        net_shape = (1, 64, 32, 1)
        activation_funcs = (relu, relu, None)  # last layer is linear (regression)
        weights_init = ("he", "he", "he")  # Define the weights initialization methods

        # Initialize neural network
        nn = MultiLayerPerceptron(net_shape, activation_funcs, weights_init)

        # Generate training data
        inputs, truths = generate_data_rand(142, 500, 0.02)
        test_inputs, test_truths = generate_data_rand(123, 50, 0.02)
        nn.train(inputs, truths, 10000, 0.005, batch_size=20, animation="plot")

        # Test with new data
        nn.test(test_inputs, test_truths)

        # Save graphs
        nn.save_plots()

    except Exception as e:
        print("Error:", e)


if __name__ == "__main__":
    main()
```

### Generalization

The figure shows that multilayer perceptron is capable of learning non-linear patterns from different datasets and can generalize beyond the training data, making them applicable to a wide range of tasks.

![Regression demo](visualization/nn/prediction.jpg)
---

### Hyperparameter Tuning: Batch Size

**Batch size** is a key hyperparameter that influences both training speed and model performance. To study its effect, I kept all other parameters constant and varied only the batch size, trained the model on the MNIST dataset. In this experiment, I used **CuPy** to replace **NumPy** in order to leverage GPU acceleration and improve computational efficiency during training.

```json
{
    "shape": [784, 392, 128, 64, 32, 10],
    "activation_funcs": ["relu", "relu", "relu", "relu", "softmax"],
    "weights_init": ["he", "he", "he", "he", "xavier"],
    "loss": "CategoricalCrossEntropy"
}
```

| Batch Size | Time (s) | Accuracy | Loss |
|:-----------:|:--------:|:--------:|:-----:|
| 1           | 2040     | 0.9814   | 0.1580 |
| 10          | 180      | 0.9769   | 0.1197 |
| 50          | 45       | 0.9719   | 0.1371 |
| 100         | 31       | 0.9676   | 0.1153 |
| 200         | 25       | 0.9593   | 0.1362 |
| 500         | 22       | 0.9414   | 0.1937 |
| 1000        | 21       | 0.9414   | 0.1937 |

#### Analysis

- **Training Time**: As batch size increases, training time decreases dramatically, reflecting better computational efficiency.

- **Accuracy**: Accuracy drops noticeably when batch size exceeds 200, indicating that very large batches may reduce generalization capability.
The drop in accuracy for large batches can sometimes be reduced by slightly increasing the learning rate.

- **Loss**: Smaller batches tend to achieve lower loss values, suggesting better fine-grained learning at the cost of longer training time.

#### Conclusion

In this experiment, batch sizes between 50 and 100 provide the best balance between training efficiency and model performance.
Smaller batches has higher accuracy but are slower, while larger batches are faster but may sacrifice precision.

---

## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
