import torch
import torch.nn as nn
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

From a Single-Layer Perceptron to a Multilayer Perceptron

A perceptron is the fundamental building block of a neural network. It takes one or more numerical inputs, multiplies each by a learnable weight, adds them up with a bias term, and passes the result through an activation function to produce a single output. By combining many perceptrons into layers, we can build networks that learn complex patterns from data.

What is a Single-Layer Perceptron?

A single-layer perceptron is the simplest neural network. It has just one layer of weights connecting inputs directly to outputs. For the problem of classification, we can use an activation function as a threshold to predict the class. However, for the problem of regression, we do not need to have an activation function, as we can use a linear function to predict the continuous values directly.

%%{init: {
    'theme': 'mc',
    'themeVariables': {
        'fontSize': '13px',
        'primaryColor': '#2d6a4f',
        'lineColor': '#52b788'},
        'flowchart': {'nodeSpacing': 15, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
    x1(["x₁"]) -->|"w₁"| sum
    x2(["x₂"]) -->|"w₂"| sum
    x3(["x₃"]) -->|"w₃"| sum
    x4(["x₄"]) -->|"w₄"| sum
    x5(["xₘ"]) -->|"wₘ"| sum

    sum(["∑ wᵢxᵢ + b"]) -->|"z"| act(["σ(z)"])
    act --> yhat(["ŷ"])

    style x1 fill:#52b788,stroke:#2d6a4f,color:#0d0900
    style x2 fill:#52b788,stroke:#2d6a4f,color:#0d0900
    style x3 fill:#52b788,stroke:#2d6a4f,color:#0d0900
    style x4 fill:#52b788,stroke:#2d6a4f,color:#0d0900
    style x5 fill:#52b788,stroke:#2d6a4f,color:#0d0900
    style sum fill:#e6a817,stroke:#c98f0a,color:#0d0900
    style act fill:#a2d2ff,stroke:#6da9e4,color:#0d0900
    style yhat fill:#52b788,stroke:#2d6a4f,color:#0d0900

A single-layer perceptron: inputs are multiplied by weights, summed with a bias, and passed through an activation function to produce the output.

\hat{y} = \sigma\!\left(\sum_{i=1}^{m} w_i x_i + b\right)

where x is the vector of inputs, w denotes the vector of weights, b is the bias, \sigma is an activation function (e.g. a step function or sigmoid), and \hat{y} is the predicted output. Because it has no hidden layers, a single-layer perceptron can only learn linearly separable patterns (i.e., it draws a single straight line or hyperplane to divide the features). This is why a single-layer perceptron cannot learn the XOR function.

Why a single-layer perceptron cannot learn XOR?

The XOR (exclusive OR) function takes two binary inputs and returns 1 when exactly one of them is 1, and 0 otherwise:

x_1	x_2	XOR output
0	0	0
0	1	1
1	0	1
1	1	0

If we plot these four points on a 2D plane (with x_1 on one axis and x_2 on the other), the two classes with outputs of 0 (at corners (0,0) and (1,1)) and outputs of 1 (at corners (0,1) and (1,0)) would sit diagonally opposite each other. No single straight line can separate the 0s from the 1s. A single-layer perceptron can only draw one straight line as its decision boundary, so it always misclassifies at least one point. Adding a hidden layer solves this: the first layer can draw two lines that each carve the plane, and the second layer combines them to capture the diagonal pattern that XOR requires.

An illustration of 2D plane to represent XOR function (Source: Deep Forward Network, XOR Problem)

What is a Multilayer Perceptron?

A multilayer perceptron (MLP) overcomes this limitation by stacking multiple layers of neurons with non-linear activations between them. The additional hidden layers allow the network to learn complex, non-linear decision boundaries.

MLP Architecture

An MLP is a type of feed-forward neural network consisting of:

Input layer: receives the raw features (no computation happens here).
One or more hidden layers: each layer applies a linear transformation \mathbf{z} = \mathbf{W}\mathbf{x} + \mathbf{b} followed by a non-linear activation function (e.g. ReLU, sigmoid).
Output layer: produces the final prediction (a probability, a class label, a regression value, etc.).

Every neuron in one layer is connected to every neuron in the next, which are called fully connected (or dense) layers. The “learning” happens by adjusting the weights \mathbf{W} and biases \mathbf{b} via gradient descent and backpropagation.

Multilayer perceptron architecture showing input, hidden, and output layers with loss function and optimizer feedback loop (Source: Zhang et al., 2024).

Example: classifying battery cell outliers with an MLP

The code below builds and trains a 3-layer MLP on the battery cell dataset to predict whether a cell is an outlier (1) or not (0), using two log-transformed features. The dataset and features are taken from Pang et al. (2025).

Step 1: Load the data

The dataset is loaded from a CSV file using pandas. We extract two input features (log_max_diff_dQ and log_max_diff_dV) and the binary outlier label.

# Load the dataset from CSV file
df_dataset = pd.read_csv(
    'df_features_per_cell_with_labels.csv')
print("Overview of the dataset:")
print(df_dataset.head(3))
print("-"*79)

df_input_features = df_dataset.loc[:, ["log_max_diff_dQ", "log_max_diff_dV"]]
print("Input features:")
print(df_input_features.head(3))
print("-"*79)

df_true_label = df_dataset.loc[:, "outlier_label"]
print("True labels:")
print(df_true_label.head(3))

Overview of the dataset:
   max_diff_dQ  log_max_diff_dQ  cycle_index  max_diff_dV  log_max_diff_dV  \
0     0.006388        -5.053343          0.0     1.564151         0.447343   
1     0.006338        -5.061258          1.0     0.023727        -3.741163   
2     0.006321        -5.063922          2.0     0.022684        -3.786091   

                      cell_index  outlier_label  
0  2017-05-12_5_4C-70per_3C_CH17              1  
1  2017-05-12_5_4C-70per_3C_CH17              0  
2  2017-05-12_5_4C-70per_3C_CH17              0  
-------------------------------------------------------------------------------
Input features:
   log_max_diff_dQ  log_max_diff_dV
0        -5.053343         0.447343
1        -5.061258        -3.741163
2        -5.063922        -3.786091
-------------------------------------------------------------------------------
True labels:
0    1
1    0
2    0
Name: outlier_label, dtype: int64

Step 2: Prepare the tensors

The pandas DataFrames are converted to PyTorch tensors so they can be used in autograd computations.

X has shape (N, 2): each row contains log_max_diff_dQ and log_max_diff_dV.
y has shape (N, 1): the binary outlier label, reshaped into a column vector so it matches the model’s output shape.

Step 3: Define the model

MLPExample is an MLP with two hidden layers:

Layer	Operation	Shape
`hidden1`	`nn.Linear(2, 12)` → ReLU	2 → 12
`hidden2`	`nn.Linear(12, 2)` → ReLU	12 → 2
`output`	`nn.Linear(2, 1)` → Sigmoid	2 → 1

ReLU (\max(0, x)) introduces non-linearity in the hidden layers.
Sigmoid (\frac{1}{1+e^{-x}}) squashes the output to [0, 1], giving a probability of being an outlier.

The forward() method chains these layers together: input → hidden1 → ReLU → hidden2 → ReLU → output → Sigmoid. This model architecture is illustrated in the following flowchart:

%%{init: {
    'theme': 'mc',
    'themeVariables': {
        'fontSize': '13px',
        'primaryColor': '#2d6a4f',
        'lineColor': '#52b788'},
        'flowchart': {'nodeSpacing': 15, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
    subgraph Input["Input Layer (2)"]
        x1(["x₁"])
        x2(["x₂"])
    end

    subgraph H1["Hidden Layer 1 (12)"]
        h1(["h₁"])
        h2(["h₂"])
        h3(["⋮"])
        h12(["h₁₂"])
    end

    subgraph A1["ReLU"]
        r1(["ReLU"])
    end

    subgraph H2["Hidden Layer 2 (2)"]
        g1(["g₁"])
        g2(["g₂"])
    end

    subgraph A2["ReLU"]
        r2(["ReLU"])
    end

    subgraph Out["Output Layer (1)"]
        o1(["o₁"])
    end

    subgraph A3["Sigmoid"]
        sig(["σ"])
    end

    yhat(["ŷ"])

    x1 --> h1 & h2 & h3 & h12
    x2 --> h1 & h2 & h3 & h12
    h1 & h2 & h3 & h12 --> r1
    r1 --> g1 & g2
    g1 & g2 --> r2
    r2 --> o1
    o1 --> sig
    sig --> yhat

    style x1 fill:#52b788,stroke:#2d6a4f,color:#0d0900
    style x2 fill:#52b788,stroke:#2d6a4f,color:#0d0900
    style h1 fill:#e6a817,stroke:#c98f0a,color:#0d0900
    style h2 fill:#e6a817,stroke:#c98f0a,color:#0d0900
    style h3 fill:#e6a817,stroke:#c98f0a,color:#0d0900
    style h12 fill:#e6a817,stroke:#c98f0a,color:#0d0900
    style r1 fill:#a2d2ff,stroke:#6da9e4,color:#0d0900
    style g1 fill:#e6a817,stroke:#c98f0a,color:#0d0900
    style g2 fill:#e6a817,stroke:#c98f0a,color:#0d0900
    style r2 fill:#a2d2ff,stroke:#6da9e4,color:#0d0900
    style o1 fill:#e6a817,stroke:#c98f0a,color:#0d0900
    style sig fill:#a2d2ff,stroke:#6da9e4,color:#0d0900
    style yhat fill:#52b788,stroke:#2d6a4f,color:#0d0900

    style Input fill:none,stroke:#2d6a4f,color:#2d6a4f
    style H1 fill:none,stroke:#c98f0a,color:#c98f0a
    style A1 fill:none,stroke:#6da9e4,color:#6da9e4
    style H2 fill:none,stroke:#c98f0a,color:#c98f0a
    style A2 fill:none,stroke:#6da9e4,color:#6da9e4
    style Out fill:none,stroke:#c98f0a,color:#c98f0a
    style A3 fill:none,stroke:#6da9e4,color:#6da9e4

MLPExample model architecture: 2 inputs → 12 hidden (ReLU) → 2 hidden (ReLU) → 1 output (Sigmoid).

Step 4: Train the model

# binary cross-entropy loss and 
# Adam optimiser with learning rate 0.001
loss_fn   = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
n_epochs  = 100
batch_size = 10

Training runs for 100 epochs in mini-batches of 10. Each iteration:

Forward pass: y_pred = model(Xbatch) computes predictions.
Compute loss: loss = loss_fn(y_pred, ybatch) measures the binary cross-entropy error.
Zero gradients: optimizer.zero_grad() clears gradients from the previous step.
Backward pass: loss.backward() computes \frac{\partial\,\text{loss}}{\partial\mathbf{w}} via autograd.
Update weights: optimizer.step() applies the Adam update rule to adjust parameters.

Step 5: Evaluate

After training, predictions are rounded to 0 or 1 and compared with the true labels to compute accuracy. The final loop prints the first 5 samples with their predicted and expected classes.

The entire code for building, training, and evaluating the MLP is shown below:

# convert the input features to PyTorch tensors
X = torch.tensor(
    df_input_features.values,
    dtype=torch.float32)

# convert the true labels to PyTorch tensors 
# and reshape to be a column vector
y = torch.tensor(
    df_true_label.values,
    dtype=torch.float32).reshape(-1, 1)

# define the model
class MLPExample(nn.Module):
    def __init__(self):
        super().__init__()

        # define the layers of the model with 
        # 2 input features, 1 output feature, 
        # and 2 hidden layers with 12 and 2 neurons respectively
        self.hidden1 = nn.Linear(2, 12)
        self.act1 = nn.ReLU()
        self.hidden2 = nn.Linear(12, 2)
        self.act2 = nn.ReLU()
        self.output = nn.Linear(2, 1)
        self.act_output = nn.Sigmoid()

    def forward(self, x):
        # define the forward pass of the model
        x = self.act1(self.hidden1(x))
        x = self.act2(self.hidden2(x))
        x = self.act_output(self.output(x))
        return x
    
model = MLPExample()
print("MLP Model Architecture:")
print(model)
print("-"*79)

# train the model
# define the loss function and optimizer, and train for 
# 100 epochs with a batch size of 10
loss_fn = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
n_epochs = 100
batch_size = 10

for epoch in range(n_epochs):
    for i in range(0, len(X), batch_size):
        Xbatch = X[i:i+batch_size]
        ybatch = y[i:i+batch_size]
        y_pred = model(Xbatch)
        loss = loss_fn(y_pred, ybatch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

# compute accuracy
y_pred = model(X)
accuracy = (y_pred.round() == y).float().mean()
print(f"Accuracy {accuracy}")
print("-"*79)

# make class predictions with the model
predictions = (model(X) > 0.45).int()

for i in range(5):
    print('%s => %d (expected %d)' % (X[i].tolist(), predictions[i], y[i]))

MLP Model Architecture:
MLPExample(
  (hidden1): Linear(in_features=2, out_features=12, bias=True)
  (act1): ReLU()
  (hidden2): Linear(in_features=12, out_features=2, bias=True)
  (act2): ReLU()
  (output): Linear(in_features=2, out_features=1, bias=True)
  (act_output): Sigmoid()
)
-------------------------------------------------------------------------------
Accuracy 0.9971098303794861
-------------------------------------------------------------------------------
[-5.053343296051025, 0.44734305143356323] => 1 (expected 1)
[-5.061258316040039, -3.7411627769470215] => 0 (expected 0)
[-5.063921928405762, -3.7860910892486572] => 0 (expected 0)
[-5.073755741119385, -3.579171657562256] => 0 (expected 0)
[-5.077345848083496, -3.634080648422241] => 0 (expected 0)

Building Blocks of a PyTorch Model

Weights and Training

A neural network model is a sequence of matrix operations. The matrices that are independent of the input and kept inside the model are called weights. Training a neural network optimises these weights so that they produce the output we want. In deep learning, the core algorithm used to optimise these weights is gradient descent.

Defining a Linear Layer

The first layer in our model hints at the shape of the input. For example, nn.Linear(2, 12) creates a fully connected (dense) layer that expects 2 input features and produces 12 outputs. Internally it computes:

\mathbf{z} = \mathbf{x}\mathbf{W}^\top + \mathbf{b}

where \mathbf{W} is a (12 \times 2) weight matrix and \mathbf{b} is a bias vector of length 12.

The batch dimension is implicit: if we pass a tensor of shape (n, 2), we get back a tensor of shape (n, 12), where n is the batch size.

Common Layer Types

PyTorch provides many layer types in torch.nn. Here are the ones we will encounter most often:

Layer	What it does
`nn.Linear(in, out)`	Fully connected layer, where every input is connected to every output.
`nn.Conv2d(in_ch, out_ch, kernel)`	2-D convolution which slides a small filter over an image; widely used in computer vision.
`nn.Dropout(p)`	Randomly zeroes a fraction p of elements during training to reduce overfitting (regularisation).
`nn.Flatten()`	Reshapes a high-dimensional tensor into a 1-D vector (per sample), typically placed between convolutional and linear layers.

Common Activation Functions

An activation function is applied element-wise after a layer’s linear transformation. Without it, stacking multiple linear layers would collapse into a single linear operation, so activations are what give a neural network its non-linear transformation.

Activation	Formula	Typical use
`nn.ReLU()`	\max(0, x)	Default choice for hidden layers in modern networks.
`nn.Sigmoid()`	\dfrac{1}{1+e^{-x}}	Squashes output to (0, 1); used in binary classification output layers.
`nn.Tanh()`	\dfrac{e^x - e^{-x}}{e^x + e^{-x}}	Squashes output to (-1, 1); common in older architectures and RNNs.
`nn.Softmax(dim)`	\dfrac{e^{x_i}}{\sum_j e^{x_j}}	Converts a vector of scores into a probability distribution; used in multi-class classification.

Loss Functions

A loss function measures how far the model’s output is from the desired output. It compares the model’s output tensor to the expected tensor (known as the label or ground truth). If labels are provided as part of the training dataset, the neural network model will be a supervised learning model. The loss function is a crucial component of training, as it provides the signal that guides the optimisation of the model’s weights.

In PyTorch, we can create and use a loss function as follows:

loss_fn = nn.BCELoss()
loss    = loss_fn(y_pred, y_true)

The returned loss is a tensor that supports automatic differentiation, calling loss.backward() computes the gradient of the loss with respect to every model weight.

Common loss functions in PyTorch:

Loss function	Typical use
`nn.MSELoss()`	Mean squared error (for regression problems).
`nn.CrossEntropyLoss()`	Cross-entropy (for multi-class classification).
`nn.BCELoss()`	Binary cross-entropy (for binary classification).

Optimization Algorithms

There are many variations of gradient descent. We choose one by creating an optimizer for our model. The optimizer is not part of the model itself, but is used alongside it during training.

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

All optimizers require a list of the parameters they need to update. We pass model.parameters() so the optimizer knows where to find the weights. During training, the optimizer takes the gradients computed by loss.backward() and applies them to the parameters according to its update rule.

Common optimizers in PyTorch:

Optimizer	Description
`torch.optim.Adam()`	Adaptive moment estimation (a good default choice).
`torch.optim.NAdam()`	Adam with Nesterov momentum.
`torch.optim.SGD()`	Stochastic gradient descent (the simplest optimizer).
`torch.optim.RMSprop()`	Scales learning rates by a running average of gradient magnitudes.

Running on CPU vs GPU

PyTorch is modular, where each layer (and tensor) can be placed on a specific device. By default, everything lives on the CPU. To leverage a GPU we explicitly move things to CUDA:

# CPU (default)
layer = nn.Linear(2, 12, device="cpu")

# GPU on NVIDIA
layer = nn.Linear(2, 12, device="cuda:0")

We can also move an entire model and its data in one call:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model  = MLPExample().to(device)
X      = X.to(device)
y      = y.to(device)

Do not mix CPU tensors with GPU tensors!

All tensors involved in a computation must be on the same device. Mixing CPU tensors with GPU tensors will raise a RuntimeError. To avoid this, we can set the default device for all tensors at the start of our code, or ensure that we consistently move data and models to the same device.

The data type (dtype) can also be specified per layer. The default torch.float32 is almost always what we want, but half-precision is available for speed on supported GPUs:

nn.Linear(2, 12, dtype=torch.float16)

Inspecting and Saving a Model

Once we have a model, we can inspect its architecture by printing it:

print(model)

For our MLPExample, this produces:

MLPExample(
  (hidden1): Linear(in_features=2, out_features=12, bias=True)
  (act1): ReLU()
  (hidden2): Linear(in_features=12, out_features=2, bias=True)
  (act2): ReLU()
  (output): Linear(in_features=2, out_features=1, bias=True)
  (act_output): Sigmoid()
)

To save and reload a model, PyTorch provides two approaches:

Option 1: Save the entire model object (uses Python’s pickle under the hood):

torch.save(model, "my_model.pth")
model = torch.load("my_model.pth")

Option 2 (recommended): Save only the weights via state_dict(), keeping the model architecture in code:

# Save
torch.save(model.state_dict(), "my_model.pth")

# Reload: rebuild the model first, then load the weights
model = MLPExample()
model.load_state_dict(torch.load("my_model.pth"))

Why Option 2?

It decouples the model definition from the saved file, making it easier to refactor the code or share weights across different projects.

Important Definitions

Epoch

In machine learning, an epoch refers to one complete pass through the entire training dataset, where every data sample is fed through the model and the parameters are updated based on the calculated error. Training typically requires multiple epochs, allowing the model to improve iteratively as it sees the data again and again.

Batch Size

In deep learning, datasets are usually divided into smaller subsets known as batches. The model processes these batches sequentially, updating the parameters after each batch. Batch size is a hyperparameter that controls how many samples are processed together, which in turn affects the frequency of weight updates.

For example, with a training dataset of 1 000 samples:

Batch size	Batches per epoch	Iterations per epoch
1 000	1	1
200	5	5
100	10	10

Larger batches: fewer updates per epoch, more stable gradients, but higher memory usage.
Smaller batches: more frequent updates, noisier gradients (which can act as regularisation), and lower memory usage.

Types of Gradient Descent by Batch Size

There are three variants of gradient descent, distinguished by how many samples are used per update:

Variant	Samples per update	Characteristics
Batch gradient descent	All samples in the training set	Smooth convergence but slow and memory-intensive.
Stochastic gradient descent (SGD)	1 random sample	Very noisy updates but fast and low memory.
Mini-batch gradient descent	A fixed subset (e.g. 32, 64, 128)	Best of both worlds: the most common choice in practice.

The figure below illustrates the convergence paths of each variant towards the loss minimum. Batch gradient descent (blue) takes smooth, direct steps. Mini-batch (red) follows a noisier but still directed path. Stochastic (green) is the most erratic but can escape shallow local minima.

Learning Rate

The learning rate is one of the most important hyperparameters for training neural networks. It controls the step size that gradient descent takes towards a local optimum of the loss function.

After computing the gradient of the loss with respect to the weights, the gradient points in the direction of steepest descent. The learning rate determines how far we move along that direction in each update:

\mathbf{w} \leftarrow \mathbf{w} - \alpha \, \nabla_{\mathbf{w}} \mathcal{L}

where \alpha is the learning rate and \nabla_{\mathbf{w}} \mathcal{L} is the gradient of the loss.

The learning rate controls the step size of gradient descent towards the local optimum of the cost function (Source: Relation Between Learning Rate and Batch Size).

Choosing the right learning rate matters:

Too small: gradient descent converges very slowly, requiring many more epochs to reach a good solution.
Too large: gradient descent may overshoot and start to diverge, never reaching the optimum.

The learning rate does not have to remain fixed throughout training. Common strategies include:

Learning rate schedules: decrease the learning rate as training progresses (e.g. step decay, cosine annealing).
Adaptive methods: optimization algorithms like Adam and RMSprop automatically adjust the effective learning rate per parameter during training.

References

Deep Forward Network, XOR Problem -1. https://medium.com/@pentagonspace4747/deep-forward-network-xor-problem-1-49bff78db5af
Zhang, W., Shen, X., Zhang, H. et al. Feature importance measure of a multilayer perceptron based on the presingle-connection layer. Knowl Inf Syst 66, 511–533 (2024). https://doi.org/10.1007/s10115-023-01959-7
Pang, Mei-Chin, Suraj Adhikari, Takuma Kasahara, Nagihiro Haba, and Saneyuki Ohno. “An Open-Access Benchmark of Statistical and Machine-Learning Anomaly Detection Methods for Battery Applications.” arXiv preprint arXiv:2511.01745 (2025).
GeeksforGeeks. Epoch in Machine Learning. https://www.geeksforgeeks.org/machine-learning/epoch-in-machine-learning/
Baeldung. Relation Between Learning Rate and Batch Size. https://www.baeldung.com/cs/learning-rate-batch-size