Backpropagation in Machine Learning: How It Works, Formula & Example

In simple words:

1. The neural network makes a prediction.
2. It checks how far the prediction is from the correct answer.
3. Backpropagation finds which weights need to change and by how much.

Why Is Backpropagation Used in Neural Networks?

A neural network has many connected parts, and each weight can affect its final prediction. Backpropagation helps train the network by solving three main problems:

• Faster training: It calculates gradients in an efficient way. This makes it possible to train neural networks with millions or even billions of parameters.
• Accurate error calculation: It calculates how the loss changes when a weight or bias changes. These gradients help the training process choose how to adjust the network.
• Works with different networks: It can be used with many types of neural networks, including simple networks, image recognition networks, and large language models, as long as their operations can be differentiated.

Backpropagation and gradient descent work together, but they do different jobs. Backpropagation calculates the gradients, while gradient descent uses them to update the weights and reduce the loss.

How Does Backpropagation Work?

Backpropagation works in four steps. The neural network makes a prediction, checks its error, finds which weights need to change, and updates them. This process repeats until the network makes better predictions.

Step 1: Forward Pass: Making a Prediction

1. The network takes the input and passes it through each layer.
2. Each neuron uses weights and a bias to process the input.
3. An activation function, such as ReLU or sigmoid, helps the neuron produce an output.
4. The final layer gives the prediction.

Step 2: Calculating the Loss

The loss tells us how far the prediction is from the correct answer.

1. The network compares its prediction with the correct answer.
2. A loss function calculates the error as a number.
3. A lower loss means the prediction is closer to the correct answer.

Mean squared error is used for many number-based predictions. Cross-entropy is used for many classification tasks.

Step 3: Backward Pass: Finding the Error

1. Backpropagation starts at the output layer and moves backward through the network.
2. Then checks how much each weight and bias affects the error.
3. It calculates gradients, which help show how the weights and biases need to change.

Step 4: Updating Weights to Reduce Error

1. The network uses the gradients to update its weights.
2. The earning rate controls how big each change will be.
3. The network then makes another prediction with the updated weights.
4. This process repeats to help reduce the error.

Backpropagation Formula in Machine Learning

The backpropagation formula helps a neural network find how much each weight affects the error. It uses the chain rule to calculate this effect. Gradient descent then uses these results to update the weights.

The Chain Rule Formula

The chain rule helps find how much a small change in a weight affects the final error. It breaks the calculation into smaller steps.

The formula is:

(Lw=Laazzw)

In simple words, this formula checks three things:

• How much the loss changes when the neuron's output changes.
• How much the output changes when the weighted sum changes.
• How much the weighted sum changes when the weight changes.

1. For an output neuron using the sigmoid activation function and squared error loss, the error signal is:

Here, the error signal shows how much the loss changes when the neuron's weighted sum changes.

2. The gradient for a weight connected to this output neuron is

                                                                                                                Lw=outaprev

This means the gradient is the error signal multiplied by the output of the previous neuron.

Also Read: Accuracy Formula in Machine Learning 

3. For a hidden neuron, backpropagation uses the error signals from the next layer to calculate its own error signal:

hidden=kwkka(1-a)

This helps the network find how much each hidden neuron contributes to the final error.

Weight Update Formula

After calculating the gradient, the network updates its weights using gradient descent.

The formula is:

wnew=wold-Lw

In simple words:

• wold is the current weight.
• is the learning rate. It controls how big the change will be.
• Lw is the gradient. It shows how the weight affects the loss.
• wnew is the updated weight.

The network subtracts the gradient from the current weight, multiplied by the learning rate. This helps reduce the loss during training.

Biases are updated in a similar way. The main difference is that the bias update uses the error signal directly.

Also Explore: RMSE in Machine Learning: Formula, Calculation, and Real-World Use

What Do the Variables in the Formula Mean?

The formulas use a few symbols to explain how backpropagation works. Here is what each symbol means.

Backpropagation Algorithm in Machine Learning

The backpropagation algorithm helps a neural network learn from its mistakes. It follows a set of steps to find errors, calculate how much each weight affects them, and update the weights. The network repeats these steps many times to improve its predictions.

Steps in the Backpropagation Algorithm

1. Initialize the weights: Start with small random values for the weights. Biases are often set to zero.
2. Make a prediction: Pass the input through the network, one layer at a time.
3. Calculate the loss: Compare the prediction with the correct answer to find the error.
4. Find the output error: Calculate how much the output layer contributes to the loss.
5. Pass the error backward: Move through the earlier layers to find how much each neuron contributes to the error.
6. Calculate the gradients: Find how much each weight affects the loss.
7. Update the weights: Use gradient descent to adjust the weights and biases.
8. Repeat the steps: Run the process again with the training data. Keep checking the loss to see if the network is improving.

Also Read: How to Implement Machine Learning Steps: A Complete Guide

How Backpropagation and Gradient Descent Work Together?

Backpropagation and gradient descent work together, but they have different jobs.

• Backpropagation calculates how each weight affects the error.
• Gradient descent uses these results to update the weights and reduce the error.

The learning rate controls how big each weight update will be. Other optimizers, such as SGD, Momentum, RMSprop, and Adam, use gradients in different ways. Backpropagation calculates the gradients they need.

A Common Challenge: Vanishing and Exploding Gradients

During backpropagation, the network passes gradients through many layers. Sometimes, these gradients become too small or too large.

• Vanishing gradients: The gradients become so small that the early layers learn very slowly.
• Exploding gradients: The gradients become too large, making training unstable.

Some ways to handle these problems include:

1. Using ReLU activation functions to help reduce vanishing gradients.
2. Choosing suitable weight values at the start of training.
3. Using batch normalization or skip connections to help training.
4. Using gradient clipping to control exploding gradients.

Want to learn logistic regression and gradient descent? Explore Executive Diploma in Data Science & AI with IIIT-B program to build your skills and get started!

Backpropagation Example in Machine Learning

Let's understand backpropagation with a simple example. We will use a small neural network to see how it makes a prediction, finds the error, and updates its weights.

A Simple Neural Network Example

Our neural network has two input values, two hidden neurons, and one output neuron. Each neuron uses the sigmoid activation function. We will use a learning rate of 0.5 to update the weights.

Here are the values we will use in this example.

Step-by-Step Calculation of Error and Weight Updates

Step 1: Forward pass.

• z1 = 0.15 × 0.5 + 0.25 × 0.8 = 0.2750, so h1 = sigmoid(0.2750) = 0.5683
• z2 = 0.20 × 0.5 + 0.30 × 0.8 = 0.3400, so h2 = sigmoid(0.3400) = 0.5842
• z_out = 0.40 × 0.5683 + 0.50 × 0.5842 = 0.5194, so o = sigmoid(0.5194) = 0.6270

Step 2: Loss. L = ½(1.0 − 0.6270)² = 0.0696

Step 3: Output error signal.

• ∂L/∂o = o − y = 0.6270 − 1.0 = −0.3730
• ∂o/∂z_out = o(1 − o) = 0.2339
• δ_out = −0.3730 × 0.2339 = −0.0872

Step 4: Gradients for the output weights.

• ∂L/∂w5 = δ_out × h1 = −0.0872 × 0.5683 = −0.0496
• ∂L/∂w6 = δ_out × h2 = −0.0872 × 0.5842 = −0.0510

Step 5: Push the error back to the hidden layer.

• δ_h1 = (w5 × δ_out) × h1(1 − h1) = (0.40 × −0.0872) × 0.2453 = −0.0086
• δ_h2 = (w6 × δ_out) × h2(1 − h2) = (0.50 × −0.0872) × 0.2429 = −0.0106

Step 6: Gradients for the hidden weights.

• ∂L/∂w1 = δ_h1 × x1 = −0.0043 and ∂L/∂w2 = δ_h1 × x2 = −0.0068
• ∂L/∂w3 = δ_h2 × x1 = −0.0053 and ∂L/∂w4 = δ_h2 × x2 = −0.0085

Step 7: Update the weights with w_new = w_old − 0.5 × gradient.

Step 8: Check the result. With the new weights, the network makes a better prediction. Its output increases from 0.6270 to 0.6340, moving closer to the correct answer of 1.0. The loss also drops from 0.0696 to 0.0670. This means the network has made a small improvement. It repeats this process thousands of times to keep improving its predictions. 

Are you a working professional and  want to  accelerate your career? Enroll now in NIRF ranked Ex. Diploma in Machine Learning & AI.

Conclusion:

Backpropagation in machine learning helps a neural network learn from its mistakes. The network makes a prediction, checks how far it is from the correct answer, and finds which weights need to change.

It uses the chain rule to calculate how each weight affects the error. Then, it makes small changes to the weights to improve the next prediction.

Tools like Adam help the network update its weights during training. These tools build on the same basic process.

The 2-2-1 example above helps you understand how backpropagation works step by step. You can try coding the same example in NumPy to practise. You can also use PyTorch, which calculates gradients and runs the backward pass for you.

Want personalized guidance on AI and upskilling? Speak with an expert for a free 1:1 counselling session today.