Homebreadcumb forward arrow iconBlogbreadcumb forward arrow iconArtificial Intelligencebreadcumb forward arrow iconGradient Descent Algorithm: Methodology, Variants & Best Practices

Gradient Descent Algorithm: Methodology, Variants & Best Practices

Last updated:
12th Jun, 2023
Read Time
6 Mins
share image icon
In this article
Chevron in toc
View All
Gradient Descent Algorithm: Methodology, Variants & Best Practices

Optimization is an integral part of machine learning. Almost all machine learning algorithms have an optimization function as a crucial segment. As the word suggests, optimization in machine learning is finding the optimal solution to a problem statement. 

Best Machine Learning and AI Courses Online

In this article, you’ll read about one of the most widely used optimization algorithms, gradient descent. The gradient descent algorithm can be used with any machine learning algorithm and is easy to comprehend and implement. So, what exactly is gradient descent? By the end of this article, you’ll have a clearer understanding of the gradient descent algorithm and how it can be used to update the model’s parameters.

In-demand Machine Learning Skills

Ads of upGrad blog

Get Machine Learning Certification from the World’s top Universities. Earn Masters, Executive PGP, or Advanced Certificate Programs to fast-track your career.

Gradient Descent

Before going deep into the gradient descent algorithm, you should know what cost function is. The cost function is a function used to measure the performance of your model for a given dataset. It finds the difference between your predicted value and expected value, thus quantifying the error margin.

The goal is to reduce the cost function so that the model is accurate. To achieve this goal, you need to find the required parameters during the training of your model. Gradient descent is one such optimization algorithm used to find the coefficients of a function to reduce the cost function. The point at which cost function is minimum is known as global minima.


Machine learning models and neural networks are frequently trained using the optimization algorithm known as gradient descent. The cost function in gradient descent especially serves as a barometer, measuring the model’s accuracy with each iteration of parameter updates. Training data is used to assist these models learning over time. The model will keep altering its parameters until the function is close to or equal to zero to produce the least inaccuracy. Machine learning models can be effective tools for computer science and artificial intelligence (AI) applications once they are accuracy-optimized.

Before getting into the code, one more concept needs to be defined: what is a gradient? It is intuitively understood to be the slope of a curve at a given position in a particular direction. It is just the first derivative at a particular location in the case of a univariate function.

The intuition behind the Gradient Descent algorithm

Suppose you have a large bowl similar to something you’ve your fruit in. This bowl is the plot for the cost function. The bottom of the bowl is the best coefficient for which the cost function is minimum. Different values are used as the coefficients to calculate the cost function. This step is repeated until the best coefficients are found.

You can imagine gradient descent as a ball rolling down a valley. The valley is the plot for the cost function here. You want the ball to reach the bottom of the valley, where the bottom of the valley represents the least cost function. Depending on the start position of the ball, it may rest on many bottoms of the valley. However, these bottoms may not be the lowest points and are known as local minima. 


Read: Boosting in Machine Learning: What is, Functions, Types & Features

Gradient Descent Algorithm- Methodology

The calculation of gradient descent begins with the initial values of coefficients for the function being set as 0 or a small random value. 

coefficient = 0 (or a small value)

  • The cost function is calculated by putting this value of the coefficient in the function.

Cost function = f(coefficient) 

  • We know from the concept of calculus that the derivative of a function is the slope of the function. Calculating the slope will help you to figure out the direction to move the coefficient values. The direction should be such that you get a lower cost(error) in the next iteration. 

del = derivative(cost function)

  •  After knowing the direction of downhill from the slope, you update the coefficient values accordingly. A learning rate (alpha) can be selected to control how much these coefficients will change in each iteration. You need to make sure that this learning rate is not too high nor too low. 

coefficient = coefficient – (alpha * del)

  • This process is repeated until the cost function becomes 0 or very close to 0. 

f(coefficient) = 0 (or close to 0)

The selection of the learning rate is important. Selecting a very high learning rate can overshoot the global minima. On the contrary, a very low learning rate can help you reach the global minima, but the convergence is very slow, taking many iterations.


Variants of Gradient Descent Algorithm

Batch Gradient Descent

Batch gradient descent is one of the most used variants of the gradient descent algorithm. The cost function is computed over the entire training dataset for every iteration. One batch is referred to as one iteration of the algorithm, and this form is known as batch gradient descent. 

Stochastic Gradient Descent

In some cases, the training set can be very large. In these cases, batch gradient descent will take a long time to compute as one iteration needs a prediction for each instance in the training set. You can use the stochastic gradient descent in these conditions where the dataset is huge. In stochastic gradient descent, the coefficients are updated for each training instance and not at the end of the batch of instances.

Mini Batch Gradient Descent

Both batch gradient descent and stochastic gradient descent have their pros and cons. However, using a mixture of batch gradient descent and stochastic gradient descent can be useful. In mini-batch gradient descent, neither the entire dataset is used nor do you use a single instance at a time. You take into consideration a group of training examples. The number of examples in this group is lesser than the entire dataset, and this group is known as a mini-batch. 

Popular AI and ML Blogs & Free Courses

Best Practices for Gradient Descent Algorithm

  • Map cost versus time: Plotting the cost with respect to time helps you visualize whether the cost is decreasing or not after each iteration. If you see the cost to remain unchanged, try updating the learning rate. 
  • Learning rate: The learning rate is very low and is often selected as 0.01 or 0.001. You need to try and see which value works best for you.
  • Rescale inputs: The gradient descent algorithm will minimize the cost function faster if all the input variables are rescaled to the same range, such as [0, 1] or [-1, 1].
  • Less passes: Usually, the stochastic gradient descent algorithm doesn’t need more than 10 passes to find the best coefficients. 

Check out: 25 Machine Learning Interview Questions & Answers

When To Use Gradient Descent Algorithm?

Ads of upGrad blog

When parameters need to be found via an optimization technique but cannot be determined analytically (for example, using linear algebra), gradient descent is the method of choice.

Advantages Of Gradient Descent Algorithm

  • Takes advantage of vectorization’s advantages.
  • Towards the minimum, a more direct route is adopted.
  • Since updates are needed after an epoch has run, computations must be efficient.
  • It is simpler to fit into the memory that has been allocated.
  • Gradient descent convergence is produced that is stable.

Disadvantages Of Gradient Descent Algorithm

  • Can converge at nearby saddle and minima sites.
  • Slower learning since an update is only made after we have examined every observation.
  • For huge datasets, perform duplicate computations for the same training sample.
  • Large datasets might not fit in the memory, making it exceedingly slow and difficult to solve.
  • We can update the model’s weights with the fresh data because we compute the full dataset.

Wrapping up

You get to know the role of gradient descent in optimizing a machine learning algorithm. One important factor to keep in mind is choosing the right learning rate for your gradient descent algorithm for optimal prediction.

upGrad provides a PG Diploma in Machine Learning and AI and a  Master of Science in Machine Learning & AI that may guide you toward building a career. These courses will explain the need for Machine Learning and further steps to gather knowledge in this domain covering varied concepts ranging from gradient descent algorithms to Neural Networks. 


Pavan Vadapalli

Blog Author
Director of Engineering @ upGrad. Motivated to leverage technology to solve problems. Seasoned leader for startups and fast moving orgs. Working on solving problems of scale and long term technology strategy.
Get Free Consultation

Select Coursecaret down icon
Selectcaret down icon
By clicking 'Submit' you Agree to  
UpGrad's Terms & Conditions

Our Popular Machine Learning Course

Frequently Asked Questions (FAQs)

11. What concept underlies the Gradient Descent?

To locate a differentiable function's local minimum, an optimization process known as a gradient descent is used. The basic goal of gradient descent in Machine Learning is to move in the gradient's opposite direction. This will result in the steepest decline and, ultimately, the lowest point.

22. What is the principal drawback of the gradient descent algorithm?

The Gradient descent algorithm has the drawback that the weight update at a given time (t) is solely determined by the learning rate and gradient. The previous steps when navigating the cost space are not considered.

33. What are some popular Gradient Descent methods?

Some popular approaches to gradient descent in Machine Learning are: Batch gradient descent: a method in which the complete training set is considered before moving forward. Convex and largely smooth describes its cost function. Stochastic Gradient Descent: A single decision is made by only considering one piece of data. Its cost function fluctuates, but as more iterations go by, the fluctuations eventually get smaller. Mini-batch Gradient Descent: where a batch of a fixed number of data is considered. Its cost function is also a fluctuation one.

Explore Free Courses

Suggested Blogs

Data Preprocessing in Machine Learning: 7 Easy Steps To Follow
Summary: In this article, you will learn about data preprocessing in Machine Learning: 7 easy steps to follow. Acquire the dataset Import all the cr
Read More

by Kechit Goyal

29 Oct 2023

Natural Language Processing (NLP) Projects & Topics For Beginners [2023]
What are Natural Language Processing Projects? NLP project ideas advanced encompass various applications and research areas that leverage computation
Read More

by Pavan Vadapalli

04 Oct 2023

15 Interesting MATLAB Project Ideas & Topics For Beginners [2023]
Learning about MATLAB can be tedious. It’s capable of performing many tasks and solving highly complex problems of different domains. If youR
Read More

by Pavan Vadapalli

03 Oct 2023

Top 16 Artificial Intelligence Project Ideas & Topics for Beginners [2023]
Summary: In this article, you will learn the 16 AI project ideas & Topics. Take a glimpse below. Predict Housing Price Enron Investigation Stock
Read More

by Pavan Vadapalli

27 Sep 2023

Top 15 Deep Learning Interview Questions & Answers
Although still evolving, Deep Learning has emerged as a breakthrough technology in the field of Data Science. From Google’s DeepMind to self-dri
Read More

by Prashant Kathuria

21 Sep 2023

Top 8 Exciting AWS Projects & Ideas For Beginners [2023]
AWS Projects & Topics Looking for AWS project ideas? Then you’ve come to the right place because, in this article, we’ve shared multiple AWS proj
Read More

by Pavan Vadapalli

19 Sep 2023

Top 15 IoT Interview Questions & Answers 2023 – For Beginners & Experienced
These days, the minute you indulge in any technology-oriented discussion, interview questions on cloud computing come up in some form or the other. Th
Read More

by Kechit Goyal

15 Sep 2023

45+ Interesting Machine Learning Project Ideas For Beginners [2023]
Summary: In this Article, you will learn Stock Prices Predictor Sports Predictor Develop A Sentiment Analyzer Enhance Healthcare Prepare ML Algorith
Read More

by Jaideep Khare

14 Sep 2023

Why GPUs for Machine Learning? Ultimate Guide
In the realm of modern technology, the convergence of data and algorithms has paved the way for groundbreaking advancements in artificial intelligence
Read More

by Pavan Vadapalli

14 Sep 2023

Schedule 1:1 free counsellingTalk to Career Expert
footer sticky close icon