What is Dimensionality Reduction in Machine Learning? Features, Techniques & Implementation

Did you know? In 2024, John Hopfield and Geoffrey Hinton won the Nobel Prize for their work on neural networks, which revolutionized machine learning. Their innovation, particularly in deep learning, has led to the rise of techniques like autoencoders. 

This breakthrough enables dimensionality reduction in ways that were previously impossible, capturing non-linear data patterns that traditional methods couldn’t.

Dimensionality reduction in machine learning refers to the process of reducing the number of features in your dataset while maintaining its core information. High-dimensional data can slow down your models and make them harder to interpret. By applying dimensionality reduction methods, you can simplify your data, making it easier to handle and analyze. 

In this article, you’ll learn how these techniques work and how you can implement them to overcome common challenges and boost your machine learning models.

Managing complex datasets and optimizing workflows can overwhelm any business. Explore the Machine Learning Courses by upGrad to enhance your data processing and decision-making capabilities. Start building your skills today!

What is Dimensionality Reduction? Definition and Features

Dimensionality reduction in machine learning is a process that reduces the number of input variables (features) in a dataset while preserving its essential information. This technique helps to simplify complex datasets, making them more manageable and interpretable. 

Working with dimensionality reduction techniques goes beyond just applying methods to datasets. You need to understand how each technique processes data, assess its impact on model performance, and fine-tune it for the best results. Here are three programs that can help you:

• Executive Diploma in Data Science & AI with IIIT-B
• Executive Diploma in Machine Learning and AI with IIIT-B
• Executive Post Graduate Certificate Programme in Data Science & AI

What makes these methods unique is their ability to simplify complex, high-dimensional data while preserving essential patterns, allowing for more accurate and efficient analysis.

• Preserving Global and Local Data Relationships: Maintains both global structure and local proximity in non-linear datasets, ensuring complex relationships aren’t lost.
• Dimensionality Flexibility: Adapts to datasets with varying degrees of complexity, from simple linear patterns to intricate non-linear structures.
• Computational Efficiency in Large Datasets: Reduces the need for computationally expensive algorithms by minimizing the number of features while retaining key data points.
• Reduction of Redundant Information: Identifies and eliminates duplicate or highly correlated features, enhancing data efficiency.
• Feature Transformation for Improved Learning: Creates new, meaningful features (through extraction) that are more suitable for machine learning models than the original ones. 

To get started with dimensionality reduction in machine learning, first build a solid foundation in basic machine learning concepts and data preprocessing techniques. Understand key topics like data normalization, feature selection, and the importance of reducing high-dimensionality. 

Also Read: How to Choose a Feature Selection Method for Machine Learning

Once comfortable with these basics, let's look into specific dimensionality reduction techniques and approaches. 

Dimensionality Reduction Techniques: Approaches and Methods

The need for various dimensionality reduction techniques arises from the fact that not all datasets behave the same way. High-dimensional data can lead to computational inefficiencies, overfitting, and difficulty in finding meaningful patterns. 

Each technique serves a unique purpose depending on the data’s structure and the problem you're solving. Some methods excel at preserving variance, while others focus on maintaining class separability or visualizing complex relationships. 

Let’s look at the different approaches, each designed to address specific challenges: 

1. Feature Selection vs. Feature Extraction

• Feature Selection involves selecting a subset of the most relevant features from the original dataset, reducing dimensionality by eliminating irrelevant or redundant features. 

It retains the original features in their raw form, ensuring no transformation of the data.

• Feature Extraction creates new features by transforming the original data into a lower-dimensional space, typically using methods like PCA or autoencoders. 

This approach often results in features that are combinations of the original ones, designed to represent the underlying patterns better.

Also Read: How to Choose a Feature Selection Method for Machine Learning

1. Linear vs. Non-linear Methods

• Linear Methods assume that the relationships between features are linear and use mathematical transformations like projection to reduce dimensions while retaining as much variance as possible. Techniques like PCA and LDA fall into this category.
• Non-linear Methods are used when the relationships between features are complex and non-linear. 

These methods, such as t-SNE and Isomap, capture intricate data structures that linear techniques might miss, making them ideal for more complex datasets.

These approaches exist to address the varying complexity of data, and understanding them will help you select the right technique, ultimately improving your model’s performance and efficiency. Let’s explore these techniques in detail.

1. Principal Component Analysis (PCA)

Principal Component Analysis (PCA) is a linear reduction technique that transforms high-dimensional data into a lower-dimensional form while preserving as much variance as possible. 

By projecting the data onto new orthogonal axes (principal components), PCA identifies the directions in which the data varies the most. These new axes capture the essential features of the data, allowing you to reduce dimensions while retaining the most significant information.

How does PCA work? 

PCA works by identifying the principal components of the data, these are the directions where the data exhibits the most variance. The algorithm uses linear algebra to calculate the eigenvectors and eigenvalues of the data’s covariance matrix. 

The eigenvectors represent the directions of maximum variance (the principal components), and the eigenvalues represent the magnitude of the variance along those directions.

Here’s how the process typically works:

1. Standardization: The data is standardized (scaled) so that each feature has a mean of 0 and a standard deviation of 1. This step ensures that the features with larger ranges do not dominate the PCA process.
2. Covariance Matrix: A covariance matrix is calculated to understand how the features of the dataset vary together.
3. Eigenvalue and Eigenvector Computation: The eigenvectors and eigenvalues of the covariance matrix are computed. The eigenvectors determine the directions of maximum variance, and the eigenvalues indicate how much variance each eigenvector explains.
4. Selecting Principal Components: Based on the eigenvalues, we select the top k eigenvectors, where k is the number of dimensions we want to reduce the data to.
5. Projection: The data is then projected onto these principal components, creating a new set of features.

Step-by-Step Guide to Implement PCA (with Python code)

Here’s a hands-on guide to applying PCA using Python and the sklearn library. We'll use the Iris dataset for simplicity.

Step 1: Importing Libraries and Loading Data 

# Import necessary libraries
import numpy as np
import pandas as pd
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt

# Load the Iris dataset
iris = load_iris()
X = iris.data  # Features
y = iris.target  # Labels

Step 2: Standardizing the Data

PCA is sensitive to the scales of the features, so it’s crucial to standardize the data before applying PCA. 

# Standardizing the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Step 3: Applying PCA

Next, we apply PCA to reduce the dataset to 2 dimensions. We'll retain two principal components for visualization purposes. 

# Apply PCA
pca = PCA(n_components=2)  # Reducing to 2 components
X_pca = pca.fit_transform(X_scaled)

# Check the explained variance ratio
print("Explained Variance Ratio:", pca.explained_variance_ratio_)

Output:

Explained Variance Ratio: [0.92461872 0.05306648]

The explained variance ratio tells us how much of the total variance is captured by each principal component.

Step 4: Visualizing the Results

We can visualize the reduced data in a 2D plot. This allows us to see the data spread across the two principal components. 

# Visualizing the first two principal components
plt.figure(figsize=(8,6))
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, cmap='viridis')
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.title('PCA of Iris Dataset')
plt.colorbar(label='Target Class')
plt.show()

Output:

Step 5: Evaluating the Reduced Dimensions

You can check how much variance was captured by the selected principal components using the explained variance ratio.

# Variance explained by the first two components
total_variance = sum(pca.explained_variance_ratio_)
print(f"Total variance explained by the first two components: {total_variance:.2f}")

Output:

Total variance explained by the first two components: 0.98

This will give you an idea of how well PCA has reduced the dimensions while preserving the essential information.

If you're still building your Python skills, now is the perfect time to strengthen that foundation. Check out the Programming with Python: Introduction for Beginners free course by upGrad to build the foundation you need before getting into ML.

 2. Linear Discriminant Analysis (LDA)

Linear Discriminant Analysis (LDA) is a supervised reduction technique that focuses on maximizing the separability between classes in the data. Unlike PCA, which focuses on capturing the variance in the data without considering class labels, LDA aims to project the data onto a lower-dimensional space. 

Its primary goal is to preserve class distinctions and maximize the separation between different classes. This makes LDA particularly useful for classification tasks where separating different categories is key.

How does LDA work?

LDA works by finding a linear combination of features that best separates two or more classes. It does this by:

1. Computing the mean of each class and the overall mean of the data.
2. Calculating the between-class scatter matrix to measure the separation between class means.
3. Calculating the within-class scatter matrix to measure the variance within each class.
4. Maximizing the ratio of the between-class scatter to the within-class scatter, which results in new dimensions (discriminants) that provide the best separation of classes.

LDA's goal is to maximize class separability while reducing the dataset's dimensionality. By projecting the data onto a space defined by these discriminants, LDA improves the distinction between different classes. 

Step-by-Step Guide to Implement LDA (with Python code)

Here’s a step-by-step guide to applying LDA using Python, where we will use the Iris dataset again for simplicity.

Step 1: Importing Libraries and Loading Data 

# Import necessary libraries
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# Load the Iris dataset
iris = load_iris()
X = iris.data  # Features
y = iris.target  # Labels

Step 2: Standardizing the Data

As with PCA, it's essential to standardize the data before applying LDA. 

# Standardizing the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Step 3: Applying LDA

We will now apply LDA to reduce the dataset to 2 components, so we can visualize the class separability. 

# Apply LDA
lda = LinearDiscriminantAnalysis(n_components=2)
X_lda = lda.fit_transform(X_scaled, y)

# Check the explained variance ratio
print("Explained Variance Ratio:", lda.explained_variance_ratio_)

Output: 

Explained Variance Ratio: [0.9912126 0.0087874]

This output shows that the first linear discriminant explains about 99.12% of the variance, while the second one explains only 0.88%. LDA aims to capture as much class separability as possible with these two components.

Step 4: Visualizing the Results

We can now visualize the reduced dataset and check how well the classes are separated in the new 2D space. 

# Visualizing the reduced data
plt.figure(figsize=(8,6))
plt.scatter(X_lda[:, 0], X_lda[:, 1], c=y, cmap='viridis')
plt.xlabel('Linear Discriminant 1')
plt.ylabel('Linear Discriminant 2')
plt.title('LDA of Iris Dataset')
plt.colorbar(label='Target Class')
plt.show()

Output:

You can also evaluate the discriminative power of the reduced components by looking at the explained variance ratio. 

# Variance explained by the components
total_variance = sum(lda.explained_variance_ratio_)
print(f"Total variance explained by the first two components: {total_variance:.2f}")

Expected Output: 

Total variance explained by the first two components: 1.00

This means that the two discriminants chosen by LDA explain 100% of the variance in the data that is related to class separation, making LDA highly effective for this task.

3. t-Distributed Stochastic Neighbor Embedding (t-SNE)

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a non-linear reduction technique primarily used for visualizing high-dimensional data in lower dimensions, typically 2D or 3D. 

Unlike linear techniques like PCA, t-SNE focuses on preserving the local structure of the data, making it ideal for visualizing clusters or patterns that might be difficult to detect in higher dimensions. It's particularly useful when you want to gain insights into complex relationships or groupings in your data.

How does t-SNE Work?

t-SNE works by converting the Euclidean distances between data points in high-dimensional space into conditional probabilities. It minimizes the divergence between probability distributions, ensuring that points close in high-dimensional space stay close in the lower-dimensional representation.

The technique emphasizes preserving local relationships, so clusters of similar data points are kept together.

Here’s how the process works:

1. Pairwise Similarity: Calculate the pairwise similarity between points in the high-dimensional space, measuring how likely two points are to be neighbors.
2. Probability Distribution: Convert the pairwise distances into probabilities, where similar points have high probabilities of being neighbors.
3. Low-Dimensional Representation: Initialize random positions in a lower-dimensional space and adjust these positions iteratively to minimize the difference between the high-dimensional and low-dimensional probability distributions.
4. Cost Function Optimization: The algorithm optimizes a cost function, typically using gradient descent, to minimize the divergence (e.g., Kullback-Leibler divergence) between the probability distributions.

t-SNE is computationally expensive but provides exceptional results when visualizing high-dimensional data.

Step-by-Step Guide to Implement t-SNE (with Python code)

Here’s a detailed guide to applying t-SNE using Python. We’ll use the Iris dataset once again for simplicity.

Step 1: Importing Libraries and Loading Data 

# Import necessary libraries
import seaborn as sns
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# Load the Iris dataset
iris = sns.load_dataset('iris')
X = iris.drop('species', axis=1)  # Features
y = iris['species']  # Labels

Step 2: Standardizing the Data

Since t-SNE is sensitive to the scale of the data, it’s important to standardize the dataset before applying the technique. 

# Standardizing the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Step 3: Applying t-SNE

Now, we’ll apply t-SNE to reduce the dataset to 2 dimensions for easy visualization. 

# Apply t-SNE
tsne = TSNE(n_components=2, random_state=42)
X_tsne = tsne.fit_transform(X_scaled)

Step 4: Visualizing the Results

We can now visualize the reduced data in a 2D plot to examine how well the data points are grouped according to their species. 

# Visualizing the reduced data
plt.figure(figsize=(8,6))
plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y, cmap='viridis')
plt.xlabel('t-SNE Component 1')
plt.ylabel('t-SNE Component 2')
plt.title('t-SNE of Iris Dataset')
plt.colorbar(label='Target Class')
plt.show()

Output:

Although t-SNE can be computationally expensive, it provides an intuitive way to visualize high-dimensional data, revealing insights that may otherwise go unnoticed.

4. Isomap

Isomap is a non-linear reduction technique that extends classical multidimensional scaling (MDS) by incorporating geodesic distances between data points. 

Unlike PCA, which only considers linear relationships, Isomap is designed to capture the non-linear structure in data by preserving the intrinsic geometry of the dataset.  

How does Isomap work?

Isomap works by using the concept of geodesic distances to measure the true distance between points on a curved manifold. Here’s how the process works:

1. Constructing a Neighborhood Graph: The first step is to create a neighborhood graph where each data point is connected to its nearest neighbors based on a distance metric (e.g., Euclidean distance).
2. Calculating Geodesic Distances: Instead of using direct Euclidean distances, Isomap calculates the geodesic distance between each pair of data points, which is the shortest path along the manifold connecting them.
3. Multidimensional Scaling (MDS): Isomap then applies classical MDS to the matrix of geodesic distances. MDS aims to place the data points in a lower-dimensional space while preserving these distances as accurately as possible.
4. Dimension Reduction: After applying MDS, the data is mapped to a lower-dimensional space that retains the intrinsic geometry, capturing the complex non-linear relationships between points. 

Step-by-Step Guide to Implement Isomap (with Python code)

Here’s a step-by-step guide to applying Isomap using Python. We’ll continue using the Iris dataset for consistency.

Step 1: Importing Libraries and Loading Data 

# Import necessary libraries
from sklearn.manifold import Isomap
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# Load the Iris dataset
iris = load_iris()
X = iris.data  # Features
y = iris.target  # Labels

Step 2: Standardizing the Data

Since Isomap, like most techniques, is sensitive to the scale of the data, we need to standardize it. 

# Standardizing the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Step 3: Applying Isomap

Now, we’ll apply Isomap to reduce the dataset to 2 components for visualization purposes. 

# Apply Isomap
isomap = Isomap(n_components=2)
X_isomap = isomap.fit_transform(X_scaled)

# Visualizing the results
print("Isomap completed!") 

Step 4: Visualizing the Results

We can visualize the data in a 2D plot to observe how the points are distributed and whether Isomap has captured the non-linear structure effectively. 

# Visualizing the reduced data
plt.figure(figsize=(8,6))
plt.scatter(X_isomap[:, 0], X_isomap[:, 1], c=y, cmap='viridis')
plt.xlabel('Isomap Component 1')
plt.ylabel('Isomap Component 2')
plt.title('Isomap of Iris Dataset')
plt.colorbar(label='Target Class')
plt.show()

Output:

The colors represent the different Iris species, and you can observe how Isomap effectively captures the complex relationships and patterns in the data. This visualization highlights the non-linear structure preserved by Isomap during the reduction.

 5. Autoencoders

Autoencoders are a type of neural network used for unsupervised dimensionality reduction in machine learning, typically for learning efficient data codings in a lower-dimensional space. The primary goal of an autoencoder is to compress data from high-dimensional space into a compact representation (encoding) and then reconstruct it back to the original input. 

The network learns the most important features during training, which allows it to preserve the essential structure of the data while reducing its dimensionality.  

How do Autoencoders work?

Autoencoders consist of two main components:

1. Encoder: The encoder network compresses the input data into a lower-dimensional space, usually referred to as the latent space or bottleneck layer. It maps the input into a smaller representation.
2. Decoder: The decoder network attempts to reconstruct the input data from this compressed representation, ensuring the most important features are preserved.

During training, the network learns to minimize the reconstruction error, the difference between the original input and the output of the decoder. The bottleneck layer (latent space) represents the reduced, compressed form of the data, and by analyzing this space, we can gain insights into the data’s essential structure.

Autoencoders can be implemented with different architectures depending on the type of data (e.g., convolutional layers for images, recurrent layers for sequential data). 

Step-by-Step Guide to Implement Autoencoders (with Python code)

Let’s go through the steps to implement a simple autoencoder using the Iris dataset with the Keras library.

Step 1: Importing Libraries and Loading Data 

# Import necessary libraries
import numpy as np
import pandas as pd
from keras.layers import Input, Dense
from keras.models import Model
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# Load the Iris dataset
iris = load_iris()
X = iris.data  # Features
y = iris.target  # Labels

Step 2: Standardizing the Data

Standardize the dataset to ensure that the data has a mean of 0 and a standard deviation of 1, which is important for the neural network to learn effectively. 

# Standardizing the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Step 3: Building the Autoencoder

We will define a simple autoencoder with one hidden layer for encoding and one hidden layer for decoding. 

# Define the input layer
input_layer = Input(shape=(X_scaled.shape[1],))

# Define the encoder layer (bottleneck)
encoded = Dense(2, activation='relu')(input_layer)

# Define the decoder layer
decoded = Dense(X_scaled.shape[1], activation='sigmoid')(encoded)

# Define the autoencoder model
autoencoder = Model(input_layer, decoded)

# Define the encoder model for obtaining the compressed representation
encoder = Model(input_layer, encoded)

# Compile the model
autoencoder.compile(optimizer='adam', loss='mean_squared_error')

# Train the model
autoencoder.fit(X_scaled, X_scaled, epochs=100, batch_size=32, shuffle=True)

Step 4: Extracting the Latent Space (Encoded Data)

After training the autoencoder, we can use the encoder part of the model to extract the reduced data in the latent space (2D representation). 

# Get the encoded data (compressed representation)
X_encoded = encoder.predict(X_scaled)

# Visualizing the reduced data
plt.figure(figsize=(8,6))
plt.scatter(X_encoded[:, 0], X_encoded[:, 1], c=y, cmap='viridis')
plt.xlabel('Encoded Feature 1')
plt.ylabel('Encoded Feature 2')
plt.title('Autoencoder of Iris Dataset')
plt.colorbar(label='Target Class')
plt.show()

Output:

Unlike linear methods like PCA, autoencoders can capture complex, non-linear relationships and are highly effective for tasks like image compression, anomaly detection, and feature learning.

Now that you’re familiar with the different techniques, the next step is to apply them to your own datasets. Experiment with each method, evaluate their impact on model performance, and fine-tune them based on your data’s complexity. 

Dealing with complex non-linear data can be overwhelming without a solid understanding of deep learning. Check out the free Fundamentals of Deep Learning and Neural Networks course by upGrad to get a strong grasp on neural network techniques. Start learning today!

Also Read: The Ultimate Guide to Deep Learning Models in 2025: Types, Uses, and Beyond

Let’s examine the advantages and disadvantages of these methods, which will help you decide on the best approach for your project. 

Advantages and Disadvantages of Dimensionality Reduction

Dimensionality reduction techniques are powerful tools, but they come with their own set of strengths and limitations. For example, while PCA is great for preserving variance in linear datasets, it may not capture complex patterns in non-linear data, which methods like t-SNE or Isomap can handle better.

Understanding these nuances will help you decide when to prioritize computational efficiency over model accuracy, or when to apply more complex techniques for higher-quality insights. 

Now that you understand the pros and cons, start applying the reduction to real-life datasets. Test different techniques based on your data’s complexity and refine your approach. 

Also Read: 8 Pros of Decision Tree Regression in Machine Learning

After experimenting with the techniques, the next step is to explore how dimensionality reduction in machine learning is applied in real-life scenarios. 

Real-Life Use Cases of Dimensionality Reduction

In real-life scenarios, we often deal with massive amounts of data, and working with all of it can be overwhelming. By reducing the number of features or dimensions, we can focus on the most important aspects, improve efficiency, and make better decisions. 

Whether in healthcare, marketing, or finance, it helps uncover patterns and insights that might otherwise be hidden. Let’s look at some examples of these techniques making a real impact.

Now that you’ve seen how the reduction is applied in real-life scenarios, consider exploring specialized techniques like non-linear autoencoders or t-SNE variations for even more complex datasets.  

Also Read: Applications of ML: Real-World Use Cases & Benefits

You can also start integrating the reduction into your end-to-end machine learning pipelines, experimenting with real-time data for practical applications. 

 Become an Expert at Dimensionality Reduction with upGrad!

To gain proficiency in dimensionality reduction in machine learning, start by building a strong foundation in machine learning, data preprocessing, and feature selection. However, many learners find it hard to understand how to apply it effectively in real-life problems.

Trusted by thousands of learners worldwide, upGrad offers courses tailored to help you develop the technical expertise and strategic thinking required to excel in Machine Learning.

In addition to the courses mentioned, here are some more options to help you sharpen your skills and stand out in the field.

• Linear Regression - Step by Step Guide
• Unsupervised Learning: Clustering
• Analyzing Patterns in Data and Storytelling

Not sure where to go next with your ML journey? upGrad’s personalized career guidance can help you explore the right learning path based on your goals. You can also visit your nearest upGrad center and start hands-on training today!  

Expand your expertise with the best resources available. Browse the programs below to find your ideal fit in Best Machine Learning and AI Courses Online.

Best Machine Learning and AI Courses Online

Discover in-demand Machine Learning skills to expand your expertise. Explore the programs below to find the perfect fit for your goals.

In-demand Machine Learning Skills

Discover popular AI and ML blogs and free courses to deepen your expertise. Explore the programs below to find your perfect fit.

Popular AI and ML Blogs & Free Courses

References: 

1. https://www.nobelprize.org/prizes/physics/2024/press-release/