Isotonic Regression in Machine Learning: A Comprehensive Guide with Python Implementation

Isotonic regression is a technique used to fit a non-decreasing function to data. It’s commonly applied in fields like recommendation systems, where the relationship between inputs and outputs needs to be monotonic, such as predicting user preferences. 

A common challenge you might face is when your data doesn’t behave in a predictable, linear manner. 

In this article, you’ll look at how isotonic regression in machine learning can address this issue, helping you improve your model's accuracy and reliability.

Enhance your AI and machine learning skills with upGrad’s online machine learning courses. Specialize in deep learning, NLP, and much more. Take the next step in your learning journey! 

What is Isotonic Regression? Key Concepts and Implementation

In recommendation systems, where user preferences tend to increase over time, predictions must respect this order. Isotonic Regression is a technique used to fit a non-decreasing function to data, ensuring that the output never decreases as the input increases. 

Handling isotonic regression models in machine learning isn’t just about fitting the data. You need the right strategies and adjustments to optimize and fine-tune your models for accurate predictions. Here are three programs that can help you:

• Masters in Data Science Degree
• Executive Diploma in Data Science & AI with IIIT-B
• Executive Diploma in Machine Learning and AI with IIIT-B

Let’s consider something familiar: setting the price of a product. You probably know that when a product has more features, it usually comes at a higher price. But the relationship isn’t always a straight line; sometimes the price might jump, level off, or rise more slowly based on the specific features. 

That’s where Isotonic Regression in machine learning can make a big difference.

Key Concepts Behind Isotonic Regression

Grasping these key concepts will show you how isotonic regression keeps your data on track, ensuring predictions align with real-world trends and remain reliable. Let’s break down the core concepts:

• Monotonicity: 

This is the heart of isotonic regression. Simply put, it means that as one variable increases, the other should either increase or stay the same. 

Let’s say you’re tracking the number of hours someone spends on fitness training each week. The more time someone spends, the more calories they’ll burn. Isotonic regression in machine learning ensures that as training time increases, the calories burned never decrease.

Also Read: Understanding Multivariate Regression in Machine Learning: Techniques and Implementation

• Pool Adjacent Violators Algorithm: 

This is the powerful algorithm that enables isotonic regression in machine learning to function. It works by adjusting data points that don’t meet the monotonicity requirement (the “violators”) until they do. 

In a stock market scenario, if a stock price prediction for tomorrow is lower than today’s prediction, isotonic regression smooths out these "violations" by making sure future predictions always stay at the same or a higher price.

Also Read: 18 Types of Regression in Machine Learning You Should Know [Explained With Examples]

• Fitting the Non-Decreasing Curve: 

Unlike linear regression that fits a straight line to data, isotonic regression fits a non-decreasing curve. This curve doesn’t have to be a straight line, it can bend and flex based on the data, but it will always move in one direction. 

Think of it as trying to fit a rubber band around a graph: it will stretch as much as needed but won’t snap back.

In e-commerce, imagine the average rating of products as you increase the number of reviews. As the number of reviews grows, the average rating can either stay the same or go up, but it can never drop, just like isotonic regression fits a curve that honors that growth.

Now that we’ve covered the key concepts, let’s walk through a real-life example of how you can implement isotonic regression in machine learning. 

Implementing Isotonic Regression in Machine Learning

For this example, we'll predict sales based on advertising spend, where we know that more money spent on advertising will likely lead to more sales, but not in a perfectly linear fashion. 

Step 1: Import Required Libraries

We’ll use scikit-learn, numpy, and matplotlib for this implementation. Scikit-learn provides the isotonic regression functionality, while numpy helps with data manipulation, and matplotlib will be used for visualizing our results. 

import numpy as np
import matplotlib.pyplot as plt
from sklearn.isotonic import IsotonicRegression

Step 2: Prepare the Data

Let’s create a simple dataset where the advertising spend increases, but the sales don't increase at a constant rate. This is a typical scenario where isotonic regression is helpful.

# Example data: Advertising spend (in thousands) vs Sales (in thousands)
ad_spend = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])  # Ad spend in thousands
sales = np.array([1, 2, 2.5, 3, 3.2, 3.5, 3.8, 4.2, 4.5, 5])  # Sales in thousands

Step 3: Apply Isotonic Regression

Now, let’s apply the isotonic regression model to the data. We’ll train the model to respect the monotonic relationship between advertising spend and sales. 

# Initialize isotonic regression model
iso_reg = IsotonicRegression(increasing=True)

# Fit the model and transform the data
sales_pred = iso_reg.fit_transform(ad_spend, sales)

Step 4: Visualize the Results

To better understand how isotonic regression fits the data, we’ll plot the original data points and the regression line. 

# Plot the original data
plt.scatter(ad_spend, sales, color='blue', label='Original Data')

# Plot the isotonic regression output
plt.plot(ad_spend, sales_pred, color='red', label='Isotonic Regression Line')

# Labels and title
plt.xlabel('Advertising Spend (Thousands)')
plt.ylabel('Sales (Thousands)')
plt.title('Sales vs Advertising Spend - Isotonic Regression')
plt.legend()

# Show plot
plt.show()

Output:

Interpretation 

• Non-linear growth: Sales increase with advertising spend, but at varying rates.
• Monotonicity preserved: Sales never decrease as advertising spend increases.
• Better fit: The red curve provides a more accurate trend.
• Real-world use: Reflects how increased investment leads to higher returns, but not uniformly.

Once you’ve implemented isotonic regression, the next step is to fine-tune your model to ensure it performs optimally. 

How to Tune Isotonic Regression Models?

Once you’ve implemented the model, simply fitting the data isn’t enough. The model might not perform as expected. For instance, if the model is too flexible, it may create unrealistic predictions that don’t align with the real-life data, like forecasting a decrease in sales as ad spend increases.

This is where tuning comes in. It ensures the model captures the true relationship between advertising spend and sales, avoiding noise and underfitting.

Let’s look at the necessary adjustments to enhance your model’s performance.

1. Increasing vs. Decreasing: 

One of the first decisions you’ll need to make is whether your model should fit a strictly increasing or decreasing function.

The increasing parameter in the IsotonicRegression class from scikit-learn controls this. Setting it to True forces the regression line to be non-decreasing, while setting it to False forces a non-increasing regression. 

The direction of monotonicity depends on the nature of the data you're working with. 

iso_reg = IsotonicRegression(increasing=True)

Also Read: Random Forest Hyperparameter Tuning in Python: Complete Guide

2. Out-of-bounds behavior:

In real-life data, it’s common to encounter values that fall outside the range of the data used to train the isotonic regression model. The out_of_bounds parameter allows you to control how the model behaves when it encounters data points outside the fitted range.

• ‘clip’: This option clips any out-of-bounds data points to the nearest boundary value.
• ‘nan’: This returns a NaN for out-of-bounds predictions, which can be useful if you want to signal that the model can’t make a valid prediction outside the trained range.

iso_reg = IsotonicRegression(out_of_bounds='clip')

Also Read: Optimizing Data Mining Models: Key Steps for Enhancing Accuracy and Performance

3. Optimizing for Accuracy

When tuning isotonic regression, you’ll also need to optimize for accuracy. Isotonic regression is sensitive to overfitting and underfitting, so it's important to adjust your model carefully to avoid both.

• Overfitting: 

Overfitting happens when the model captures noise instead of the true trend, often in smaller datasets or overly flexible models. Although isotonic regression is more robust than linear regression, it can still overfit with noisy data.

Solution: Use cross-validation to evaluate performance on unseen data, simplify the model by reducing input features, and consider adding regularization. Adjusting the training set size can also help ensure it’s representative.

• Underfitting: 

Underfitting happens when the model doesn’t capture the data’s underlying trend, often due to restrictive parameters or insufficient data. If the regression line is too flat or lacks variability, it's likely underfitting.

Solution: Increase the training data, adjust isotonic regression parameters for a better fit, or explore more complex models and additional features to better explain the data.

Also Read: The Data Science Process: Key Steps to Build Data-Driven Solutions

4. Handling Larger Datasets

Isotonic regression is relatively efficient, but when dealing with large datasets, it can become computationally expensive. Here are some tips for scaling isotonic regression models for larger or more complex datasets:

• Use Batch Processing: 

For very large datasets, it can be useful to break the data into smaller batches and fit isotonic regression on each batch separately. Afterward, you can merge the results. 

This can help speed up training while ensuring the model doesn’t run out of memory.

• Feature Scaling: 

While isotonic regression itself doesn’t require feature scaling, scaling your input data (such as normalizing values between 0 and 1) can sometimes improve the model’s performance, especially when your data has outliers or extreme values.

• Use Efficient Data Structures: 

If you’re working with a large dataset, consider using more memory-efficient data structures like sparse matrices or libraries like Dask for parallel computing. This can help process large datasets more efficiently without overwhelming system memory.

Also Read: 10 Best Data Structures for Machine Learning Model Optimization in 2025

Now that you’ve grasped the basics of isotonic regression, start by applying isotonic regression to your own datasets, fine-tuning the model for better performance. As you experiment, consider evaluating your model using cross-validation to ensure its generalization on unseen data.

To explore further, try combining isotonic regression with ensemble methods to boost model performance. You can also explore monotonic neural networks to handle complex, non-linear relationships in large-scale data.

Advance Your Machine Learning Skills with upGrad!

Projects like building a recommendation system or predicting sales based on advertising spend offer unique learning experiences with isotonic regression. These models teach you how to capture non-decreasing relationships in data, ensuring that your predictions align with real-world trends. However, applying isotonic regression comes with its own set of challenges.

To truly excel, focus on understanding the underlying trends in your data and use cross-validation to ensure your model’s strength. For further growth, learn advanced techniques like combining isotonic regression in machine learning with ensemble methods or exploring monotonic neural networks.

In addition to the courses mentioned above, here are some more free courses that can help you enhance your skills:  

• Advanced Prompt Engineering Course with ChatGPT
• JavaScript Basics from Scratch
• Artificial Intelligence in the Real World

Feeling uncertain about your next step? Get personalized career counseling to identify the best opportunities for you. Visit upGrad’s offline centers for expert mentorship, hands-on workshops, and networking sessions to connect you with industry leaders!

Expand your expertise with the best resources available. Browse the programs below to find your ideal fit in Best Machine Learning and AI Courses Online.

Best Machine Learning and AI Courses Online

Discover in-demand Machine Learning skills to expand your expertise. Explore the programs below to find the perfect fit for your goals.

In-demand Machine Learning Skills

Discover popular AI and ML blogs and free courses to deepen your expertise. Explore the programs below to find your perfect fit.

Popular AI and ML Blogs & Free Courses

References:
https://arxiv.org/abs/2411.06342 
https://arxiv.org/pdf/2411.06342