Binary Logistic Regression: Concepts, Implementation, and Applications

Binary logistic regression models the relationship between predictor variables and a binary outcome, predicting discrete outcomes like cancer diagnosis or customer churn. 

Unlike linear regression, which predicts continuous values, logistic regression utilizes the logistic function to output probabilities ranging from 0 to 1. 

It is estimated using Maximum Likelihood Estimation (MLE), which maximizes the likelihood of observing the given data, making it ideal for classification tasks.

In this blog, we'll explore the key concepts, implementation steps, and practical applications of binary logistic regression.

What is Binary Logistic Regression?

Binary logistic regression is a statistical method used for modeling the relationship between a binary dependent variable and one or more independent variables. It predicts the probability of the dependent variable taking one of two possible outcomes, typically coded as 0 or 1. 

This method is commonly applied in fields such as healthcare, marketing, and the social sciences for classification tasks, including determining the likelihood of an event (e.g., disease occurrence or customer churn).

Ready to elevate your AI skills with binary logistic regression? Explore upGrad’s top programs, designed to help you master logistic regression, gain hands-on experience, and solve real-world challenges in AI and machine learning.

• Executive Diploma in Machine Learning and AI
• Master’s Degree in Artificial Intelligence and Data Science
• Master of Science in Machine Learning & AI

Definition and Key Concepts

Binary logistic regression is based on the logistic function, which transforms a linear combination of input variables into a probability score ranging from 0 to 1. The model estimates the odds of an event occurring, expressed as a ratio of different probability distributions

The coefficients in the logistic regression equation represent the log-odds of the dependent variable for a one-unit change in the predictor variables.

Key concepts:

• Logistic function: Converts linear combinations of input variables into a probability score between 0 and 1.
• Odds and log-odds: The odds refer to the ratio of the probability of the event occurring to the probability of it not occurring, while log-odds are the natural logarithm of the odds.
• Maximum likelihood estimation (MLE): The method used for estimating model parameters by maximizing the likelihood of observing the data.

Understand the foundations of linear regression and its connection to logistic regression with the Linear Regression - Step by Step Guide course. Apply these concepts to solve data problems and build predictive models.

Now that we've covered the basics of BLR, let's explore regression analysis as the broader framework that underpins models like BLR.

What is Regression Analysis?

Regression analysis is a statistical technique used to model and analyze the relationship between a dependent variable and one or more independent variables. The purpose is to predict the dependent variable or to understand the underlying data patterns. 

Logistic regression is a specialized form of regression analysis where the dependent variable is categorical, typically a binary variable.

Regression methods differ in their approach depending on the nature of the dependent variable:

• Linear regression: Used when the dependent variable is continuous.
• Logistic regression: Applied when the dependent variable is categorical (e.g., binary or multinomial).

Also Read: Different Types of Regression Models You Need to Know

Binary vs Multinomial Logistic Regression

Binary logistic regression is used when the dependent variable has two possible outcomes, whereas multinomial logistic regression is used when the dependent variable has more than two categories. 

Understanding the distinction between these two models is critical for choosing the appropriate approach for a given problem. 

The following table summarizes the key differences between binary and multinomial logistic regression:

Combine generative AI with binary logistic regression in the Gen AI Mastery Certificate for Data Analysis course. Gain practical skills to build more efficient models and optimize predictions for real-world applications.

Also Read: How to Perform Multiple Regression Analysis?

Now, we’ll explore the mathematical foundation and practical applications of binary logistic regression, focusing on the logistic function and its use in binary predictions.

Mathematical Foundation and Practical Application of Binary Logistic Regression

Binary logistic regression is widely used to predict binary outcomes based on one or more independent variables. It is particularly effective in scenarios that require classification, such as medical diagnosis, financial risk management strategies, and customer behavior prediction. 

The model applies the logistic function to map inputs to probabilities, making it a go-to tool for decision-making across various industries.

The Mathematical Principles Behind Binary Logistic Regression

The equation for binary logistic regression is a transformation of the linear regression model, ensuring that the prediction falls within the 0 to 1 range.

Model Equation:

Where,

• P(Y=1|X) is the estimated probability of the event occurring.
• X1, X2, ….Xn are the independent variables (predictors)
• B0, B1, B2, …..Bn are the coefficients, representing the influence of each predictor.

This formula transforms a linear combination of input variables into a probability score, using the sigmoid function, which is essential for modeling binary outcomes such as "pass/fail" or "yes/no".

Also Read: Top 5 Machine Learning Models Explained For Beginners 

The Role of Probability and Odds in Logistic Models

Logistic regression uses odds and probabilities to model binary outcomes. The odds represent the likelihood of an event relative to its complement. 

The transformation of these odds into log-odds is what makes logistic regression a powerful method for classification.

Odds and Log-Odds:

• Odds: The ratio of the probability of an event happening to the probability of it not happening.
• Log-Odds: The natural logarithm of the odds, which the logistic model directly estimates as a linear function of the predictors.

By focusing on log-odds, the model ensures that the predictions scale proportionally and stay within the 0 to 1 interval. The coefficients in the model provide insight into how changes in each predictor influence the odds of the event.

Also Read: Beyond Data: The Power of Subjective Probability!

Practical Example: Predicting Customer Behavior Using Demographics

Binary logistic regression is widely used to predict customer behavior, such as the likelihood of making a purchase based on demographic factors. 

The dependent variable is binary: purchase (1) or no purchase (0).

Process:

• Data Collection: Gather customer data, including age, income, location, and previous purchases.
• Feature Selection: Identify key predictors such as age, income, and previous purchasing behavior.
• Model Formulation: The logistic regression equation is:

• Model Training: Use Maximum Likelihood Estimation (MLE) to estimate coefficients.
• Prediction: Predict the probability of a new customer making a purchase based on their demographics.
• Model Evaluation: Assess model accuracy using metrics such as precision, recall, and the AUC-ROC curve.

A retailer could use this model to segment customers into "high likelihood to purchase" and "low likelihood to purchase" groups, allowing them to target high-probability customers with personalized offers or marketing campaigns.

Application Beyond Marketing:
This approach is also used in other fields, such as finance, to predict loan default (i.e., whether a loan will be paid back or not) based on a customer's financial history. 

In healthcare, it is used to predict disease risk (e.g., diabetes: high-risk vs. low-risk) based on demographic and health data.

Also Read: How to Implement Machine Learning Steps: A Complete Guide

Now, let's move on to building binary logistic regression models, where we apply these concepts to model development and performance assessment.

Building and Evaluating Binary Logistic Regression Models

Building and evaluating binary logistic regression models involves selecting features, training on labeled data, and assessing performance through metrics like accuracy, precision, recall, and the ROC curve. 

Tools like Scikit-learn in Python streamline the process, ensuring models generalize well to unseen data.

Fitting a Binary Logistic Regression Model Using Scikit-learn

Scikit-learn provides a straightforward approach for fitting a binary logistic regression model, offering an easy interface for both building and evaluating models. 

The LogisticRegression class from Scikit-learn allows for fitting a model using training data, automatically handling the logistic function and applying regularization as needed.

Steps to fit a model:

1. Data Preparation:

• Load and clean the dataset. Ensure that the target variable is binary (0 or 1).
• Split the data into training and testing sets using the train_test_split function.

2. Model Initialization:

Instantiate a logistic regression model in Python :

# Import necessary libraries
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, roc_curve, auc
import matplotlib.pyplot as plt

# Load dataset and prepare binary classification (0 = non-setosa, 1 = setosa)
from sklearn.datasets import load_iris
data = load_iris()
df = pd.DataFrame(data.data, columns=data.feature_names)
df['target'] = data.target
df['target'] = df['target'].apply(lambda x: 1 if x == 0 else 0)  # Convert to binary (setosa vs non-setosa)

# Split data into features (X) and target (y)
X = df.drop('target', axis=1)
y = df['target']

# Split into training and testing sets (80% train, 20% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize and fit logistic regression model
model = LogisticRegression()
model.fit(X_train, y_train)

# Predictions
predictions = model.predict(X_test)

# Evaluation metrics
accuracy = accuracy_score(y_test, predictions)
precision = precision_score(y_test, predictions)
recall = recall_score(y_test, predictions)
f1 = f1_score(y_test, predictions)
cm = confusion_matrix(y_test, predictions)

# ROC Curve and AUC
fpr, tpr, thresholds = roc_curve(y_test, model.predict_proba(X_test)[:, 1])
auc_score = auc(fpr, tpr)

# Output
print(f"Accuracy: {accuracy:.2f}")
print(f"Precision: {precision:.2f}")
print(f"Recall: {recall:.2f}")
print(f"F1-Score: {f1:.2f}")
print(f"Confusion Matrix:\n{cm}")
print(f"AUC Score: {auc_score:.2f}")

# Plot ROC curve
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, color='blue', label=f'ROC Curve (AUC = {auc_score:.2f})')
plt.plot([0, 1], [0, 1], color='gray', linestyle='--')  # Diagonal line for random model
plt.title('ROC Curve')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend(loc='lower right')
plt.show()

Output:

Accuracy: 1.00
Precision: 1.00
Recall: 1.00
F1-Score: 1.00
Confusion Matrix:
[[ 0 10]
[ 0 40]]
AUC Score: 1.00

This output means:

• The model is perfectly predicting the binary classes (setosa or non-setosa).
• The confusion matrix shows 0 false positives and 10 true negatives, indicating the model is highly accurate.

Graph :

Step-by-Step Explanation:

1. Data Loading and Preprocessing:

• The load_iris() function loads the Iris dataset, which contains three classes of iris flowers. We convert it into a binary classification problem (setosa vs non-setosa) by assigning 1 to setosa and 0 to all other species.
• The df['target'] column is modified to be binary by using .apply(lambda x: 1 if x == 0 else 0).

2. Data Splitting:

• The data is split into features (X) and the target (y).
• The dataset is then split into training and testing sets (80% training and 20% testing) using the train_test_split function.

3. Model Fitting:

• We initialize a LogisticRegression model and fit it using the training data (X_train and y_train).

4. Prediction:

• After fitting the model, we use it to predict the target variable for the test data (X_test).

5. Evaluation:

• Accuracy: The overall correctness of the model.
• Precision: The percentage of true positives among all predicted positives.
• Recall: The percentage of true positives among all actual positives.
• F1-Score: The balance between precision and recall.
• Confusion Matrix: Shows the true positive, false positive, true negative, and false negative values.

These metrics are printed to evaluate the model's performance.

Also Read: Understanding the Role of Anomaly Detection in Data Mining

6. ROC Curve and AUC:

• The Receiver Operating Characteristic (ROC) curve is plotted, showing the trade-off between sensitivity and specificity.
• The Area Under the Curve (AUC) is calculated, representing the model's ability to discriminate between the two classes.
• The ROC curve is displayed with a diagonal line representing a random classifier.

Next, let's examine the advantages and limitations of binary logistic regression to better understand its suitability for different analyses.

Also Read: Difference Between Linear and Non-Linear Data Structures 

Advantages and Limitations of Binary Logistic Regression

Binary logistic regression is a straightforward and efficient method for binary classification tasks.  While it offers clear interpretability and computational efficiency, it also has constraints, particularly when dealing with complex or nonlinear data.

Below is a summary of its key advantages and limitations.

Also Read: 15 Key Techniques for Dimensionality Reduction in Machine Learning

Now, let's explore how upGrad offers resources and expert guidance to accelerate your machine learning journey.

How Can upGrad Help You on Your Machine Learning Journey? 

Binary logistic regression is widely used in industries like healthcare, marketing, and finance. For example, in cybersecurity, LR is used to classify whether a transaction is fraudulent or legitimate, based on transaction patterns and user behavior.  This helps prevent fraud in real-time

To implement binary logistic regression effectively, start by learning Python, focusing on libraries like NumPy, Pandas, and scikit-learn. Learn the theory behind it, such as the logistic function, odds, and MLE. Expand your skills by exploring regularization, overfitting, and tuning models in Python.

Many learners struggle with unstructured resources, making complex topics like LR confusing. upGrad provides a structured machine learning curriculum with hands-on projects, mentorship, and a clear learning path.

Some additional courses include: 

• Master of Science in Data Science
• DBA in Emerging Technologies with Concentration in Generative AI
• Executive Programme in Generative AI for Leaders

Personalized expert guidance by upGrad provides tailored support, while offline centers provide collaboration with instructors and peers, offering real-world insights to accelerate your application of machine learning concepts. 

Expand your expertise with the best resources available. Browse the programs below to find your ideal fit in Best Machine Learning and AI Courses Online.

Best Machine Learning and AI Courses Online

Discover in-demand Machine Learning skills to expand your expertise. Explore the programs below to find the perfect fit for your goals.

In-demand Machine Learning Skills

Discover popular AI and ML blogs and free courses to deepen your expertise. Explore the programs below to find your perfect fit.

Popular AI and ML Blogs & Free Courses

Reference:
https://www.jurnal.yoctobrain.org/index.php/ijaimi/article/view/137