Blog_Banner_Asset
    Homebreadcumb forward arrow iconBlogbreadcumb forward arrow iconArtificial Intelligencebreadcumb forward arrow iconDecision Tree Regression Functionality, Terms, Implementation [With Example]

Decision Tree Regression Functionality, Terms, Implementation [With Example]

Last updated:
24th Dec, 2020
Views
Read Time
6 Mins
share image icon
In this article
Chevron in toc
View All
Decision Tree Regression Functionality, Terms, Implementation [With Example]

To begin with, a regression model is a model that gives as output a numeric value when given some input values that are also numeric. This differs from what a classification model does. It classifies the test data into various classes or groups involved in a given problem statement.

Best Machine Learning and AI Courses Online

The size of the group can be as small as 2 and as big as 1000 or more. There are multiple regression models like linear regression, multivariate regression, Ridge regression, logistic regression, and many more. Decision tree regression models also belong to this pool of regression models.

The predictive model will either classify or predict a numeric value that makes use of binary rules to determine the output or target value. The decision tree model, as the name suggests, is a tree like model that has leaves, branches, and nodes.

Ads of upGrad blog

In-demand Machine Learning Skills

Learn Machine Learning Online Course from the World’s top Universities. Earn Masters, Executive PGP, or Advanced Certificate Programs to fast-track your career.

Read: Machine Learning Project Ideas

Terminologies to Remember

Before we delve into the algorithm, here are some important terminologies that you all should be aware of.

  1. Root node: It is the topmost node from where the splitting begins.
  2. Splitting: Process of subdividing a single node into multiple sub-nodes. 
  3. Terminal node or leaf node: Nodes that don’t split further are called terminal nodes. 
  4. Pruning: The process of removal of sub nodes .
  5. Parent node: The node that splits further into sub nodes.
  6. Child node: The sub nodes that have emerged out from the parent node.

How does it work?

The decision tree breaks down the data set into smaller subsets. A decision leaf splits into two or more branches that represent the value of the attribute under examination. The topmost node in the decision tree is the best predictor called the root node. ID3 is the algorithm that builds up the decision tree.

It employs a top to down approach and splits are made based on standard deviation. Just for a quick revision, Standard deviation is the degree of distribution or dispersion of a set of data points from its mean value. It quantifies the overall variability of the data distribution.

A higher value of dispersion or variability means greater is the standard deviation indicating the greater spread of the data points from the mean value. We use standard deviation to measure the uniformity of the sample. If the sample is totally homogeneous, its standard deviation is zero.

And similarly, higher is the degree of heterogeneity, greater will be the standard deviation. Mean of the sample and the number of samples are required to calculate standard deviation. We use a mathematical function — Coefficient of Deviation that decides when the splitting should stop It is calculated by dividing the standard deviation by the mean of all the samples.

Source

The final value would be the average of the leaf nodes. Say, for example, if the month November is the node that splits further into various salaries over the years in the month of November (until 2020). For the year 2021, the salary for the month of November would be the average of all the salaries under the node November.

Moving on to the standard deviation of two classes or attributes(like for the above example, salary can be based either on an hourly basis or monthly basis). The formula would look like the following:

Source

where P(c) is the probability of occurrence of the attribute c, S(c)is the corresponding standard deviation of the attribute c. The method of reduction in standard deviation is based on the decrease in standard deviation after a dataset has split.

To construct an accurate decision tree, the goal should be to find attributes that return upon calculation, and return the highest standard deviation reduction. In simple words, the most homogenous branches.

The process of creating a Decision tree for regression covers four important steps.

1. Firstly, we calculate the standard deviation of the target variable. Consider the target variable to be salary like in previous examples. With the example in place, we will calculate the standard deviation of the set of salary values.

2. In step 2, the data set is further split into different attributes. talking about attributes, as the target value is salary, we can think of the possible attributes as — months, hours, the mood of the boss, designation, year in the company, and so on. Then, the standard deviation for each branch is calculated using the above formula. the standard deviation so obtained is subtracted from the standard deviation before the split. The result at hand is called the standard deviation reduction.

3. Once the difference has been calculated as mentioned in the previous step, the best attribute is the one for which the standard deviation reduction value is largest. That means, the standard deviation before the split should be greater than the standard deviation before the split. Actually, mod of the difference is taken and so vice versa is also possible.

4. The entire dataset is classified based on the importance of the selected attribute. On the non-leaf branches, this method is continued recursively till all the available data is processed. Now consider month is selected as the best splitting attribute based on the standard deviation reduction value. So we will have 12 branches for each month. These branches will further split to select the best attribute from the remaining set of attributes.

5. In reality, we require some finishing criteria. For this, we make use of the coefficient of deviation or CV for a branch that becomes smaller than a certain threshold like 10%. When we achieve this criterion we stop the tree building process. Because no further splitting happens, the value that falls under this attribute will be the average of all the values under that node.

Implementation

Decision Tree Regression can be implemented using Python language and scikit-learn library. It can be found under the sklearn.tree.DecisionTreeRegressor.

Some of the important parameters are as follows:

  1. criterion: To measure the quality of a split. It’s value can be “mse” or the mean squared error, “friedman_mse”, and “mae” or the mean absolute error. Default value is mse.
  2. max_depth: It represents the maximum depth of the tree. Default value is None.
  3. max_features: It represents the number of features to look for when deciding the best split. Default value is None. 
  4. splitter: This parameter is used to choose the split at each node. Available values are “best” and “random”. Default value is best.

Check out: Machine Learning Interview Questions

Example from sklearn documentation

>>> from sklearn.datasets import load_diabetes

>>> from sklearn.model_selection import cross_val_score

>>> from sklearn.tree import DecisionTreeRegressor

>>> X, y = load_diabetes(return_X_y=True)

>>> regressor = DecisionTreeRegressor(random_state=0)

>>> cross_val_score(regressor, X, y, cv=10)

                  # doctest: +SKIP

array([-0.39…, -0.46…,  0.02…,  0.06…, -0.50…,

Ads of upGrad blog

      0.16…,  0.11…, -0.73…, -0.30…, -0.00…])

Popular AI and ML Blogs & Free Courses

What Next?

Also, If you’re interested to learn more about Machine learning, check out IIIT-B & upGrad’s Executive PG Programme in Machine Learning & AI which is designed for working professionals and offers 450+ hours of rigorous training, 30+ case studies & assignments, IIIT-B Alumni status, 5+ practical hands-on capstone projects & job assistance with top firms.

Profile

Pavan Vadapalli

Blog Author
Director of Engineering @ upGrad. Motivated to leverage technology to solve problems. Seasoned leader for startups and fast moving orgs. Working on solving problems of scale and long term technology strategy.
Get Free Consultation

Select Coursecaret down icon
Selectcaret down icon
By clicking 'Submit' you Agree to  
UpGrad's Terms & Conditions

Our Popular Machine Learning Course

Explore Free Courses

Suggested Blogs

Bagging vs Boosting in Machine Learning: Difference Between Bagging and Boosting
88130
Owing to the proliferation of Machine learning applications and an increase in computing power, data scientists have inherently implemented algorithms
Read More

by Pavan Vadapalli

12 Feb 2024

6 Types of Regression Models in Machine Learning You Should Know About
272134
Introduction Linear regression and logistic regression are two types of regression analysis techniques that are used to solve the regression problem
Read More

by Pavan Vadapalli

12 Feb 2024

Production System in Artificial Intelligence and its Characteristics
76064
The AI market has witnessed rapid growth on the international level, and it is predicted to show a CAGR of 37.3% from 2023 to 2030. The production sys
Read More

by Pavan Vadapalli

11 Feb 2024

Natural Language Processing (NLP) Projects & Topics For Beginners [2023]
105045
What are Natural Language Processing Projects? NLP project ideas advanced encompass various applications and research areas that leverage computation
Read More

by Pavan Vadapalli

04 Feb 2024

Sentiment Analysis Projects & Topics For Beginners [2024]
37608
Are you studying sentiment analysis and want to test your knowledge? If you are, then you’ve come to the right place. In this article, we’re discussin
Read More

by Pavan Vadapalli

04 Feb 2024

Top 8 Exciting AWS Projects & Ideas For Beginners [2023]
95320
AWS Projects & Topics Looking for AWS project ideas? Then you’ve come to the right place because, in this article, we’ve shared multiple AWS proj
Read More

by Pavan Vadapalli

04 Feb 2024

AWS Salary in India in 2023 [For Freshers & Experienced]
903427
Summary: In this article, you will learn about AWS Salary in India For Freshers & Experienced. AWS Salary in India INR 6,07,000 per annum AW
Read More

by Pavan Vadapalli

28 Jan 2024

Image Segmentation Techniques [Step By Step Implementation]
61319
What do you see first when you look at your selfie? Your face, right? You can spot your face because your brain is capable of identifying your face an
Read More

by Pavan Vadapalli

28 Jan 2024

Data Preprocessing in Machine Learning: 7 Easy Steps To Follow
146604
Summary: In this article, you will learn about data preprocessing in Machine Learning: 7 easy steps to follow. Acquire the dataset Import all the cr
Read More

by Kechit Goyal

28 Jan 2024

Schedule 1:1 free counsellingTalk to Career Expert
icon
footer sticky close icon