Blog_Banner_Asset
    Homebreadcumb forward arrow iconBlogbreadcumb forward arrow iconArtificial Intelligencebreadcumb forward arrow iconAn Intuition Behind Sentiment Analysis: How To Do Sentiment Analysis From Scratch?

An Intuition Behind Sentiment Analysis: How To Do Sentiment Analysis From Scratch?

Last updated:
7th Dec, 2020
Views
Read Time
8 Mins
share image icon
In this article
Chevron in toc
View All
An Intuition Behind Sentiment Analysis: How To Do Sentiment Analysis From Scratch?

Introduction

Text is the most important means of perceiving information for human beings. The majority amount of intelligence gained by humans is through learning and comprehending the meaning of texts and sentences around them.

After a certain age, humans develop an intrinsic reflex to understand the inference of any word/text without even knowing. For machines, this task is completely different. To assimilate the meanings of texts and sentences, machines rely on the fundamentals of Natural Language Processing (NLP).

Best Machine Learning and AI Courses Online

Deep learning for natural language processing is pattern recognition applied to words, sentences, and paragraphs, in much the same way that computer vision is pattern recognition applied to pixels of image.

Ads of upGrad blog

None of these deep learning models truly understand text in a human sense; rather, these models can map the statistical structure of written language, which is sufficient to solve many simple textual tasks. Sentiment analysis is one such task, for example: classifying the sentiment of strings or movie reviews as positive or negative.

These have large scale applications in the industry too. For example: a goods and services company would like to gather the data of the number of positive and negative reviews it has received for a particular product to work upon the product-life cycle and improve its sales figures and gather customer feedback.

In-demand Machine Learning Skills

Join the Artificial Intelligence Course online from the World’s top Universities – Masters, Executive Post Graduate Programs, and Advanced Certificate Program in ML & AI to fast-track your career.

Preprocessing

The task of sentiment analysis can be broken down into a simple supervised machine learning algorithm, where we usually have an input X, which goes into a predictor function to get Yhat. We then compare our prediction with the true value Y, This gives us the cost which we then use to update the parameters (theta) of our text processing model. 

To tackle the task of extracting sentiments from a previously unseen stream of texts, the primitive step is to gather a labeled dataset with separate positive and negative sentiments. These sentiments can be: good review or bad review, sarcastic remark or non-sarcastic remark, etc.

The next step is to create a vector of dimension V, where V corresponds to the entire vocabulary size of the corpus of text. This vocabulary vector will contain every word (no word is repeated) that is present in our dataset and will act as a lexicon for our machine which it can refer to. Now we preprocess the vocabulary vector to remove redundancies. The following steps are performed:

  1. Eliminating URLs and other non-trivial information (which does not help determine the meaning of a sentence)
  2. Tokenizing the string into words: suppose we have the string “I love machine learning”, now by tokenizing we simply break the sentence into single words and store it in a list as [I, love, machine, learning]
  3. Removing stop words like “and”, “am”, “or”, “I”, etc.
  4. Stemming: we transform each word to its stem form. Words like “tune”, “tuning” and “tuned” have semantically the same meaning, so reducing them to its stem form that is “tun” will reduce the vocabulary size
  5. Converting all words to lowercase

To summarize the preprocessing step, let’s take a look at an example: say we have a positive string “I am loving the new product at upGrad.com”. The final preprocessed string is obtained by removing the URL, tokenizing the sentence into single list of words, removing the stop words like “I, am, the, at”, then stemming the words “loving” to “lov” and “product” to “produ” and finally converting it all to lowercase which results in the list [lov, new, produ].

Feature Extraction

After the corpus is preprocessed, the next stride would be to extract features from the list of sentences. Like all other neural networks, deep-learning models don’t take as input raw text: they only work with numeric tensors. The preprocessed list of words are hence in need to be converted into numerical values. This can be done in the following way. Assume that given a compilation of strings with positive and negative strings such as (assume this as the dataset):

 

Positive stringsNegative strings
  • I am happy because I am learning NLP
  • I am happy
  • I am sad, I am not learning NLP
  • I am sad

Now to convert each of these strings into a numerical vector of dimension 3, we create a dictionary to map the word, and the class it appeared in (positive or negative) to the number of times that word appeared in its corresponding class.

VocabularyPositive frequencyNegative frequency
I33
am33
happy20
because10
learning11
NLP11
sad02
not01

After generating the aforementioned dictionary, we look at each of the strings individually, and then sum the number positive and negative frequency number of the words that appear in the string leaving the words that do not appear in the string. Let’s take the string ‘“I am sad, I am not learning NLP” and generate the vector of dimension 3.

“I am sad, I am not learning NLP”

VocabularyPositive frequencyNegative frequency
I33
am33
happy2   0
because1  0
learning11
NLP11
sad02
not01
Sum = 8Sum = 11

We see that for the string “I am sad, I am not learning NLP”, only two words “happy, because” are not contained in the vocabulary, now to extract features and create the said vector, we sum the positive and negative frequency columns separately leaving out the frequency number of the words that are not present in the string, in this case we leave “happy, because”. We obtain the sum as 8 for the positive frequency and 9 for the negative frequency.

Popular AI and ML Blogs & Free Courses

Hence, the string “I am sad, I am not learning NLP” can be represented as a vector X = [1, 8, 11], which makes sense as the string is semantically in a negative context. The number “1” present in index 0 is the bias unit which will remain “1” for all forthcoming strings and the numbers “8”,“11” represent the sum of positive and negative frequencies respectively.

In a similar manner, all the strings in the dataset can be converted to a vector of dimension 3 comfortably. 

Read more: Sentiment Analysis Using Python: A Hands-on Guide

Applying Logistic Regression

Feature extraction makes it easy to understand the essence of the sentence but machines still need a more crisp way to flag an unseen string into positive or negative. Here logistic regression comes into play that makes use of the sigmoid function which outputs a probability between 0 and 1 for each vectorised string. 

Figure 1: Graphical notation of sigmoid function

Ads of upGrad blog

Figure 1 shows that whenever the dot product of theta and X is negative, the prediction function classifies the string and negative and whenever the prediction function is positive, it outputs the string as positive.

Also Read: Top 4 Data Analytics Project Ideas: Beginner to Expert Level 

What Next?

Sentiment Analysis is an essential topic in machine learning. It has numerous applications in multiple fields. If you want to learn more about this topic, then you can head to our blog and find many new resources.

On the other hand, if you want to get a comprehensive and structured learning experience, also if you’re interested to learn more about machine learning, check out IIIT-B & upGrad’s PG Diploma in Machine Learning & AI which is designed for working professionals and offers 450+ hours of rigorous training, 30+ case studies & assignments, IIIT-B Alumni status, 5+ practical hands-on capstone projects & job assistance with top firms.

Profile

Pavan Vadapalli

Blog Author
Director of Engineering @ upGrad. Motivated to leverage technology to solve problems. Seasoned leader for startups and fast moving orgs. Working on solving problems of scale and long term technology strategy.
Get Free Consultation

Select Coursecaret down icon
Selectcaret down icon
By clicking 'Submit' you Agree to  
UpGrad's Terms & Conditions

Our Popular Machine Learning Course

Frequently Asked Questions (FAQs)

1Q1. Why is Random Forest Algorithm best for machine learning?

Random Forest algorithm belongs to the category of supervised learning algorithms, which are extensively used in developing different machine learning models. The random forest algorithm can be applied for both classification and regression models. What makes this algorithm the most suitable for machine learning is the fact that it works brilliantly with high-dimensional information since machine learning mostly deals with subsets of data. Interestingly, the random forest algorithm is derived from the decision trees algorithm. But, you can train using this algorithm in a much shorter span of time than by using decision trees since it uses only specific features. It offers greater efficiency in machine learning models and so it is preferred more.

2Q2. How is machine learning different from deep learning?

Both deep learning and machine learning are subfields of the entire umbrella that we call artificial intelligence. However, these two subfields come with their own differences. Deep learning is essentially a subset of machine learning. However, using deep learning, machines can analyze videos, images, and other forms of unstructured data, which can be difficult to achieve by employing just machine learning. Machine learning is all about enabling computers to think and act by themselves, with minimum human intervention. In contrast, deep learning is used to help machines think based on structures resembling the human brain.

3Q3. Why do data scientists prefer the random forest algorithm?

There are many benefits of using the random forest algorithm, which make it the preferred choice among data scientists. Firstly, it provides highly accurate results when compared to other linear algorithms like logistic and linear regression. Even though this algorithm can be challenging to explain, it is easier to inspect and interpret the results based on its underlying decision trees. You can use this algorithm with equal ease even when new samples and features get added to it. It is easy to use even when some data is missing.

Explore Free Courses

Suggested Blogs

Artificial Intelligence course fees
5061
Artificial intelligence (AI) was one of the most used words in 2023, which emphasizes how important and widespread this technology has become. If you
Read More

by venkatesh Rajanala

29 Feb 2024

Artificial Intelligence in Banking 2024: Examples & Challenges
5443
Introduction Millennials and their changing preferences have led to a wide-scale disruption of daily processes in many industries and a simultaneous g
Read More

by Pavan Vadapalli

27 Feb 2024

Top 9 Python Libraries for Machine Learning in 2024
75060
Machine learning is the most algorithm-intense field in computer science. Gone are those days when people had to code all algorithms for machine learn
Read More

by upGrad

19 Feb 2024

Top 15 IoT Interview Questions & Answers 2024 – For Beginners & Experienced
64134
These days, the minute you indulge in any technology-oriented discussion, interview questions on cloud computing come up in some form or the other. Th
Read More

by Kechit Goyal

19 Feb 2024

Data Preprocessing in Machine Learning: 7 Easy Steps To Follow
149970
Summary: In this article, you will learn about data preprocessing in Machine Learning: 7 easy steps to follow. Acquire the dataset Import all the cr
Read More

by Kechit Goyal

18 Feb 2024

Artificial Intelligence Salary in India [For Beginners & Experienced] in 2024
907576
Artificial Intelligence (AI) has been one of the hottest buzzwords in the tech sphere for quite some time now. As Data Science is advancing, both AI a
Read More

by upGrad

18 Feb 2024

24 Exciting IoT Project Ideas & Topics For Beginners 2024 [Latest]
752446
Summary: In this article, you will learn the 24 Exciting IoT Project Ideas & Topics. Take a glimpse at the project ideas listed below. Smart Agr
Read More

by Kechit Goyal

18 Feb 2024

Natural Language Processing (NLP) Projects & Topics For Beginners [2023]
106457
What are Natural Language Processing Projects? NLP project ideas advanced encompass various applications and research areas that leverage computation
Read More

by Pavan Vadapalli

17 Feb 2024

45+ Interesting Machine Learning Project Ideas For Beginners [2024]
325986
Summary: In this Article, you will learn Stock Prices Predictor Sports Predictor Develop A Sentiment Analyzer Enhance Healthcare Prepare ML Algorith
Read More

by Jaideep Khare

16 Feb 2024

Schedule 1:1 free counsellingTalk to Career Expert
icon
footer sticky close icon