Top 15 NLP tools in 2021 Every Machine Learning Engineer Should Have Hands-on

NLP is one of the most sought-after domain in the field of AI/Data Science in 2020. It has a wide variety of applications and finds its use cases adopted by many industries. The top Industries that practice NLP today are Finance/Fintech, Banking, Law, Healthcare, Insurance, Retail, Advertisement & media, Publishing media, the list can go on.

So, if someone is looking to build a career in AI, then definitely NLP to should be on top of their list. Lately, there have been leaps and bound research associated with it. But if one can get lost in the ocean, so let me list down Top NLP tools to use in 2020. 

I will also rank them as helpful, essential, and indispensable where helpful is the least rank & indispensable is the highest. 

A. General Purpose

2. NLTK: The good NLTK is still relevant in 2020 for a variety of text preprocessing task like tokenization, stemming, tagging, parsing, semantic reasoning, etc. But even if NLTK is easy-to-use, today it has limited use case application. Many of the modern algorithms don’t need a lot of text preprocessing. 

• Github: github.com/nltk/nltk 
• Verdict: Helpful 
• Reason: Relevancy in 2020 

2. Spacy: Spacy is the perfect all-in-one NLP library with very intuitive and easy to use API. Like the NLTK it also supports all variety of preprocessing task. But the best part of Spacy is its support for many common NLP task like NER, POS tagging, tokenization, statistical modelling, syntax-driven sentence segmentation, etc., out of the box with 59+ languages. The upcoming spacy 3.0 will be a game-changer with support for transformer architecture. 

• Github: github.com/explosion/spaCy 
• Verdict: Indispensable 
• Reason: Easy, support for a wide variety of common task out of the box and speed. 

3. Clean-text: Python provides the regex for string manipulation, but working with its pattern is a painful job. This job can be done with ease using Clean-text. It is quite simple & easy to use but at the same time, also powerful. It can even clean non-alphanumeric ASCII characters. 

• Github: github.com/jfilter/clean-text 
• Verdict: Helpful 
• Reason: Limited use case but quite easy to use. 

Read: Top Deep Learning Tools

B. Deep Learning based tools: 

4. Hugging Face Transformers: Models based on Transformers are the current sensation of the world of NLP. Hugging Face transformers library provides all SOTA models (like BERT, GPT2, RoBERTa, etc.) used with TF 2.0 and Pytorch. Their pre-trained models can be used out-of-the-box for a wide variety of downstream task like NER, sequence classification, extractive question answering, language modelling, text generation, summarization, translation. It also provides support for fine-tuning on a custom dataset. Check out their excellent docs and model appendix to get started. 

• Github: github.com/huggingface/transformers 
• Verdict: Indispensable 
• Reason: Current sensation of the world of NLP, provides large no of pre-trained models for a wide variety of downstream task 

5. Spark NLP: Lately, it is Spark NLP which is making the most noise in the world of NLP, especially in the Healthcare sector. As it uses Apache Spark as backend, excellent performance and speed are guaranteed. Benchmarks provided by them claim the best training performance compared to Hugging Face transformers, TensorFlow, Spacy.

One thing that stands out is the access to the number of words embedding like BERT, ELMO, Universal sentence Encoder, GloVe, Word2Vec, etc., provided by it. It also allows training a model for any use case due to its general-purpose nature. Many companies, including FAANG, are using it. 

• Github: github.com/JohnSnowLabs/spark-nlp 
• Verdict: Indispensable 
• Reason: Excellent production-grade performance, general-purpose nature. 

6. Fast AI: It is built on top of Pytorch and can be used to design any framework, including NLP based. Its APIs are very intuitive with a goal of minimal code and emphasis on practicality over theory. It can also easily integrate with Hugging face transformers. The author of the library is Jeremy Howard, who always stresses on use of best practices. 

• Github: github.com/fastai/fastai 
• Verdict: Essential 
• Reason: Useful APIs, emphasis on practicality. 

7. Simple Transformers: It based on Hugging Face transformers and act kind of easy high-level API for it. But don’t assume this as its limitation. For anyone who is not looking to custom design architecture but wants to develop a model based on standard steps, then no other library is better than it.

It supports all mostly used NLP use case like Text Classification, Token Classification, Question Answering, Language Modeling, Language Generation, Multi-Modal Classification, Conversational AI, Text Representation Generation. It also has excellent docs. 

• Github: github.com/ThilinaRajapakse/simpletransformers 
• Verdict: Essential 
• Reason: Act like easy & high-level API for Hugging Face transformers 

Also Read: How to make chatbot in Python?

C. Niche Use Cases: 

8. Rasa: It is by far the most complete Conversational AI tool to build Smart Chatbot, text and voice-based assistant. It is extremely flexible to train. 

• Github: 
• Verdict: Helpful 
• Reason: Limited use case but at the same time best in class. 

9. TextAttack: A seasoned ML practitioner always weights testing more than training. This framework is for adversarial attacks, adversarial training, and data augmentation in NLP. It helps to check the robustness of the NLP system. It can be a bit confusing to start with it but follow their docs to get started and understand the motivation behind the use of it. 

• Github: github.com/QData/TextAttack 
• Verdict: Essential
• Reason: Unique and powerful tool. 

10. Sentence Transformer: Generating embedding or transforming text into vectors is the key building block of designing any NLP framework. One of the old school methods is to use TF-IDF, but it lacks context. Use of transformers can address this issue. There are quite a few tools which can generate transformer-based embeddings (even hugging face transformer can be tweak & used), but none of them makes it as utterly simple as sentence transformer. 

• Github: github.com/UKPLab/sentence-transformers 
• Verdict: Helpful 
• Reason: Limited use case but get the job done. 

11. BertTopic: If anyone is looking to design powerful Topic modelling system then look no further away than BERTTopic. It uses BERT embeddings and c-TF-IDF (author’s modified version of TF-IDF) to create dense clusters allowing for easily interpretable topics whilst keeping important words in the topic descriptions. 

• Github: github.com/MaartenGr/BERTopic 
• Verdict: Helpful 
• Reason: Limited use case but at the same time best in class 

12. Bert Extractive Summarizer: This is yet another awesome tool based on hugging face transformer which can be used for text summarization. It summarizes input text based on context, so you don’t need to worry about missing valuable information. 

• Github: github.com/dmmiller612/bert-extractive-summarizer 
• Verdict: Helpful 
• Reason: Limited use case but at the same time best in class 

D. Other (Non-Coding) Tools: 

13. Doccano: It is a simple but powerful data tagging tool and can be used to tag sentiment analysis, named entity recognition, text summarization, etc. There are quite a few tools out there, but Doccano is the easiest to set up and quickest to get-go. 

• Github: github.com/doccano/doccano 
• Verdict: Essential 
• Reason: Quick and easy to get-go, support multiple formats. 

14. Github Actions: Currently, the best feature of Github is not free (even private) code hosting but its Github action. It is one of the better CI/CD tool out there. If somehow you are not to using it, then you are missing a lot. A CI/CD tool makes development speedy & dependable. 

• Verdict: Indispensable 
• Reason: Free CI/CD tool with great community support. 

15. DVC (Data Version Control): Data is the heart of any Data Science project, so managing it is key. DVC takes inspiration from the Git. It integrates with Git effortlessly. It enables us to change our versioned data back and forth or Data time travel. It also works with cloud storage like aws s3, azure blob storage, gcp cloud storage, etc. 

• Github: github.com/iterative/dvc 
• Verdict: Indispensable 
• Reason: Works with the git, cloud storage and can be used to manage a humongous size of data 

If you want to master machine learning and learn how to train an agent to play tic tac toe, to train a chatbot, etc. check out upGrad’s Machine Learning & Artificial Intelligence PG Diploma course.