Homebreadcumb forward arrow iconBlogbreadcumb forward arrow iconBig Databreadcumb forward arrow iconHow Big Data and Machine Learning are Uniting Against Cancer

How Big Data and Machine Learning are Uniting Against Cancer

Last updated:
8th Jan, 2018
Read Time
15 Mins
share image icon
In this article
Chevron in toc
View All
How Big Data and Machine Learning are Uniting Against Cancer

Cancer is not one disease. It is many diseases. Let us understand the cause of cancer by a simple example. If you take a photocopy of a document, due to some issues, other dots or smears appear on it even though they are not present in the original copy. In the same way, in gene replication processes, errors occur inadvertently. Most of the time the genes with errors will not be able to sustain and will ultimately perish.

In some rare cases, the mutated gene with mistakes will survive and get further replicated uncontrollably. Uncontrollable replication of mutated genes is the primary cause of cancer. This mutation can happen in any of the twenty thousand genes in our body. Variation in any one or a combination of genes makes cancer a severe disease to conquer. To eradicate cancer, we need methods to destroy the rogue cells without harming the functional cells of the body; which makes it doubly hard to defeat.

Cancer and its complexity

Cancer is a disease with a long tail distribution. Long tail distribution means there are various reasons for this condition to occur and there is no single solution for eradicating it. There are diseases which affect a large percentage of the population but have a sole cause of occurrence. For example, let us consider Cholera. Eating food or drinking water contaminated by the bacterium Vibrio Cholerae is the cause of cholera. Cholera can occur only because of Vibrio Cholerae, and there is no another reason. Once we find out the only cause of a disease, then it is relatively easy to conquer it.

Big Data and Machine Learning United Against Cancer UpGrad Blog
What if a condition occurs because of multiple reasons? A mutation can occur in any of the twenty thousand genes in our body. Not only that, but we also need to consider their combinations. Cancer may not just happen because of a random mutation in a gene but also because of a combination of gene mutations. The number of causes for cancer becomes exponential, and there is no single mechanism to cure it. For example, a mutation of any of these genes ALK, BRAF, DDR2, EGFR, ERBB2, KRAS, MAP2K1, NRAS, PIK3CA, PTEN, RET, and RIT1 can cause lung cancer. There are many ways for cancer to occur and that’s why it is a disease with long tail distribution.

Ads of upGrad blog

In our arsenal for waging this war on cancer and conquering it, big data and machine learning are critical tools. How can big data help in fighting this war? What does machine learning have to do with cancer? How are they going to help in fighting a disease with many causes, a condition with a long tail distribution? Firstly, how and where is this big data generated? Let us find answers to these questions.

Gene Sequencing and explosion in data

Gene sequencing is one area which is producing humongous amounts of data. Exactly how much data? According to the Washington Post, the human data generated through gene sequencing (approximately 2.5 lakh sequences) takes up about a fourth of the size of YouTube’s yearly data production. If all this data were combined with all the extra information that comes with sequencing genomes and recorded on 4GB DVDs, it would be a stack about half a mile high.

Explore Our Software Development Free Courses

The methods for gene sequencing have improved over the years, and the cost for the same has plummeted exponentially. In the year 2008, the cost of gene sequencing was 10 million dollars. As of today, it is only a 1000 dollars. In the future, it is expected to reduce further. It is estimated that one billion people will have their genes sequenced by 2025. So, within the next decade, the genomics data generated will be somewhere between 2 – 40 exabytes in a year. An exabyte is ten followed by 17 zeros.

Before coming to how data will help in curing cancer, let us take one concrete example and see how data can help in conquering a disease. Data and its analysis helped in finding out the cause of one infectious disease and fight it, not now but in nineteenth-century itself! Yes, in the nineteenth century! The name of that disease is Cholera.

Clustering in the Nineteenth Century – the Cholera breakthrough

John Snow was an anesthesiologist and cholera broke out in September 1854 near Snow’s house. To know the reason for cholera, Snow decided to note the spatial dimensions of the patients on the city map. He marked the location of the home address of patients on London’s city map. With this exercise, John Snow understood that people suffering from cholera were clustered around some specific water wells. He firmly believed that a contaminated pump was responsible for the epidemic and against the will of the local authorities replaced the pump. This replacement drastically reduced the spread of cholera.

Snow subsequently published a map of the outbreak to support his theory, showing the locations of the 13 public wells in the area, and the 578 cholera deaths mapped by home address. This map ultimately led to the understanding that cholera was an infectious disease and quickly spread through the medium of water. John Snow’s experiment is the earliest example of applying the clustering algorithm to know the cause of illness and help eradicate it. In the nineteenth century, John Snow could apply clustering algorithm on a London city map with a pencil. With cancer as the target disease, this level of analysis is not possible with the same ease as John Snow’s Analysis. We need sophisticated tools and technologies to mine this data. That is where we leverage the capabilities of modern technologies like Machine Learning and Big Data.

Explore our Popular Software Engineering Courses

Big data and Machine learning – tools to fight cancer

Vast amounts of data along with machine learning algorithms will help us in our fight with cancer in many ways. It can help us with diagnosis, treatment, and prognosis. Mainly, it will help customise the therapy according to the patient, which is not possible otherwise. It will also help deal with the long tail of the distribution.

Big Data and Machine Learning United Against Cancer UpGrad Blog
Given the enormous amounts of Electronic Medical Records (EMR), data generated and recorded by various hospitals; it is possible to use ‘labelled’ data in diagnosing cancer. Techniques like Natural Language Programming (NLP) are utilised for making sense of doctor’s prescriptions and Deep Learning Neural Networks are deployed to analyse CT and MRI scans. The different types of machine learning algorithms search the EMR databases and find hidden patterns. These hidden patterns will help in diagnosing cancers.

A college student was able to design an Artificial Neural Network from the comfort of her home and developed a model that can diagnose breast cancer with a high degree of accuracy.

In-Demand Software Development Skills

Diagnosis with Big Data and Machine Learning

Brittanny Wenger was 16 years old when her older cousin was diagnosed with breast cancer. This inspired her to make the process better by improving the diagnostics. Fine Needle Aspiration (FNA) was a less invasive method of biopsy and the quickest method of diagnosis. The doctors were reluctant to use FNA because the results are not reliable. Brittanny thought of using her programming skills to do something about it. She decided to improve the reliability of FNA which would enable the women to choose less invasive and comfortable diagnostic methods.

Brittanny found public domain data from the University of Wisconsin that included Fine Needle Aspiration. She coded an Artificial Neural Network (ANN) which is inspired by the design of human brain architecture. She used cloud technologies to process the data and train the ANN to find the similarities. After many attempts and errors finally, her network was able to detect breast cancer from an FNA test data with 99.1% sensitivity to malignancy. This method is applicable for diagnosing other cancers as well.

The accuracy of diagnosis is dependent upon the amount and quality of the data available. The more the data available, the more the algorithms will be able to query the database, find similarities and come out with valuable models.

Treatment with Big Data and Machine Learning

Big data and Machine learning will be helpful not only for diagnosis but treatment as well. John and Kathy were married for three decades. At the age of 49, Kathy was diagnosed with stage III breast cancer. John, CIO of a Boston hospital helped plan her treatment with the help of big data tools that he designed and brought into existence.

In 2008, five Harvard affiliated hospitals shared their databases and created a powerful search tool known as ‘Shared Health Research Information Network’ (SHRINE). By the time of Kathy’s diagnosis, her doctors could sift through a database of 6.1 million records to find insightful information. Doctors queried ‘SHRINE’ with questions like “50-year-old Asian women, diagnosed with stage III breast cancer and their treatments”. Armed with this information doctors were able to treat her with chemotherapy drugs by targeting the estrogen-sensitive tumour cells by avoiding surgery.

By the time Kathy completed her chemotherapy regimen the radiologists could no longer find any tumour cells. This is one example of how big data tools can help in customising the treatment plan according to the requirement of each.

As cancer is a long tail distribution a ‘one size fits all’ philosophy will not work. For customising treatments depending on the patient’s history, their gene sequence, results of diagnostic tests, a mutation found in their genes or a combination of their genes and environment, big data and machine learning tools are indispensable.

upGrad’s Exclusive Software Development Webinar for you –

SAAS Business – What is So Different?


Drug Discovery with Big Data and Machine Learning

Big data and Machine learning will not only help in diagnosis and treatment but also will revolutionise drug discovery. Researchers can use open data and computational resources to discover new uses for the drugs which are already approved by agencies like FDA for other purposes. For example, scientists at University of California at San Francisco found by number crunching that a drug called ‘pyrvinium pamoate’ which is used to treat pinworms – could shrink hepatocellular carcinoma, a type of liver cancer, in mice. This disease which is associated with the liver is the second highest contributor to cancer deaths in the world.

Big Data and Machine Learning United Against Cancer UpGrad Blog
Not only is big data used for discovering new uses for old drugs but can also be used for detecting new drugs. By crunching data related to different drugs, chemicals, and their properties, symptoms of various diseases, the chemical composition of the drugs used for those conditions and side effects of these medications collected from different media; new drugs can be devised for various types of cancer. This will significantly reduce the time taken to come up with new medicines without wasting millions of dollars in the process.

Using big data and machine learning will no doubt improve the process of diagnosis, treatment and drug discovery in treating cancer, but it is not without challenges. There are many stumbling blocks and problems on the road ahead. If these blocks are not removed, and these challenges are not faced, then our enemy will get the upper hand and will defeat us in the future battle.

Read our Popular Articles related to Software Development

Challenges in using Big Data and Machine Learning to fight Cancer


Except for a few large and technically advanced hospitals, most of them are yet to be digitised. They are still following the old methods of capturing and recording data in massive stacks of files. Due to lack of technical expertise, affordability, economies of scale and various other reasons, digitisation has not taken place. Provision of open source EMR software, teaching how helpful these digital records could be in treating the patients and how profitable it is to the hospitals are some steps in the right direction.

Data locked in enterprise warehouses

As of today, only a few hospitals can digitally capture patient records. This apparatus too is locked away in enterprise warehouses and inaccessible to the world at large.

Hospitals are reluctant to share their databases with other hospitals. Even if they are willing, they are plagued by the different database schemas and architectures. Critical thinking is required on this front about how hospitals can share their databases among themselves for their mutual benefit without being suspicious of each other. A consensus needs to be reached about the schema in which this data should be shared as well, for the benefit of all hospitals. This patient data should be democratised and utilised for the betterment of the future of mankind.


Big Data and Machine Learning United Against Cancer UpGrad Blog
Patient data should not be allowed to be employed for the growth of a single organisation. Utmost care should be taken to anonymise the individual to whom the data belongs. If a person’s lipstick preference is leaked, then there is not much harm. If a person’s medical history is leaked, then it will have a significant impact on his life and prospects.

The government should take positive steps in this direction and should help create a big data infrastructure for storing medical records of patients from all hospitals. It should make it compulsory for all hospitals to share their database within this shared infrastructure. Access to this database should be made free for patient treatment and research.

Improvement in efficiency of Machine Learning Algorithms

Machine learning is not a magic pill for cancer diagnosis and treatments. It is a tool that if used well can help in our journey to conquer cancer. Machine learning is still in a nascent stage and has its disadvantages. For example, the data on which these algorithms are trained needs to be very close to the data on which they are utilised for producing results. If there is a huge difference in them, then the algorithm will not be able to provide meaningful results which can be employed.

There are many machine learning algorithms which exist with their own peculiar assumptions, advantages, and disadvantages. If we can find a way to combine all these different algorithms for achieving the results required by us, i.e. curing cancer, needless to say, we would have found a hugely beneficial outcome. The famous machine learning scientist Pedro Domingos calls it “The Master Algorithm”, who also wrote a popular science book of the same name.
According to Pedro, there are five different schools of thought in machine learning. The symbolist, connectionist, Bayesian, evolutionaries and analogisers. It is difficult to go into all these different types of machine learning systems in this article. I will cover all the five types of machine learning systems in one of my future blogs. For now, we need to understand that all these different methods have advantages and disadvantages of their own. If we can combine them, then we can derive highly impactful insights from our data. This will be immensely useful not only for all kinds of predictions and forecasts but also for our fight against a vengeful enemy – cancer.

To summarise, cancer is a formidable enemy which keeps changing its form frequently. We do possess new weapons in our arsenal now in the form of big data and machine learning, however, to face it competently. But to demolish it entirely we need a more powerful weapon than what we presently possess. The name of that weapon is ‘The Master Algorithm’.

We also need to make some changes in the strategies and methods with which we are fighting this enemy. These changes are creating a big data infrastructure, making it compulsory for hospitals to share anonymised patient records, maintaining the security of the database and allowing free access to the database for patient treatment and research to cure cancer.

Ads of upGrad blog

Get data science certification from the World’s top Universities. Learn Executive PG Programs, Advanced Certificate Programs, or Masters Programs to fast-track your career.

Wrapping up

If you are interested to know more about Big Data, check out our Advanced Certificate Programme in Big Data from IIIT Bangalore.

Learn Software Engineering degrees online from the World’s top Universities. Earn Executive PG Programs, Advanced Certificate Programs, or Masters Programs to fast-track your career.

Thulasiram is a veteran with 20 years of experience in production planning, supply chain management, quality assurance, Information Technology, and training. Trained in Data Analysis from IIIT Bangalore and UpGrad, he is passionate about education and operations and ardent about applying data analytic techniques to improve operational efficiency and effectiveness. Presently, working as Program Associate for Data Analysis at UpGrad.
Get Free Consultation

Select Coursecaret down icon
Selectcaret down icon
By clicking 'Submit' you Agree to  
UpGrad's Terms & Conditions

Our Popular Big Data Course

Frequently Asked Questions (FAQs)

1 Is machine learning the same thing as AI?

Machine learning is not the same thing as artificial intelligence, though both are very closely linked. AI helps computers develop human-like thinking abilities and learning and troubleshooting capabilities. On the other hand, machine learning is essentially an application of AI; it is a subset of the umbrella term. Machine learning is the technique where computers learn without being instructed by adopting mathematical data models. Again, artificial intelligence helps machines simulate human-like reasoning to make decisions based on new information by the use of logic and mathematics. AI is what makes a device smart, and machine learning helps such smart machines develop their intelligence.

2Do data scientists need to write code?

Yes, data scientists usually have to write code and need to know how to write code even if they do not need to engage in programming daily. Ideally, a data scientist is someone who is an expert in statistics as well as software engineering. But how much programming data scientists need to engage in depends on the tools they are using and their job roles and responsibilities. When data scientists have to write code, it mainly involves using Python and R languages to develop analysis scripts that generate actionable insights. They also have to develop prototypes of digital products and also write production code.

3Why is the R programming language called so?

R is one of the most sophisticated programming languages used today in data science. It is an open-source language used for scripting in data visualization and predictive analytics. The name of this programming language has been derived from the first letter of the names of its developers – Robert Gentleman and Ross Ihaka. R provides the environment necessary to carry out graphics and statistical computing. It offers a wide range of graphical and statistical techniques and also supports excellent extensibility through the use of packages. R offers an integrated suite of applications which are highly useful in the field of data science.

Explore Free Courses

Suggested Blogs

Top 10 Hadoop Commands [With Usages]
In this era, with huge chunks of data, it becomes essential to deal with them. The data springing from organizations with growing customers is way lar
Read More

by Rohit Sharma

12 Apr 2024

Characteristics of Big Data: Types & 5V’s
Introduction The world around is changing rapidly, we live a data-driven age now. Data is everywhere, from your social media comments, posts, and lik
Read More

by Rohit Sharma

04 Mar 2024

50 Must Know Big Data Interview Questions and Answers 2024: For Freshers & Experienced
Introduction The demand for potential candidates is increasing rapidly in the big data technologies field. There are plenty of opportunities in this
Read More

by Mohit Soni

What is Big Data – Characteristics, Types, Benefits & Examples
Lately the term ‘Big Data’ has been under the limelight, but not many people know what is big data. Businesses, governmental institutions, HCPs (Healt
Read More

by Abhinav Rai

18 Feb 2024

Cassandra vs MongoDB: Difference Between Cassandra & MongoDB [2023]
Introduction Cassandra and MongoDB are among the most famous NoSQL databases used by large to small enterprises and can be relied upon for scalabilit
Read More

by Rohit Sharma

31 Jan 2024

13 Ultimate Big Data Project Ideas & Topics for Beginners [2024]
Big Data Project Ideas Big Data is an exciting subject. It helps you find patterns and results you wouldn’t have noticed otherwise. This skill
Read More

by upGrad

16 Jan 2024

Be A Big Data Analyst – Skills, Salary & Job Description
In an era dominated by Big Data, one cannot imagine that the skill set and expertise of traditional Data Analysts are enough to handle the complexitie
Read More

by upGrad

16 Dec 2023

12 Exciting Hadoop Project Ideas & Topics For Beginners [2024]
Hadoop Project Ideas & Topics Today, big data technologies power diverse sectors, from banking and finance, IT and telecommunication, to manufact
Read More

by Rohit Sharma

29 Nov 2023

Top 10 Exciting Data Engineering Projects & Ideas For Beginners [2024]
Data engineering is an exciting and rapidly growing field that focuses on building, maintaining, and improving the systems that collect, store, proces
Read More

by Rohit Sharma

21 Sep 2023

Schedule 1:1 free counsellingTalk to Career Expert
footer sticky close icon