Blog_Banner_Asset
    Homebreadcumb forward arrow iconBlogbreadcumb forward arrow iconBig Databreadcumb forward arrow iconApache Oozie Tutorial: Introduction, Workflow & Easy Examples

Apache Oozie Tutorial: Introduction, Workflow & Easy Examples

Last updated:
22nd Feb, 2020
Views
Read Time
14 Mins
share image icon
In this article
Chevron in toc
View All
Apache Oozie Tutorial: Introduction, Workflow & Easy Examples

In this article, we are going to learn about the scheduler system and why it is essential in the first place. We will also discuss why it is essential to have a scheduler in the Hadoop system. Also, we will deeply learn about Apache Oozie and a few of its concepts of Apache Oozie, such as time-based job, word count workflow job, oozie bundle, oozie coordinator, oozie workflow.

Scheduler System

As we all know that there are many jobs which are interdependent on each other and completion of one job only can start another job. For example, in the system of the Hadoop Ecosystem, Hive Job gets the input to work from the output of MapReduce. In this way, there is more process which receives feedback from the production of other processes. For this purpose of organizing the job and bringing a solution, jobs are scheduled using a scheduler system. 

Read: Apache Spark vs Mapreduce

One can now very quickly all those situations of scheduling using Apache Oozie. For the working of the Ecosystem of Hadoop, Apache Oozie is essential. 

Ads of upGrad blog

Apache Oozie: Introduction

In the distributed environment of Hadoop, Jobs are executed and managed by a scheduler system called Apache Oozie. Many various kinds of jobs can be combined using apache oozie, and a job pipeline of one’s desire can be easily created. The task of MapReduce, Sqoop, Pig, or Hive can be quickly scheduled using apache oozie.

An individual can schedule their job easily using the Apache Oozie. One can also run parallel jobs of two or more at the same time with each other while creating the sequence of the task. The Scheduler System, called Apache System, is very extensible, reliable, and scalable.

Action in the workflow can be triggered by the Oozie, which is a web application of Open Source Java. It is the responsibility of Apache Oozie to start the job in the workflow. For the execution of the task, Apache Oozie uses the execution engine of Hadoop.

Through polling and callback, detection of task completion can be done by Apache Oozie. The task is provided with a unique callback HTTP URL at the beginning of starting a job by the Oozie. When the task is finished, Oozie will notify the unique callback HTTP URL about the completion of the task. Sometimes the tasks are polled for completion, in case the callback URL is failed to be invoked by the task. 

Explore our Popular Software Engineering Courses

In Apache Oozie, One can find three kinds of jobs, which are:

Oozie Bundles – Oozie Bundles refer to multiple workflow jobs and coordinators packages.

Oozie Coordinator Jobs – According to the availability of data and time, these kinds of workflow jobs get triggered. 

Oozie Workflow Jobs – Execution of Actions in the sequence are specified by Directed Acyclic Graphs (DAGs)

Oozie Workflow

Direct Acyclic Graph (DAG) arranges the actions to be done in a sequence in the workflow of Hadoop. One step is dependent on another action, and the next action can start only after the execution of the previous actworkion is finished because it needs to take the output of one job as the input for another job.

Read: Must Read Big Data Interview Questions

Java action, shell action, MapReduce action, Hive action, Pig action are some of the workflow actions which can be scheduled and executed by the Apache Oozie scheduler system. One can also specify the condition for a job to run. In the workflow, one can tell the scheduler to run this job or action if the output comes like this or the action meets the requirement.

Based on the job, various kinds of activities can be created by an individual. There can be a unique kind of tags for each type of action. Before the workflow execution, the HDFS path should be placed with the jars or script and the workflow. 

Command: oozie job –oozie http://localhost:11000/oozie -config job.properties -run

http://host_name:11000 is the link address to go to the Oozie web console to check the job status. One can check the job status by just doing a click on the job after opening this Oozie web console.

A fork can be used when one needs to run many jobs together at the same time. When the fork is used, it requires an end node to fork and in this case one needs to take help of Join. Join should be used for each fork. When many jobs are executed together, nodes are assumed as the single c.

A single fork will have single nodes, and each Join will assume only on a single node as their child of the single fork. One can parallelly do the creation of 2 tables at the same time together.

Decision tags are also very useful to use in this system when one needs to run any action based on output. Decision tags help in deciding which operation to run after getting the required output to run a specific action. For instance, one needs not to create any hive table once it is already created. So if the table is already existing, then by adding the decision tag, an individual can stop the creation of the steps of the table. There are also switch cards present in decision nodes, and it is similar to the switch case.

The property file is also known as the config file. It comes handy when the management of the value of the param, script, name-node, and job-tracker becomes difficult. 

Apache Oozie Coordinator

There is some workflow that needs to be regularly scheduled, and there is some workflow that is complex to schedule. Both kinds of workflow can be quickly scheduled by using Oozie Coordinator. Event predicates, data, and time are used as the basis for the workflow trigeneration by Oozie Coordinators. On the satisfaction of a condition, the Job coordinator starts the workflows.

Here are some of the Definition one needs to understand for the coordinator jobs:

  1. frequency – For the purpose of the job execution, frequency is mentioned, and it is counted in minutes. 
  2. timezone – This tells us about the coordinator application’s timezone.
  3. end – It refers to the job’s end datetime.
  4. start – It refers to the job’s start datetime.

Let us now learn more about the Control Information’s properties:

execution- The jobs are executed in order, and the execution specifies it. Whenever different job coordinator meets multiple criteria of execution, then execution comes to tell the order in which the jobs should be executed. These are the kinds of execution:

  • LAST_ONLY
  • LIFO
  • FIFO – It is the standard default execution one can find but can also be changed to another type of execution according to the desire of the individual.
  • Concurrency – It is the property to control the maximum no. of action that can be taken for a job when the jobs are running parallely. One is the default value of the concurrency, which means at one time, only one action can be taken parallelly.
  • Timeout – It is the property that helps to decide the time limit of an action to wait before it is discarded. The action will be immediately timed out if the value is 0, and no input conditions can satisfy the materialization of action. The action can also wait forever without being discarded ever if one has mentioned the value as -1. -1 is the default value of timeout. 

Command: oozie job –oozie http://localhost:11000/oozie -config <path to coordinator.properties file> -run

Read: In-demand Big Data Skills Necessary to Land ‘Big’ Data Jobs

Apache Oozie Bundle

The data pipeline is the set of coordinator applications. The execution and definition of the data pipeline are allowed by the Oozie Bundle system. Coordinator applications are not clearly dependent on each other in an Oozie bundle. Data application pipeline can be easily created by using the coordinator applications’ data dependency. The bundle can be rerun, resumed, suspended, stopped, and started by any person. Operational control of the Oozie Bundle is effortless and better to use.

Kick-off-time – It is the time of the submitting and starting of the coordinator applications by a bundle.

Now, as we move forward, we will know how the creation of the workflow job is done:

In-Demand Software Development Skills

Apache Oozie Word Count Workflow Job

With the use of Apache Oozie, one can do the execution of Word Count Job. All the files will be placed after the directory of WordCountTest is created. All the word count jar is placed after the creation of the lib directory.

Job and the associated parameters will be specified in the file of workflow.xml and job.properties after it is created.

1. job.properties

ResourceManager and NameNode’s are defined in the file of job.properties after it is created. The path of NameNode resolves the path of the directory of the workflow. Jobs are submitted to YARN with the help of the path of JobTracker. HDFS stores the path of the workflow.xml file and is provided by the user.

2. workflow.xml

All the execution and actions are defined in the file called workflow.xml after the user creates it. WorkflowRunnerTest is the name of the workflow-app which has to be specified by the user. After that, the Start node will be determined by the user.

For a workflow job, the entry point is called the start node. The job will start after the point of start is mentioned in the first node of the workflow. After the mentioning of the start node, the job will start from the next node called intersection0. 

Read: Big Data Project Ideas for Beginners

In the next step, in the action node, the user needs to specify the task that should be performed. The mission of WordCount of MapReduce will be executed now. The task of the WordCount of MapReduce is completed after the user specifies the required configuration. The user then defines the address of the NameNode and the Job Tracker. 

The next action is to prepare the element. The cleaning up of the directory is done by using the feature. The directory cleanup is done before the action execution. Now we are going to operate to delete in HDFS. If the out1 folder is created already, we will remove the out1 folder. Before the execution of the job, deletion or creation of a folder is done using prepare tags. Output value class, output key class, reducer class, mapper class, job queue name are some of the properties of the MapReduce specified by the user.

Please have a look at the coding of the workflow.xml file:

<workflow-app xmlns=”uri:0ozie:workflow:0.1″ name=”WorkflowRunnerTest “>

<start to=”intersection0″/>

<action name=”intersection0″>

<map-reduce>

<job-tracke>localhost:8032</job-tracker>

<name-node>hdfs://localhost:8020</name-node>

<prepare> <delete path=”hdfs://localhost:8020/00zieout/outl”/></prepare>

<configuration>

<property>

<name>mapred.job.queue.name</name>

<value>default</value>

</property>

<property>

<name>mapred.mapper.class</name>

<value>MapperClass</value>

</property>

<property>

<name>mapred.reducer.class</name>

<value>ReducerClass</value>

</property>

<property>

<name>mapred.output.key.class</name>

<value>org.apache.hadoop.io. Text</value>

</property>

<property>

<name>mapred.output.value.class</name>

<property>

<name>mapred.output.key.class</name>

<value>org.apache.hadoop.io. Text</value>

</property>

<pгoperty>

<name>mapred.output.value.class</name>

<value>org.apache.hadoop.io. IntWritable</value>

</property>

<ргоperty>

<name>mapred.input.dir</name>

<value>/data</value>

</property>

<property>

<name>mapred.output.dir</name>

<value>/oozieout/out1</value>

</property>

</configuration>

</map-reduce>

<ok to=”end”/>

<error to=”fail”/>

</action>

<kill name=”fail”>

<message>Map/Reduce failed, error message</message>

</kill>

<end name=”end”/>

</workflow-app>

 In HDFS, the Output and Input directory is the last configuration task of MapReduce. Input directory is also known as the data directory. In the NameNode root path, the Data directory gets stored. On the failure of the job, the element will be killed according to the specification given by the user in the end.

Goto : data

Go back to dir listing

Advanced view/download options

Below are important points about this VM, please go through it without fail.

1) Hadoop and all other components are present in /usr/lib/

JDK : /usr/lib/jvm/jdkl.7.0 67

Eclipse : /homme/upgrad/Desktop/eclipse

Hadoop : /usr/lib/hadoop-2.2.0

Pig : /usr/lib/pig-0.12.0

Hive : /usr/lib/hive-0.13.1-bin

Hbase : /usr/lib/hbase-0.96.2-hadoop2

Oozie : /usr/lib/oozię-4.0.0

Sqoop : /usr/lib/sqoop-1.4.4

Flume-ng : /usr/lib/flume-ng

2) The paths of all the components are set.

JDK : .bashrc

Hadoop : .bashrc

Pig : /etc/profile.d/pig.sh

Hive : /etc/profile.d/hive.sh

In the property oozie.wf.application.path in the file of job.properties, the user needs to specify and move the WordCountTest folder in HDFS. Now we will perform a copy of the folder of WordCountTest folder in the root directory of the Hadoop.

Command: hadoop fs -put WordCountTest /

upgrad@localhost:/usr/lib/0ozie-4.0.0

File Edit View Search Terminal Help

[upgrad@localhost oozie-4.0.0]$ hadoop fs -put WordCountTest /

17/12/19 18:11:03 WARN util.NativeCodeLoader: Unable to load native-hadoop

library for your platform… using builtin-java classes where applicable

To check If the folder is uploaded not uploaded in the root directory of HDFS, then the user needs to check the folder by going to the NameNode Web UI for verification.

Now using this code, we will do the execution of the job of the workflow and go ahead:

Command: oozie job -oozie http://localhost:11000/o0ozie -config job.properties –run

upgrad@localhost:/usr/lib/oozie-4.0.0/WordCountTest

File Edit View Search Terminal Help

[upgrad@localhost oozie-4.0.0]$ cd WordCountTest

[upgrad@localhost WordCount Test]$ oozie job -oozie http://localhost:11000/00zie

– config job.properties -run

job: 0000009-171219160449620-0ozie-edur-W

[upgrad@localhost WordCountTest]$

  • Time-Based Word Count Coordinator Job | Apache Oozie Tutorial

We are going to create a controller that will be executed every specified time interval. In the end, time base word count job will be completed across the time interval. Using Apache Oozie, we can create a scheduled job and run them in a periodical manner.

Let us move forward and let us create an Oozie coordinator job. There will be three files create that is workflow.xml, coordinator.xml & workflow.xml files. Wordcount jar file will be placed inside the lib directory.

Let us look into properties file:

frequency=60

startTime=2017-12-19T13\:29Z

endTime=2017-2-19T13\:34Z

timezone=UTC

 nameNode=hdfs://localhost:8020

jobTracker=localhost:8032

queueName=default

 workflowPath=${nameNode}/WordCountTest_TimeBased

 oozie.coord.application.path=${nameNode}/WordCountTest_TimeBased

 Here we have to specify the frequency at which the work should be executed. The unit of frequency here is in minutes. Here, in our case, the coordinator job should be running every 60 minutes. Data set will be captured using the frequency specified that is produced and scheduled to run the coordinator application.

Use this below format to specify frequency:

${coord:minutes(int n)}

 n

 ${coord:minutes(45)} –> 45

${coord:hours(int n)}

 n * 60

 ${coord:hours(3)} –> 180

${coord:days(int n)}

 variable

 ${coord:days(2)} –> minutes in 2 full days from the current date

${coord:months(int n)}

 variable

 ${coord:months(1)} –> minutes in a 1 full month from the current date

Here we have to specify the startTime and the endTime of the job. Where startTime is the start date, and endTime is the end date of the coordinated job.

Finally, we have to specify the application path where all files are stored.

All the properties will be well defined in the coordinator.properties file. Frequency, name, and timezone should be specified in the properties . Hence let us create elements in the coordinator XML file.

<coordinator-app name=”coordinator1” frequency =”${frequency}” start=”${startTime}”

end=”${endTime}” timezone=”${timezone}” xmlns=”uri:oozie:coordinator:0.1”>

<action>

<workflow>

<app-path>${workflowPath}</app-path>

</workflow>

</action>

</coordinate-app>

 Now let us create workflow.xml for our job.

<workflow-app xmlns=”ari:oozie:vorkflows6.” name= Warktlovfunnertest>

<start to=”intersection9″/>

kaction name=intersectione>

<map-reduce>

<job-tracker>localhost:8032</job-tracker>

<name-node>hdfs://localhost:8020</name-node>

<prepare> <delete path=”h5ts//Loalhost 3320/00aie ireBasadcut/ut/></prepare>

<configuration>

<property>

<name>mapred.job.queue.name</name>

<value>default</value>

</property>

<property>

<name>mapred.mapper.class</name>

<value>MapperClass</values

</property>

<property>

<name>mapred.reducer.class</name>

<value>ReducerClass</value>

</property>

<property>

<name>mapred.output.key.class</name>

<value>org.apache.hadoop.io. Text</value>

</property>

<property>

<name>mapred.output.value.class</name>

 Also make:

<value>org.apache.hadoop.io.Text</value>

</property>

<pгоperty>

<name>mapred.output.value.class</name>

<value>org.apache.hadoop.io.Intwritable</value>

</property>

<property>

<name>mapred.input.dir</name>

<value>/data</value>

</property>

<property>

<name>mapred.output.dir</name>

<value>/oozieTimeBasedout/out1</value>

</property>

</configuration>

</map-reduce>

<ok to=”end”/>

<error to=”fail”/>

</action>

<kill name=”fail”>

<message>Map/Reduce failed, error message</message>

</kill>

<end name=end”/>

</workflow-app>

Now let us move this to the HDFS directory.

Finally we have to configure Coordinator job.

<contiguration>

<property>

<name>startTime</name>

<value>2017-12-19T13:29Z<value>

</property>

<property>

<name>workflowPath</name>

<value>hdfs:/localhost:8020WordCountTest TimeBased</value>

</property>

<property>

<name>oozie.coord.application.path</name>

<value>hdfs:/Mocalhost 8020/WordCountTest TimeBased<value>

</property>

<property>

<name>timezone</name>

<value>UTC<value>

<iproperty>

<property>

<name>user.name</name>

<value>upgrad</value>

</property>

<property>

<name>mapreduce job.user.name</name>

<value>upgrad<value>

</property>

<property>

<name>queueName</name>

<value>default</value>

<property>

<property>

Now lets see the output created.

!! 1

“Save” 1

“reboot” 1

(Just 1

+ 1

-C 1

-f 1

-n 1

-r 2

safemode 1

. 1             

./bin/oozie-start.sh 1

./bin/start-hbase.sh 2

./flume-ng 1

bashrc 2

/etc/profile.d/hbase. sh 1

/etc/profile.d/hive.sh 1

/etc/profile.d/oozie.sh 1

/etc/profile.d/pig.sh 1

/etc/profile.d/sqoop.sh 1

Explore Our Software Development Free Courses

 Conclusion

Ads of upGrad blog

Finally, we came to an end of the tutorial and hoped that you liked the tutorial and learned something about Apache Oozie. This article is a good start for any beginner who is interested in learning the basic concept of the Apache Oozie. 

If you are interested to know more about Big Data, check out our Advanced Certificate Programme in Big Data from IIIT Bangalore.

Learn Software Development Courses online from the World’s top Universities. Earn Executive PG Programs, Advanced Certificate Programs or Masters Programs to fast-track your career.

Profile

Utkarsh Singh

Blog Author
Get Free Consultation

Select Coursecaret down icon
Selectcaret down icon
By clicking 'Submit' you Agree to  
UpGrad's Terms & Conditions

Our Popular Big Data Course

Frequently Asked Questions (FAQs)

1What is the job of Apache Oozie?

The primary job Apache Oozie aims to fulfil is to schedule Apache Hadoop jobs. It is a Java web application. Oozie also cooperates by offering operational services that usually work with a Hadoop cluster. Moreover, it can put the various jobs together in one block. Also, its integration with the Hadoop stack is a crucial reason to consider Apache Oozie. Apache MapReduce, Apache Hive, Apache Pig, and Apache Snoop, all the Hadoop jobs, are supported extensively by Oozie. Apache Oozie is also capable of scheduling Java program or shell script jobs. Mainly, Oozie works with Hadoop operations which incorporates cluster administrators to carry on complex data transformation in very less time.

2Why should I consider using Apache Oozie in my organisation?

Many organisations are avidly using Apache Oozie nowadays. It could be troublesome to begin with; however, it gets easier once you get the hang of it. With Oozie, users can manage Hadoop workflows from numerous interfaces. The other prevalent reason companies prefer working with Apache Oozie is that it is free. Despite not being able to fulfil the customers’ operational needs, being free is a considerable bargain. Oozie is very easy to work with, especially creating workflows, managing schedulers, and writing scripts. People do not want to change their technology if they begin their journey with Oozie. Managing Oozie with Hadoop workflows is never easy, but a Big Data project around schedulers makes it worthwhile.

3What are the cons of Apache Oozie?

Apache Oozie is less attentive to actions, dependencies, and other operations. For instance, for partitions, the format is MM.DD.YY; doing it another way around won’t work out. The next disadvantage of Oozie is its limited actions like FS action, Pig action, SSH action, and Shell action. These actions are very compact as they allow passing only one action to the next. Oozie is also restricted to using only HDFS for MapReduce jobs which could be a limitation.

Explore Free Courses

Suggested Blogs

Characteristics of Big Data: Types &#038; 5V&#8217;s
5360
Introduction The world around is changing rapidly, we live a data-driven age now. Data is everywhere, from your social media comments, posts, and lik
Read More

by Rohit Sharma

04 Mar 2024

50 Must Know Big Data Interview Questions and Answers 2024: For Freshers &#038; Experienced
7029
Introduction The demand for potential candidates is increasing rapidly in the big data technologies field. There are plenty of opportunities in this
Read More

by Mohit Soni

What is Big Data &#8211; Characteristics, Types, Benefits &#038; Examples
185198
Lately the term ‘Big Data’ has been under the limelight, but not many people know what is big data. Businesses, governmental institutions, HCPs (Healt
Read More

by Abhinav Rai

18 Feb 2024

Cassandra vs MongoDB: Difference Between Cassandra &#038; MongoDB [2023]
5460
Introduction Cassandra and MongoDB are among the most famous NoSQL databases used by large to small enterprises and can be relied upon for scalabilit
Read More

by Rohit Sharma

31 Jan 2024

13 Ultimate Big Data Project Ideas &#038; Topics for Beginners [2024]
99668
Big Data Project Ideas Big Data is an exciting subject. It helps you find patterns and results you wouldn’t have noticed otherwise. This skill
Read More

by upGrad

16 Jan 2024

Be A Big Data Analyst – Skills, Salary &#038; Job Description
899645
In an era dominated by Big Data, one cannot imagine that the skill set and expertise of traditional Data Analysts are enough to handle the complexitie
Read More

by upGrad

16 Dec 2023

12 Exciting Hadoop Project Ideas &#038; Topics For Beginners [2024]
20654
Hadoop Project Ideas & Topics Today, big data technologies power diverse sectors, from banking and finance, IT and telecommunication, to manufact
Read More

by Rohit Sharma

29 Nov 2023

Top 10 Exciting Data Engineering Projects &#038; Ideas For Beginners [2024]
39930
Data engineering is an exciting and rapidly growing field that focuses on building, maintaining, and improving the systems that collect, store, proces
Read More

by Rohit Sharma

21 Sep 2023

Big Data Architects Salary in India: For Freshers &#038; Experienced [2024]
899182
Big Data – the name indicates voluminous data, which can be both structured and unstructured. Many companies collect, curate, and store data, but how
Read More

by Rohit Sharma

04 Sep 2023

Schedule 1:1 free counsellingTalk to Career Expert
icon
footer sticky close icon