DATA ENGINEERING BOOTCAMP

Python for Data Engineering

Start with the basics of Python programming and learn how to use existing functions and create user definedfunctions. Work on different Python packages such as Pandas andNumPy and understand visualizations using Python

Topics:

• Introduction to Python
• Code and Data
• Building Blocks
• Strings
• Data Structures
• Flow Control
• Functions
• Modules
• Files
• NumPy
• Pandas
• Regular Expressions
• Visualization

Design and Architecture of a Relational Database

Learn database structure and design and database modelling concepts. Participants will learn database modelling methodologies and On-Prem vs Cloud databases.

Topics:

• Introduction to Relational Database
• Architecture of Relational database
• Important Aspects of Relational Databases
• Database Structure and Design
• Database Design
• Data modelling methodologies
• SQL Components
• Transaction and Concurrency
• Database Joins and Performance Tuning
• Backup and Recovery
• On-Prem vs Cloud databases

Data Analysis using SQL

Understand essential SQL database Commands and learn to filter data and operators in SQL. Learn about Aggregation and Summary functions and combining tables and explore advanced Data Analysis using SQL.

Topics

• What is SQL and Why is it Important?
• SQL Database Admin Commands
• The Basics of SQL
• Filtering Data Using WHERE Clause in SQL
• Aggregation and Summary Functions in SQL
• Miscellaneous Analysis in SQL
• Table to Table Relationship in SQL
• Combining Tables
• Advanced SQL Data Analysis
• Making Efficient Analysis

MongoDB

Master the intricacies of database management with a focus on Schema Design, modeling, and advanced techniques like replication and sharding. Elevate your expertise further by delving into MongoDB on the cloud, and gaining hands-on experience in crafting robust data architectures. This integrated learning experience not only equips you with the skills to design and model databases effectively but also enables you to harness the power of cloud-based MongoDB solutions, positioning you as a proficient and versatile data professional in the dynamic landscape of modern database technologies.

Topics

• Introduction to MongoDB
• MongoDB Fundamentals
• CRUD Operations
• Schema Design and Modelling
• Advanced Operations
• Replication and Sharding
• Administration and Security
• MongoDB with other Applications
• MongoDB in the Cloud

Big Data Processing using Hadoop

Understand distributed storage, computing, and MapReduce processing. Learn how to ingest structured and unstructured data in Hadoop using Sqoop and Flume. Understand how to process Big Data using MapReduce, Pig, and Hive and run Big Data loads on AWS EMR and AWS S3.

Topics:

• Introduction to Big Data and Hadoop
• Hadoop Distributed File System – HDFS and YARN
• MapReduce Processing in Hadoop
• Data Ingestion and Egestion into Hadoop
• Data Processing in Hadoop
• NoSQL and HBase
• Apache Oozie
• Introduction to Spark
• Hadoop Cloud on Amazon/Elastic MapReduce

Streaming Big Data with Spark

Master development of Spark applications using the interactive shell or as batch applications. Understand the Spark application and runtime architecture. Understand how Structured Streaming works and understand use cases for Spark Streaming and Structured Streaming.

Topics:

• The Spark Runtime
• ETL with Spark
• NLP SparkSQL and DataFrames
• Introduction to Stream Processing with Spark
• Stateful processing with Spark Streaming
• Sliding Window Operations with Spark Streaming
• Introduction to Structured Streaming
• Introduction to Apache Kafka
• Kafka Integration with Spark Streaming
• Kafka Integration with Structured Streaming
• Using Spark Streaming with Kinesis (Part-1)
• Using Spark Streaming with Kinesis (Part-2)
• Additional Spark Streaming Integrations

Apache Kafka

Understand Apache Kafka Ecosystem, Architecture, Core Concepts, and Operations and learn to write code for Kafka. Learn to connect Kafka to third-party systems and learn Stream Processing with Kafka.

Topics:

• Why Kafka?
• First Steps with Kafka
• Kafka as a Distributed Log
• Reliability and Performance
• Setting up a Development Project
• Producing Messages with Kafka
• Consuming Messages
• Improving the Reliability and Performance of Our Clients
• What is Kafka Connect?
• Kafka Streams
• Stateless Stream Processing
• Stateful Stream Processing
• Securing Apache Kafka
• Real-world Use Cases of Apache Kafka

Big Data on AWS

Understand the AWS Services stack for Big Data Analytics and learn Data Collection, Catalogs, and preparation. Learn more about storing and processing Big Data on AWS and advance ML features on EMR.

Topics:

• Big Data and AWS
• Data Collection, Catalogs, and Preparation
• Storing Large data on AWS
• Processing your data on AWS
• Advanced Topics on Big Data