Streaming Data Architectures: Processing Streaming Data

Streaming Data Architectures: Processing Streaming Data

Overview/Description
Expected Duration
Lesson Objectives
Course Number
Expertise Level

Overview/Description

Spark is an analytics engine built on Hadoop that works with big data, data science and processing batch, and streaming data. In this 11-video course, discover how to develop applications in Spark to work with streaming data and explore different ways to process streams and generate output. Key concepts covered here include installing the latest version of PySpark; configuring a streaming data source using Netcat and writing applications to process the stream; and effects of using the Update mode for output of your stream processing application. Learn how to write an application to listen for new files added to a directory; compare the Append output to the Update mode and distinguish between the two; and develop applications that limit files processed in each trigger and use Spark's Complete mode for output. Next, learners perform aggregation operations on streaming data with the DataFrame API (application programming interface); work with Spark SQL to process streaming data by using SQL queries; and learn ways to use Spark for streaming data and ways to process streams and generate output.

Expected Duration (hours)
0.9

Lesson Objectives

Streaming Data Architectures: Processing Streaming Data

Course Overview

install the latest available version of PySpark

configure a streaming data source using Netcat and write an application to process the stream

describe the effects of using the Update mode for the output of your stream processing application

write an application to listen for new files being added to a directory and process them as soon as they come in

compare the Append output to the Update mode and distinguish between the two

develop applications that limit the files processed in each trigger and use Spark's Complete mode for the output

perform aggregation operations on streaming data using the DataFrame API

work with Spark SQL in order to process streaming data using SQL queries

define and apply standard, re-usable transformations for streaming data

recall they key ways to use Spark for streaming data and explore the ways to process streams and generate output

Course Number:
it_dssdardj_02_enus

Expertise Level
Intermediate