Spark Streaming 

Home » Spark Streaming 

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Spark Streaming can be used to stream live data and processing can happen in real time. Spark Streaming’s ever-growing user base consists of household names like Uber, Netflix and Pinterest.When it comes to Real Time Data Analytics, Spark Streaming provides a single platform to ingest data for fast and live processing in Apache Spark. Through this blog, I will introduce you to this new exciting domain of Spark Streaming and we will go through a complete use case, Twitter Sentiment Analysis using Spark Streaming.

What is Streaming?

Data Streaming is a technique for transferring data so that it can be processed as a steady and continuous stream. Streaming technologies are becoming increasingly important with the growth of the Internet.

What Is Streaming - Spark Streaming - EdurekaFigure: What is Streaming?

Why Spark Streaming?

We can use Spark Streaming to stream real-time data from various sources like Twitter, Stock Market and Geographical Systems and perform powerful analytics to help businesses.

Spark Streaming - Spark Streaming - EdurekaFigure: Why Spark Streaming?

 

GET STARTED WITH SPARK

Spark Streaming Overview

Spark Streaming is used for processing real-time streaming data. It is a useful addition to the core Spark API. Spark Streaming enables high-throughput and fault-tolerant stream processing of live data streams.

Spark Streaming Overview - Spark Streaming - EdurekaFigure: Streams in Spark Streaming 

The fundamental stream unit is DStream which is basically a series of RDDs to process the real-time data.

Spark Streaming Features

  1. Scaling: Spark Streaming can easily scale to hundreds of nodes.
  2. Speed: It achieves low latency.
  3. Fault Tolerance: Spark has the ability to efficiently recover from failures.
  4. Integration: Spark integrates with batch and real-time processing.
  5. Business Analysis: Spark Streaming is used to track the behavior of customers which can be used in business analysis.

Spark Streaming Workflow

Spark Streaming workflow has four high-level stages. The first is to stream data from various sources. These sources can be streaming data sources like Akka, Kafka, Flume, AWS or Parquet for real-time streaming. The second type of sources includes HBase, MySQL, PostgreSQL, Elastic Search, Mongo DB and Cassandra for static/batch streaming. Once this happens, Spark can be used to perform Machine Learning on the data through its MLlib API. Further, Spark SQL is used to perform further operations on this data. Finally, the streaming output can be stored into various data storage systems like HBase, Cassandra, MemSQL, Kafka, Elastic Search, HDFS and local file system.

Overview - Spark Streaming - Edureka

Figure: Overview Of Spark Streaming

Spark Streaming Fundamentals

  1. Streaming Context
  2. DStream
  3. Caching
  4. Accumulators, Broadcast Variables and Checkpoints

Streaming Context

Streaming Context consumes a stream of data in Spark. It registers an Input DStream to produce a Receiverobject. It is the main entry point for Spark functionality. Spark provides a number of default implementations of sources like Twitter, Akka Actor and ZeroMQ that are accessible from the context.

Streaming Context - Spark Streaming - EdurekaA StreamingContext object can be created from a SparkContext object. A SparkContext represents the connection to a Spark cluster and can be used to create RDDs, accumulators and broadcast variables on that cluster.

1
2
3
import org.apache.spark._
import org.apache.spark.streaming._
var ssc = new StreamingContext(sc,Seconds(1))

DStream

Discretized Stream (DStream) is the basic abstraction provided by Spark Streaming. It is a continuous stream of data. It is received from a data source or a processed data stream generated by transforming the input stream.

DStream Operation - Spark Streaming - EdurekaFigure: Extracting words from an Input DStream

Internally, a DStream is represented by a continuous series of RDDs and each RDD contains data from a certain interval.

Input DStreams: Input DStreams are DStreams representing the stream of input data received from streaming sources.

Input DStream - Spark Streaming - EdurekaFigure: The Receiver sends data onto the Input DStream where each Batch contains RDDs

Every input DStream is associated with a Receiver object which receives the data from a source and stores it in Spark’s memory for processing.

Transformations on DStreams:

Any operation applied on a DStream translates to operations on the underlying RDDs. Transformations allow the data from the input DStream to be modified similar to RDDs. DStreams support many of the transformations available on normal Spark RDDs.

DStream Transformations - Spark Streaming - EdurekaFigure: DStream Transformations

The following are some of the popular transformations on DStreams:

map(func) map(func) returns a new DStream by passing each element of the source DStream through a function func.
flatMap(func) flatMap(func) is similar to map(func) but each input item can be mapped to 0 or more output items and returns a new DStream by passing each source element through a function func.
filter(func) filter(func) returns a new DStream by selecting only the records of the source DStream on which func returns true.
reduce(func) reduce(func) returns a new DStream of single-element RDDs by aggregating the elements in each RDD of the source DStream using a function func.
groupBy(func) groupBy(func) returns the new RDD which basically is made up with a key and corresponding list of items of that group.

LEARN SPARK FROM EXPERTS

Output DStreams: 

Output operations allow DStream’s data to be pushed out to external systems like databases or file systems. Output operations trigger the actual execution of all the DStream transformations.

Output Operations - Spark Streaming - EdurekaFigure: Output Operations on DStreams

Caching

DStreams allow developers to cache/ persist the stream’s data in memory. This is useful if the data in the DStream will be computed multiple times. This can be done using the persist() method on a DStream.

Caching - Spark Streaming - EdurekaFigure: Caching into 2 Nodes

For input streams that receive data over the network (such as Kafka, Flume, Sockets, etc.), the default persistence level is set to replicate the data to two nodes for fault-tolerance.

Accumulators, Broadcast Variables and Checkpoints

Accumulators: Accumulators are variables that are only added through an associative and commutative operation. They are used to implement counters or sums. Tracking accumulators in the UI can be useful for understanding the progress of running stages. Spark natively supports numeric accumulators. We can create named or unnamed accumulators.

Broadcast Variables: Broadcast variables allow the programmer to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. They can be used to give every node a copy of a large input dataset in an efficient manner. Spark also attempts to distribute broadcast variables using efficient broadcast algorithms to reduce communication cost.

Checkpoints: Checkpoints are similar to checkpoints in gaming. They make it run 24/7 and make it resilient to failures unrelated to the application logic.

Checkpoints - Spark Streaming - Edureka
Figure: 
Features of Checkpoints

Use Case – Twitter Sentiment Analysis

Now that we have understood the core concepts of Spark Streaming, let us solve a real-life problem using Spark Streaming.

Problem Statement: To design a Twitter Sentiment Analysis System where we populate real-time sentiments for crisis management, service adjusting and target marketing.

Applications of Sentiment Analysis:

  • Predict the success of a movie
  • Predict political campaign success
  • Decide whether to invest in a certain company
  • Targeted advertising
  • Review products and services

Spark Streaming Implementation:

Find the Pseudo Code below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
//Import the necessary packages into the Spark Program
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.SparkContext._
...
import java.io.File
object twitterSentiment {
def main(args: Array[String]) {
if (args.length < 4) {
System.err.println("Usage: TwitterPopularTags <consumer key> <consumer secret> " "<access token> <access token secret> [<filters>]")
System.exit(1)
}
StreamingExamples.setStreamingLogLevels()
//Passing our Twitter keys and tokens as arguments for authorization
val Array(consumerKey, consumerSecret, accessToken, accessTokenSecret) = args.take(4)
val filters = args.takeRight(args.length - 4)
// Set the system properties so that Twitter4j library used by twitter stream
// Use them to generate OAuth credentials
System.setProperty("twitter4j.oauth.consumerKey", consumerKey)
...
System.setProperty("twitter4j.oauth.accessTokenSecret", accessTokenSecret)
val sparkConf = new SparkConf().setAppName("twitterSentiment").setMaster("local[2]")
val ssc = new Streaming Context
val stream = TwitterUtils.createStream(ssc, None, filters)
//Input DStream transformation using flatMap
val tags = stream.flatMap { status => Get Text From The Hashtags }
//RDD transformation using sortBy and then map function
tags.countByValue()
.foreachRDD { rdd =>
val now = Get current time of each Tweet
rdd
.sortBy(_._2)
.map(x => (x, now))
//Saving our output at ~/twitter/ directory
.saveAsTextFile(s"~/twitter/$now")
}
//DStream transformation using filter and map functions
val tweets = stream.filter {t =>
val tags = t. Split On Spaces .filter(_.startsWith("#")). Convert To Lower Case
tags.exists { x => true }
}
val data = tweets.map { status =>
val sentiment = SentimentAnalysisUtils.detectSentiment(status.getText)
val tagss = status.getHashtagEntities.map(_.getText.toLowerCase)
(status.getText, sentiment.toString, tagss.toString())
}
data.print()
//Saving our output at ~/ with filenames starting like twitters
data.saveAsTextFiles("~/twitters","20000")
ssc.start()
ssc.awaitTermination()
 }
}

Click here to get the full source code of Twitter Sentiment Analysis using Spark Streaming.

Results:

The following are the results that are displayed in the Eclipse IDE while running the Twitter Sentiment Streaming program.

Eclipse Output - Spark Streaming - Edureka

Figure: Sentiment Analysis Output in Eclipse IDE

As we can see in the screenshot, all the tweets are categorized into Positive, Neutral and Negative according to the sentiment of the contents of the tweets.

The output of the Sentiments of the Tweets is stored into folders and files according to the time they were created. This output can be stored on the local file system or HDFS as necessary. The output directory looks like this:

Output Directory - Spark Streaming - EdurekaFigure: Output folders inside our ‘twitter’ project folder

Here, inside the twitter directory, we can find the usernames of the Twitter users along with the timestamp for every tweet as shown below:

Output Usernames - Spark Streaming - EdurekaFigure: Output file containing Twitter usernames with timestamp

Now that we have got the Twitter usernames and timestamp, let us look at the Sentiments and tweets stored in the main directory. Here, every tweet is followed by the sentiment emotion. This Sentiment that is stored is further used for analyzing a vast multitude of insights by companies.

Output Tweets & Sentiments - Spark Streaming - EdurekaFigure: Output file containing tweets with sentiments

Tweaking Code:

Now, let us modify our code a little to get sentiments for specific hashtags (topics). Currently, Donald Trump, the President of the United States is trending across news channels and online social media. Let us look at the sentiments associated with the keyword ‘Trump‘.

Donald Trump Sentiments - Spark Streaming - EdurekaFigure: Performing Sentiment Analysis on Tweets with ‘Trump’ Keyword

GET FULL SOURCE CODE

Source: Spark Streaming Tutorial | Twitter Sentiment Analysis Using Apache Spark | Edureka

By | 2018-02-21T16:40:48+00:00 February 1st, 2018|Spark Streaming, Technologies|