Sentiment Analysis on Demonetization in India using Apache Spark

In this article, We have explored the Sentiments of People in India during Demonetization. Even by using small data, I could still gain a lot of valuable insights. I have used Spark SQL and Inbuild graphs provided by Databricks.
India is the second-most populous country in the world, with over 1.271 billion people, more than a sixth of the world’s population. Let us find out the views of different people on the demonetization by analyzing the tweets from Twitter.

Attribute Information or Dataset Details:

col_namedata_typecomment
retweetedstringnull
isRetweetstringnull
retweetCountstringnull
screenNamestringnull
statusSourcestringnull
replyToUIDstringnull
idstringnull
replyToSIDstringnull
truncatedstringnull
createdstringnull
replyToSNstringnull
favoriteCountstringnull
favoritedstringnull
textstringnull
Xbigintnull
_c0bigintnull

Table Created in Databricks Environment

Technology Used

  • Apache Spark
  • Spark SQL
  • DataFrame-based API
  • Databricks Notebook

Free Account creation in Databricks

Creating a Spark Cluster

Basics about Databricks notebook

Code for Spark SQL to get Indias Tweet reaction during Demonetization

%sql 

select sum(retweetCount) as RetweetCount,created from demonetization group by created

Plot Option for Chart

Code for Spark SQL to get Types of Devices used for Tweet

%sql 

select sum(retweetCount), substring_index(substring_index(statusSource, ">", -2),"<",1) as status_source
from demonetization group by substring_index(substring_index(statusSource, ">", -2),"<",1) 

Plot Option for Pie Chart

Code for Spark SQL to get Number of Retweet During Demonetization

%sql 

select sum(favoriteCount),created
from demonetization group by created

Code for Spark SQL to get Reaction of People on Demonetization

%sql 

select CASE 
 WHEN text like '%Respect%' THEN "POSITIVE"
 WHEN text like '%symptom%' THEN "POSITIVE"
 WHEN text like '%terrorists%' THEN "POSITIVE"\
 WHEN text like '%National%' THEN "POSITIVE"
 WHEN text like '%reform%' THEN"POSITIVE"
 WHEN text like '%support%' THEN "POSITIVE"
 WHEN text like '%#CorruptionFreeIndia%' THEN "POSITIVE"
 WHEN text like '%respect%' THEN "POSITIVE"
 WHEN text like '%Gandhi%' THEN "POSITIVE"
 WHEN text like '%vote%' THEN "POSITIVE"
 WHEN text like '%fishy%' THEN "NEGATIVE"
 WHEN text like '%disclosure%' THEN "NEGATIVE"
WHEN text like '%Reddy Wedding%' THEN "NEGATIVE"
WHEN text like '%protesting%' THEN "NEGATIVE"
WHEN text like '%hards%' THEN "NEGATIVE"
WHEN text like '%Kerala%' THEN "NEGATIVE"
WHEN text like '%hurt%' THEN "NEGATIVE"
WHEN text like '%USELESS%' THEN "NEGATIVE"
WHEN text like '%Disaster!%' THEN "NEGATIVE"
WHEN text like '%Black%' THEN "NEGATIVE"
WHEN text like '%negative%' THEN "NEGATIVE"
WHEN text like '%impact%' THEN "NEGATIVE"
WHEN text like '%opposing%' THEN "NEGATIVE"
ELSE "NEUTRAL"
     END AS Reaction
from demonetization 
By Bhavesh