Classifying gender based on personal preferences Part 1

Gender is a social construct. The way males and females are treated differently since birth moulds their behaviour and personal preferences into what society expects for their gender.

This small dataset is designed to provide an idea about whether a person’s gender can be predicted with an accuracy significantly above 50% based on their personal preferences.

Attribute Information or Dataset Details:

  1. FavoriteColor
  2. FavoriteMusicGenre
  3. FavoriteBeverage
  4. FavoriteSoftDrink
  5. Gender

Technology Used​

  1. Apache Spark
  2. Spark SQL 
  3. Apache Spark MLLib
  4. Scala
  5. DataFrame-based API
  6. Databricks Notebook


Welcome to this project on Mobile Price Classification in Apache Spark Machine Learning using Databricks platform community edition server which allows you to execute your spark code, free of cost on their server just by registering through email id.

In this project we explore Apache Spark and Machine Learning on the Databricks platform.

I am a firm believer that the best way to learn is by doing. That’s why I haven’t included any purely theoretical lectures in this tutorial: you will learn everything on the way and be able to put it into practice straight away. Seeing the way each feature works will help you learn Apache Spark machine learning thoroughly by heart.

We’re going to look at how to set up a Spark Cluster and get started with that. And we’ll look at how we can then use that Spark Cluster to take data coming into that Spark Cluster, process that data using a Machine Learning model, and generate some sort of output in the form of a prediction. That’s pretty much what we’re going to learn about predictive model.

In this project we will be perform Mobile Price Classification using Logistic Regression algorithm.

We will learn:

  1. Preparing the Data for Processing.
  2. Basics flow of data in Apache Spark, loading data, and working with data, this course shows you how Apache Spark is perfect for Machine Learning job.
  3. Learn basics of Databricks notebook by enrolling into Free Community Edition Server
  4. Define the Machine Learning Pipeline
  5. Train a Machine Learning Model
  6. Testing a Machine Learning Model
  7. Evaluating a Machine Learning Model (i.e. Examine the Predicted and Actual Values)

The Goal is to provide you with practical tools that will be beneficial for you in the future. While doing that, you’ll develop a model with a real use opportunity.

I am really excited you are here, I hope you are going to follow all the way to the end of the Project. It is fairly straight forward fairly easy to follow through the article we will show you step by step each line of code & we will explain what it does and why we are doing it.

Free Account creation in Databricks

Creating a Spark Cluster

Basics about Databricks notebook

Loading Data into Databricks Environment

Download Data

Load Data in Dataframe


val csv ="inferSchema","true").option("header", "true").csv("/FileStore/tables/DataSet.csv")


|         Cool|              Rock|           Vodka|       7UP/Sprite|     F|
|      Neutral|           Hip hop|           Vodka|  Coca Cola/Pepsi|     F|
|         Warm|              Rock|            Wine|  Coca Cola/Pepsi|     F|
|         Warm|  Folk/Traditional|         Whiskey|            Fanta|     F|
|         Cool|              Rock|           Vodka|  Coca Cola/Pepsi|     F|
|         Warm|        Jazz/Blues|   Doesn't drink|            Fanta|     F|
|         Cool|               Pop|            Beer|  Coca Cola/Pepsi|     F|
|         Warm|               Pop|         Whiskey|            Fanta|     F|
|         Warm|              Rock|           Other|       7UP/Sprite|     F|
|      Neutral|               Pop|            Wine|  Coca Cola/Pepsi|     F|
|         Cool|               Pop|           Other|       7UP/Sprite|     F|
|         Warm|               Pop|           Other|       7UP/Sprite|     F|
|         Warm|               Pop|            Wine|       7UP/Sprite|     F|
|         Warm|        Electronic|            Wine|  Coca Cola/Pepsi|     F|
|         Cool|              Rock|            Beer|  Coca Cola/Pepsi|     F|
|         Warm|        Jazz/Blues|            Wine|  Coca Cola/Pepsi|     F|
|         Cool|               Pop|            Wine|       7UP/Sprite|     F|
|         Cool|              Rock|           Other|  Coca Cola/Pepsi|     F|
|         Cool|              Rock|           Other|  Coca Cola/Pepsi|     F|
|         Cool|               Pop|   Doesn't drink|       7UP/Sprite|     F|
only showing top 20 rows

Print Schema of Dataframe​




 |-- FavoriteColor: string (nullable = true)
 |-- FavoriteMusicGenre: string (nullable = true)
 |-- FavoriteBeverage: string (nullable = true)
 |-- FavoriteSoftDrink: string (nullable = true)
 |-- Gender: string (nullable = true)

Statistics of Dataframe

%%scala"FavoriteColor", "FavoriteMusicGenre", "FavoriteBeverage", "FavoriteSoftDrink", "Gender").describe().show()


|  count|           66|                66|              66|               66|    66|
|   mean|         null|              null|            null|             null|  null|
| stddev|         null|              null|            null|             null|  null|
|    min|         Cool|        Electronic|            Beer|       7UP/Sprite|     F|
|    max|         Warm|              Rock|            Wine|            Other|     M|

Create Temporary View so we can perform Spark SQL on Data



Spark SQL


select * from GenderClassificationData;

Generating Graph

By Bhavesh