Machine Learning Project for Glass Identification.
Problem Statement or Business Problem
From USA Forensic Science Service; 6 types of glass; defined in terms of their oxide content (i.e. Na, Fe, K, etc)
The study of the classification of types of glass was motivated by the criminological investigation. At the scene of the crime, the glass left can be used as evidence…if it is correctly identified!
Attribute Information or Dataset Details:
- Id number: 1 to 214
- RI: refractive index
- Na: Sodium (unit measurement: weight percent in corresponding oxide, as are attributes 4-10)
- Mg: Magnesium
- Al: Aluminum
- Si: Silicon
- K: Potassium
- Ca: Calcium
- Ba: Barium
- Fe: Iron
- Type of glass: (class attribute)
- building_windows_float_processed = 1
- building_windows_non_float_processed = 2
- vehicle_windows_float_processed = 3
- vehicle_windows_non_float_processed (none in this database) = 4
- containers = 5
- tableware = 6
- headlamps = 7
Technology Used
- Apache Spark
- Spark SQL
- Apache Spark MLLib
- Scala
- DataFrame-based API
- Databricks Notebook
Challenges
Process .data file (ie file with .data as Extensions) with user define a schema for data
Convert String data to Numeric format so we can process the data in Apache Spark ML Library.
Introduction
Welcome to this project on predicting the type of Glass in Apache Spark Machine Learning using Databricks platform community edition server which allows you to execute your spark code, free of cost on their server just by registering through email id.
In this project we explore Apache Spark and Machine Learning on the Databricks platform.
I am a firm believer that the best way to learn is by doing. That’s why I haven’t included any purely theoretical lectures in this tutorial: you will learn everything on the way and be able to put it into practice straight away. Seeing the way each feature works will help you learn Apache Spark machine learning thoroughly by heart.
We’re going to look at how to set up a Spark Cluster and get started with that. And we’ll look at how we can then use that Spark Cluster to take data coming into that Spark Cluster, process that data using a Machine Learning model, and generate some sort of output in the form of a prediction. That’s pretty much what we’re going to learn about predictive model.
In this project we will be predicting the type of Glass using the Decision Tree Classification Model algorithm.
We will learn:
- Preparing the Data for Processing.
- Basic flow of data in Apache Spark, loading data, and working with data, this course shows you how Apache Spark is perfect for Machine Learning jobs.
- Learn basics of Databricks notebook by enrolling into Free Community Edition Server
- Define the Machine Learning Pipeline
- Train a Machine Learning Model
- Testing a Machine Learning Model
- Evaluating a Machine Learning Model (i.e. Examine the Predicted and Actual Values)
The Goal is to provide you with practical tools that will be beneficial for you in the future. While doing that, you’ll develop a model with a real use opportunity.
I am really excited you are here, I hope you are going to follow all the way to the end of the Project. It is straightforward and fairly easy to follow through the article. We will show you step by step each line of code & we will explain what it does and why we are doing it.
Free Account creation in Databricks
Creating a Spark Cluster
Basics about Databricks notebook
Loading Data into Databricks Environment
Download Data
Defining User Define Schema to Load Data in Dataframe
%scala
import org.apache.spark.sql.Encoders
case class Glass(Id_number: Double,
refractive_index: Double,
Sodium : Double,
Magnesium: Double,
Aluminum: Double,
Silicon: Double,
Potassium: Double,
Calcium: Double,
Barium: Double,
Iron: Double,
Type_of_glass: Double
)
val GlassSchema = Encoders.product[Glass].schema
val GlassDF = spark.read.schema(GlassSchema).option("header", "false").csv("/FileStore/tables/glass.data")
GlassDF.show(10)
Output:
+---------+----------------+------+---------+--------+-------+---------+-------+------+----+-------------+
|Id_number|refractive_index|Sodium|Magnesium|Aluminum|Silicon|Potassium|Calcium|Barium|Iron|Type_of_glass|
+---------+----------------+------+---------+--------+-------+---------+-------+------+----+-------------+
| 1.0| 1.52101| 13.64| 4.49| 1.1| 71.78| 0.06| 8.75| 0.0| 0.0| 1.0|
| 2.0| 1.51761| 13.89| 3.6| 1.36| 72.73| 0.48| 7.83| 0.0| 0.0| 1.0|
| 3.0| 1.51618| 13.53| 3.55| 1.54| 72.99| 0.39| 7.78| 0.0| 0.0| 1.0|
| 4.0| 1.51766| 13.21| 3.69| 1.29| 72.61| 0.57| 8.22| 0.0| 0.0| 1.0|
| 5.0| 1.51742| 13.27| 3.62| 1.24| 73.08| 0.55| 8.07| 0.0| 0.0| 1.0|
| 6.0| 1.51596| 12.79| 3.61| 1.62| 72.97| 0.64| 8.07| 0.0|0.26| 1.0|
| 7.0| 1.51743| 13.3| 3.6| 1.14| 73.09| 0.58| 8.17| 0.0| 0.0| 1.0|
| 8.0| 1.51756| 13.15| 3.61| 1.05| 73.24| 0.57| 8.24| 0.0| 0.0| 1.0|
| 9.0| 1.51918| 14.04| 3.58| 1.37| 72.08| 0.56| 8.3| 0.0| 0.0| 1.0|
| 10.0| 1.51755| 13.0| 3.6| 1.36| 72.99| 0.57| 8.4| 0.0|0.11| 1.0|Printing Schema of Dataframe
%scala GlassDF.printSchema() Output: root |-- Id_number: double (nullable = true) |-- refractive_index: double (nullable = true) |-- Sodium: double (nullable = true) |-- Magnesium: double (nullable = true) |-- Aluminum: double (nullable = true) |-- Silicon: double (nullable = true) |-- Potassium: double (nullable = true) |-- Calcium: double (nullable = true) |-- Barium: double (nullable = true) |-- Iron: double (nullable = true) |-- Type_of_glass: double (nullable = true)
Get Statistics of Data
%scala
GlassDF.select("Id_number", "refractive_index", "Sodium", "Magnesium", "Aluminum", "Silicon", "Potassium", "Calcium", "Barium", "Iron", "Type_of_glass" ).describe().show()
Output:
+-------+-----------------+--------------------+------------------+-----------------+------------------+------------------+-------------------+------------------+-------------------+--------------------+------------------+
|summary| Id_number| refractive_index| Sodium| Magnesium| Aluminum| Silicon| Potassium| Calcium| Barium| Iron| Type_of_glass|
+-------+-----------------+--------------------+------------------+-----------------+------------------+------------------+-------------------+------------------+-------------------+--------------------+------------------+
| count| 214| 214| 214| 214| 214| 214| 214| 214| 214| 214| 214|
| mean| 107.5| 1.5183654205607469|13.407850467289723|2.684532710280374|1.4449065420560752| 72.65093457943925|0.49705607476635494| 8.95696261682243|0.17504672897196263|0.057009345794392506|2.7803738317757007|
| stddev|61.92064814475594|0.003036863739385533|0.8166035557149831|1.442407844870442|0.4992696456004845|0.7745457947651084| 0.6521918455589797|1.4231534872813942|0.49721926059970345| 0.09743870063650084|2.1037386462007546|
| min| 1.0| 1.51115| 10.73| 0.0| 0.29| 69.81| 0.0| 5.43| 0.0| 0.0| 1.0|
| max| 214.0| 1.53393| 17.38| 4.49| 3.5| 75.41| 6.21| 16.19| 3.15| 0.51| 7.0|
+-------+-----------------+--------------------+------------------+-----------------+------------------+------------------+-------------------+------------------+-------------------+--------------------+------------------+Creating a Classification Model
In this tutorial , you will implement a classification model that uses features of glass to predict the type of glass
Import Spark SQL and Spark ML Libraries
First, import the libraries you will need:
%scala import org.apache.spark.sql.functions._ import org.apache.spark.sql.Row import org.apache.spark.sql.types._ import org.apache.spark.ml.feature.VectorAssembler
Split the Data
It is common practice when building machine learning models to split the source data, using some of it to train the model and reserving some to test the trained model. In this project, you will use 70% of the data for training, and reserve 30% for testing.
%scala
val splits = GlassDF.randomSplit(Array(0.7, 0.3))
val train = splits(0)
val test = splits(1)
val train_rows = train.count()
val test_rows = test.count()
println("Training Rows: " + train_rows + " Testing Rows: " + test_rows)Prepare the Training Data
To train the Regression model, you need a training data set that includes a vector of numeric features, and a label column. In this project, you will use the VectorAssembler class to transform the feature columns into a vector, and then rename the type of glass column to the label.
VectorAssembler()
VectorAssembler(): is a transformer that combines a given list of columns into a single vector column. It is useful for combining raw features and features generated by different feature transformers into a single feature vector, in order to train ML models like logistic regression and decision trees.
VectorAssembler accepts the following input column types: all numeric types, boolean type, and vector type.
In each row, the values of the input columns will be concatenated into a vector in the specified order.
%scala
import org.apache.spark.ml.feature.VectorAssembler
val assembler = new VectorAssembler().setInputCols(Array("Id_number", "refractive_index", "Sodium", "Magnesium", "Aluminum", "Silicon", "Potassium", "Calcium", "Barium", "Iron")).setOutputCol("features")
val training = assembler.transform(train).select($"features", $"Type_of_glass".alias("label"))
training.show()
Output:
+--------------------+-----+
| features|label|
+--------------------+-----+
|[1.0,1.52101,13.6...| 1.0|
|[2.0,1.51761,13.8...| 1.0|
|[3.0,1.51618,13.5...| 1.0|
|[4.0,1.51766,13.2...| 1.0|
|[5.0,1.51742,13.2...| 1.0|
|[6.0,1.51596,12.7...| 1.0|
|[7.0,1.51743,13.3...| 1.0|
|[9.0,1.51918,14.0...| 1.0|
|[11.0,1.51571,12....| 1.0|
|[12.0,1.51763,12....| 1.0|
|[13.0,1.51589,12....| 1.0|
|[14.0,1.51748,12....| 1.0|
|[15.0,1.51763,12....| 1.0|
Train a Classification Model
Next, you need to train a classification model using the training data. To do this, create an instance of the Decision Tree Classification Model algorithm you want to use and use its fit method to train a model based on the training DataFrame. In this exercise, you will use a Decision Tree Classification Model algorithm – though you can use the same technique for any of the classification algorithms supported in the spark.ml API.
%scala
import org.apache.spark.ml.classification.DecisionTreeClassificationModel
import org.apache.spark.ml.classification.DecisionTreeClassifier
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
val lr = new DecisionTreeClassifier().setLabelCol("label").setFeaturesCol("features")
val model = lr.fit(training)
println("Model Trained!")Prepare the Testing Data
Now that you have a trained model, you can test it using the testing data you reserved previously. First, you need to prepare the testing data in the same way as you did the training data by transforming the feature columns into a vector. This time you’ll rename the type of glass column to trueLabel.
%scala
val testing = assembler.transform(test).select($"features", $"Type_of_glass".alias("trueLabel"))
testing.show()
Output:
+--------------------+---------+
| features|trueLabel|
+--------------------+---------+
|[8.0,1.51756,13.1...| 1.0|
|[10.0,1.51755,13....| 1.0|
|[18.0,1.52196,14....| 1.0|
|[21.0,1.5175,12.8...| 1.0|
|[23.0,1.51736,12....| 1.0|
|[28.0,1.51721,12....| 1.0|
|[30.0,1.51784,13....| 1.0|
|[33.0,1.51775,12....| 1.0|
|[34.0,1.51753,12....| 1.0|
|[40.0,1.52213,14....| 1.0|
|[43.0,1.51779,13....| 1.0|
|[44.0,1.5221,13.7...| 1.0|
|[46.0,1.519,13.49...| 1.0|
|[47.0,1.51869,13....| 1.0|
|[48.0,1.52667,13....| 1.0|
|[49.0,1.52223,13....| 1.0|
|[50.0,1.51898,13....| 1.0|
|[53.0,1.51808,13....| 1.0|
|[57.0,1.51215,12....| 1.0|
|[60.0,1.51754,13....| 1.0|
+--------------------+---------+
only showing top 20 rows
Test the Model
Now you’re ready to use the transform method of the model to generate some predictions. You can use this approach to predict the type of glass; but in this case, you are using the test data which includes a known true label value, so you can compare the predicted Type of glass.
%scala
val prediction = model.transform(testing)
val predicted = prediction.select("features", "prediction", "trueLabel")
predicted.show(100)
Output:
+--------------------+----------+---------+
| features|prediction|trueLabel|
+--------------------+----------+---------+
|[8.0,1.51756,13.1...| 1.0| 1.0|
|[10.0,1.51755,13....| 1.0| 1.0|
|[18.0,1.52196,14....| 1.0| 1.0|
|[21.0,1.5175,12.8...| 1.0| 1.0|
|[23.0,1.51736,12....| 1.0| 1.0|
|[28.0,1.51721,12....| 1.0| 1.0|
|[30.0,1.51784,13....| 1.0| 1.0|
|[33.0,1.51775,12....| 1.0| 1.0|
|[34.0,1.51753,12....| 1.0| 1.0|
|[40.0,1.52213,14....| 1.0| 1.0|
|[43.0,1.51779,13....| 1.0| 1.0|
|[44.0,1.5221,13.7...| 1.0| 1.0|
|[46.0,1.519,13.49...| 1.0| 1.0|
|[47.0,1.51869,13....| 1.0| 1.0|
|[48.0,1.52667,13....| 1.0| 1.0|
|[49.0,1.52223,13....| 1.0| 1.0|
|[50.0,1.51898,13....| 1.0| 1.0|
|[53.0,1.51808,13....| 1.0| 1.0|
|[57.0,1.51215,12....| 1.0| 1.0|
|[60.0,1.51754,13....| 1.0| 1.0|
|[65.0,1.52172,13....| 1.0| 1.0|
|[67.0,1.52152,13....| 1.0| 1.0|
|[69.0,1.52152,13....| 1.0| 1.0|
|[70.0,1.523,13.31...| 1.0| 1.0|
|[72.0,1.51848,13....| 1.0| 2.0|
|[74.0,1.51631,13....| 2.0| 2.0|
|[76.0,1.5159,13.0...| 2.0| 2.0|Evaluating a Model (We got 97% Accuracy)
%scala
val evaluator = new MulticlassClassificationEvaluator()
.setLabelCol("trueLabel")
.setPredictionCol("prediction")
.setMetricName("accuracy")
val accuracy = evaluator.evaluate(prediction)
Output:
evaluator: org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator = MulticlassClassificationEvaluator: uid=mcEval_c8e98c368fa7, metricName=accuracy, metricLabel=0.0, beta=1.0, eps=1.0E-15
accuracy: Double = 0.9714285714285714
