Identify the Type of animal (7 Types) based on the available attributes

Machine Learning Project Animal Classification. The study of the classification of types of animal, Identify the Type of animal (7 Types) based on the available attributes

Data Set Information

A simple database containing 17 Boolean-valued attributes. The “type” attribute appears to be the class attribute. Here is a breakdown of which animals are in which type:

Class# — Set of animals:
====== ====================================================
1 — (41) aardvark, antelope, bear, boar, buffalo, calf, cavy, cheetah, deer, dolphin, elephant, fruitbat, giraffe, girl, goat, gorilla, hamster, hare, leopard, lion, lynx, mink, mole, mongoose, opossum, oryx, platypus, polecat, pony, porpoise, puma, pussycat, raccoon, reindeer, seal, sealion, squirrel, vampire, vole, wallaby,wolf
2 — (20) chicken, crow, dove, duck, flamingo, gull, hawk, kiwi, lark, ostrich, parakeet, penguin, pheasant, rhea, skimmer, skua, sparrow, swan, vulture, wren
3 — (5) pitviper, seasnake, slowworm, tortoise, tuatara
4 — (13) bass, carp, catfish, chub, dogfish, haddock, herring, pike, piranha, seahorse, sole, stingray, tuna
5 — (4) frog, frog, newt, toad
6 — (8) flea, gnat, honeybee, housefly, ladybird, moth, termite, wasp
7 — (10) clam, crab, crayfish, lobster, octopus, scorpion, seawasp, slug, starfish, worm

Attribute Information

animal name: Unique for each instance
hair: Boolean
feathers: Boolean
eggs: Boolean
milk: Boolean
airborne: Boolean
aquatic: Boolean
predator: Boolean
toothed: Boolean
backbone: Boolean
breathes: Boolean
venomous: Boolean
fins: Boolean
legs: Numeric (set of values: {0,2,4,5,6,8})
tail: Boolean
domestic: Boolean
catsize: Boolean
type: Numeric (integer values in range [1,7])

Technology Used

Apache Spark
Spark SQL
Apache Spark MLLib
Scala
DataFrame-based API
Databricks Notebook

Challenges

Process .data file (ie file with .data as Extensions) with user define a schema for data

Convert String data to Numeric format so we can process the data in Apache Spark ML Library.

Introduction

Welcome to this project on creating prediction model to Identify Type of Animal in Apache Spark Machine Learning using Databricks platform community edition server which allows you to execute your spark code, free of cost on their server just by registering through email id.

In this project we explore Apache Spark and Machine Learning on the Databricks platform.

I am a firm believer that the best way to learn is by doing. That’s why I haven’t included any purely theoretical lectures in this tutorial: you will learn everything on the way and be able to put it into practice straight away. Seeing the way each feature works will help you learn Apache Spark machine learning thoroughly by heart.

We’re going to look at how to set up a Spark Cluster and get started with that. And we’ll look at how we can then use that Spark Cluster to take data coming into that Spark Cluster, process that data using a Machine Learning model, and generate some sort of output in the form of a prediction. That’s pretty much what we’re going to learn about predictive model.

In this project we will be Predict the type of Animal using Decision Tree Classification algorithm.

We will learn:

Preparing the Data for Processing.
Basics flow of data in Apache Spark, loading data, and working with data, this course shows you how Apache Spark is perfect for a Machine Learning job.
Learn the basics of Databricks notebook by enrolling in Free Community Edition Server
Define the Machine Learning Pipeline
Train a Machine Learning Model
Testing a Machine Learning Model
Evaluating a Machine Learning Model (i.e. Examine the Predicted and Actual Values)

The goal is to provide you with practical tools that will be beneficial for you in the future. While doing that, you’ll develop a model with a real use opportunity.

I am really excited you are here, I hope you are going to follow all the way to the end of the Project. It is fairly straight forward fairly easy to follow through the article we will show you step by step each line of code & we will explain what it does and why we are doing it.

Free Account creation in Databricks

Click Here

Creating a Spark Cluster

Click Here

Basics about Databricks notebook

Click Here

Loading Data into Databricks Environment

Click Here

Download Data

Click Here

Defining User Define Schema to Load Data in Dataframe

%scala

import org.apache.spark.sql.Encoders
import org.apache.spark.sql.types

case class Animal(animalname: String, 
                hair: Integer, 
                feathers: Integer, 
                eggs: Integer, 
                milk: Integer, 
                airborne: Integer, 
                aquatic: Integer, 
                predator: Integer, 
                toothed: Integer,
                backbone: Integer, 
                breathes: Integer, 
                venomous: Integer, 
                fins: Integer, 
                legs: Integer, 
                tail: Integer, 
                domestic: Integer, 
                catsize: Integer, 
                types: Integer)

val AnimalSchema = Encoders.product[Animal].schema

val AnimalDF = spark.read.schema(AnimalSchema).option("header", "false").csv("/FileStore/tables/zoo.data")

AnimalDF.show()

Output:

+----------+----+--------+----+----+--------+-------+--------+-------+--------+--------+--------+----+----+----+--------+-------+-----+
|animalname|hair|feathers|eggs|milk|airborne|aquatic|predator|toothed|backbone|breathes|venomous|fins|legs|tail|domestic|catsize|types|
+----------+----+--------+----+----+--------+-------+--------+-------+--------+--------+--------+----+----+----+--------+-------+-----+
|  aardvark|   1|       0|   0|   1|       0|      0|       1|      1|       1|       1|       0|   0|   4|   0|       0|      1|    1|
|  antelope|   1|       0|   0|   1|       0|      0|       0|      1|       1|       1|       0|   0|   4|   1|       0|      1|    1|
|      bass|   0|       0|   1|   0|       0|      1|       1|      1|       1|       0|       0|   1|   0|   1|       0|      0|    4|
|      bear|   1|       0|   0|   1|       0|      0|       1|      1|       1|       1|       0|   0|   4|   0|       0|      1|    1|
|      boar|   1|       0|   0|   1|       0|      0|       1|      1|       1|       1|       0|   0|   4|   1|       0|      1|    1|
|   buffalo|   1|       0|   0|   1|       0|      0|       0|      1|       1|       1|       0|   0|   4|   1|       0|      1|    1|
|      calf|   1|       0|   0|   1|       0|      0|       0|      1|       1|       1|       0|   0|   4|   1|       1|      1|    1|
|      carp|   0|       0|   1|   0|       0|      1|       0|      1|       1|       0|       0|   1|   0|   1|       1|      0|    4|
|   catfish|   0|       0|   1|   0|       0|      1|       1|      1|       1|       0|       0|   1|   0|   1|       0|      0|    4|
|      cavy|   1|       0|   0|   1|       0|      0|       0|      1|       1|       1|       0|   0|   4|   0|       1|      0|    1|
|   cheetah|   1|       0|   0|   1|       0|      0|       1|      1|       1|       1|       0|   0|   4|   1|       0|      1|    1|
|   chicken|   0|       1|   1|   0|       1|      0|       0|      0|       1|       1|       0|   0|   2|   1|       1|      0|    2|
|      chub|   0|       0|   1|   0|       0|      1|       1|      1|       1|       0|       0|   1|   0|   1|       0|      0|    4|
|      clam|   0|       0|   1|   0|       0|      0|       1|      0|       0|       0|       0|   0|   0|   0|       0|      0|    7|
|      crab|   0|       0|   1|   0|       0|      1|       1|      0|       0|       0|       0|   0|   4|   0|       0|      0|    7|
|  crayfish|   0|       0|   1|   0|       0|      1|       1|      0|       0|       0|       0|   0|   6|   0|       0|      0|    7|
|      crow|   0|       1|   1|   0|       1|      0|       1|      0|       1|       1|       0|   0|   2|   1|       0|      0|    2|
|      deer|   1|       0|   0|   1|       0|      0|       0|      1|       1|       1|       0|   0|   4|   1|       0|      1|    1|
|   dogfish|   0|       0|   1|   0|       0|      1|       1|      1|       1|       0|       0|   1|   0|   1|       0|      1|    4|
|   dolphin|   0|       0|   0|   1|       0|      1|       1|      1|       1|       1|       0|   1|   0|   1|       0|      1|    1|
+----------+----+--------+----+----+--------+-------+--------+-------+--------+--------+--------+----+----+----+--------+-------+-----+
only showing top 20 rows

Collecting all String Columns into an Array

%scala

var StringfeatureCol = Array("animalname");

StringIndexer encodes a string column of labels to a column of label indices.

Example of StringIndexer

import org.apache.spark.ml.feature.StringIndexer

val df = spark.createDataFrame(
  Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c"))
).toDF("id", "category")

df.show()

val indexer = new StringIndexer()
  .setInputCol("category")
  .setOutputCol("categoryIndex")

val indexed = indexer.fit(df).transform(df)

indexed.show()

Output:

+---+--------+
| id|category|
+---+--------+
|  0|       a|
|  1|       b|
|  2|       c|
|  3|       a|
|  4|       a|
|  5|       c|
+---+--------+

+---+--------+-------------+
| id|category|categoryIndex|
+---+--------+-------------+
|  0|       a|          0.0|
|  1|       b|          2.0|
|  2|       c|          1.0|
|  3|       a|          0.0|
|  4|       a|          0.0|
|  5|       c|          1.0|
+---+--------+-------------+

Define the Pipeline

A predictive model often requires multiple stages of feature preparation.

A pipeline consists of a series of transformer and estimator stages that typically prepare a DataFrame for modeling and then train a predictive model.

In this case, you will create a pipeline with stages:

A StringIndexer estimator that converts string values to indexes for categorical features
A VectorAssembler that combines categorical features into a single vector

%scala

import org.apache.spark.ml.attribute.Attribute
import org.apache.spark.ml.feature.{IndexToString, StringIndexer}
import org.apache.spark.ml.{Pipeline, PipelineModel}

val indexers = StringfeatureCol.map { colName =>
  new StringIndexer().setInputCol(colName).setHandleInvalid("skip").setOutputCol(colName + "_indexed")
}

val pipeline = new Pipeline()
                    .setStages(indexers)      

val AnimalFinalDF = pipeline.fit(AnimalDF).transform(AnimalDF)

Print Schema

%scala

AnimalFinalDF.printSchema()

Output:

root
 |-- animalname: string (nullable = true)
 |-- hair: integer (nullable = true)
 |-- feathers: integer (nullable = true)
 |-- eggs: integer (nullable = true)
 |-- milk: integer (nullable = true)
 |-- airborne: integer (nullable = true)
 |-- aquatic: integer (nullable = true)
 |-- predator: integer (nullable = true)
 |-- toothed: integer (nullable = true)
 |-- backbone: integer (nullable = true)
 |-- breathes: integer (nullable = true)
 |-- venomous: integer (nullable = true)
 |-- fins: integer (nullable = true)
 |-- legs: integer (nullable = true)
 |-- tail: integer (nullable = true)
 |-- domestic: integer (nullable = true)
 |-- catsize: integer (nullable = true)
 |-- types: integer (nullable = true)
 |-- animalname_indexed: double (nullable = false)

Split the Data

It is common practice when building machine learning models to split the source data, using some of it to train the model and reserving some to test the trained model. In this project, you will use 70% of the data for training, and reserve 30% for testing.

%scala

val splits = AnimalFinalDF.randomSplit(Array(0.7, 0.3))
val train = splits(0)
val test = splits(1)
val train_rows = train.count()
val test_rows = test.count()
println("Training Rows: " + train_rows + " Testing Rows: " + test_rows)

Prepare the Training Data

VectorAssembler()

VectorAssembler(): is a transformer that combines a given list of columns into a single vector column. It is useful for combining raw features and features generated by different feature transformers into a single feature vector, in order to train ML models like logistic regression and decision trees.

VectorAssembler accepts the following input column types: all numeric types, boolean type, and vector type.

In each row, the values of the input columns will be concatenated into a vector in the specified order.

%scala

import org.apache.spark.ml.feature.VectorAssembler

val assembler = new VectorAssembler().setInputCols(Array( "hair", "feathers", "eggs", "milk", "airborne", "aquatic", "predator", "toothed", "backbone", "breathes", "venomous", "fins", "legs", "tail", "domestic", "catsize" )).setOutputCol("features")

val training = assembler.transform(train).select($"features", $"types".alias("label"))

training.show(false)

Output:

+------------------------------------------------------------------+-----+
|features                                                          |label|
+------------------------------------------------------------------+-----+
|(16,[0,3,6,7,8,9,12,15],[1.0,1.0,1.0,1.0,1.0,1.0,4.0,1.0])        |1    |
|(16,[2,5,6,7,8,11,13],[1.0,1.0,1.0,1.0,1.0,1.0,1.0])              |4    |
|(16,[0,3,6,7,8,9,12,15],[1.0,1.0,1.0,1.0,1.0,1.0,4.0,1.0])        |1    |
|(16,[0,3,6,7,8,9,12,13,15],[1.0,1.0,1.0,1.0,1.0,1.0,4.0,1.0,1.0]) |1    |
|(16,[0,3,7,8,9,12,13,15],[1.0,1.0,1.0,1.0,1.0,4.0,1.0,1.0])       |1    |
|(16,[0,3,7,8,9,12,13,14,15],[1.0,1.0,1.0,1.0,1.0,4.0,1.0,1.0,1.0])|1    |
|(16,[2,5,7,8,11,13,14],[1.0,1.0,1.0,1.0,1.0,1.0,1.0])             |4    |
|(16,[2,5,6,7,8,11,13],[1.0,1.0,1.0,1.0,1.0,1.0,1.0])              |4    |
|(16,[0,3,7,8,9,12,14],[1.0,1.0,1.0,1.0,1.0,4.0,1.0])              |1    |
|(16,[0,3,6,7,8,9,12,13,15],[1.0,1.0,1.0,1.0,1.0,1.0,4.0,1.0,1.0]) |1    |
|(16,[1,2,4,8,9,12,13,14],[1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0])       |2    |
|(16,[2,5,6,12],[1.0,1.0,1.0,6.0])                                 |7    |
|(16,[1,2,4,6,8,9,12,13],[1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0])        |2    |
|(16,[2,5,6,7,8,11,13,15],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0])       |4    |
|(16,[3,5,6,7,8,9,11,13,15],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]) |1    |
|(16,[1,2,4,8,9,12,13,14],[1.0,1.0,1.0,1.0,1.0,2.0,1.0,1.0])       |2    |
|(16,[1,2,4,5,8,9,12,13],[1.0,1.0,1.0,1.0,1.0,1.0,2.0,1.0])        |2    |
|(16,[0,3,7,8,9,12,13,15],[1.0,1.0,1.0,1.0,1.0,4.0,1.0,1.0])       |1    |
|(16,[2,9,12],[1.0,1.0,6.0])                                       |6    |
|(16,[2,5,6,7,8,9,12],[1.0,1.0,1.0,1.0,1.0,1.0,4.0])               |5    |
+------------------------------------------------------------------+-----+
only showing top 20 rows

Train a Classification Model

Next, you need to train a Classification model using the training data. To do this, create an instance of the Decision Tree Classification algorithm you want to use and use its fit method to train a model based on the training DataFrame. In this project, you will use a Decision Tree Classifier algorithm – though you can use the same technique for any of the regression algorithms supported in the spark.ml API.

%scala

import org.apache.spark.ml.classification.DecisionTreeClassificationModel
import org.apache.spark.ml.classification.DecisionTreeClassifier
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator

val lr = new DecisionTreeClassifier().setMaxBins(86).setLabelCol("label").setFeaturesCol("features")

val model = lr.fit(training)

println("Model Trained!")

Prepare the Testing Data

Now that you have a trained model, you can test it using the testing data you reserved previously. First, you need to prepare the testing data in the same way as you did the training data by transforming the feature columns into a vector. This time you’ll rename the Type column to trueLabel.

%scala

val testing = assembler.transform(test).select($"features", $"types".alias("trueLabel"))
testing.show()

Output:

+--------------------+---------+
|            features|trueLabel|
+--------------------+---------+
|(16,[0,3,7,8,9,12...|        1|
|(16,[2,5,6,7,8,11...|        4|
|(16,[2,6],[1.0,1.0])|        7|
|(16,[2,5,6,12],[1...|        7|
|(16,[0,3,7,8,9,12...|        1|
|(16,[1,2,4,8,9,12...|        2|
|(16,[0,3,4,7,8,9,...|        1|
|(16,[2,4,9,12],[1...|        6|
|(16,[0,3,7,8,9,12...|        1|
|(16,[2,5,6,7,8,11...|        4|
|(16,[1,2,4,8,9,12...|        2|
|[1.0,0.0,0.0,1.0,...|        1|
|(16,[0,3,7,8,9,12...|        1|
|(16,[1,2,5,6,8,9,...|        2|
|(16,[0,3,6,7,8,9,...|        1|
|(16,[1,2,6,8,9,12...|        2|
|(16,[2,5,7,8,11,1...|        4|
|(16,[5,6,7,8,10,1...|        3|
|(16,[2,9],[1.0,1.0])|        7|
|(16,[2,5,7,8,11,1...|        4|
+--------------------+---------+
only showing top 20 rows

Test the Model

Now you’re ready to use the transform method of the model to generate some predictions. You can use this approach to predict the type of animal; but in this case, you are using the test data which includes a known true label value, so you can compare the Type of animal.

%scala

val prediction = model.transform(testing)
val predicted = prediction.select("features", "prediction", "trueLabel")
predicted.show(100)

Output:

+--------------------+----------+---------+
|            features|prediction|trueLabel|
+--------------------+----------+---------+
|(16,[0,3,7,8,9,12...|       1.0|        1|
|(16,[2,5,6,7,8,11...|       4.0|        4|
|(16,[2,6],[1.0,1.0])|       7.0|        7|
|(16,[2,5,6,12],[1...|       7.0|        7|
|(16,[0,3,7,8,9,12...|       1.0|        1|
|(16,[1,2,4,8,9,12...|       2.0|        2|
|(16,[0,3,4,7,8,9,...|       1.0|        1|
|(16,[2,4,9,12],[1...|       6.0|        6|
|(16,[0,3,7,8,9,12...|       1.0|        1|
|(16,[2,5,6,7,8,11...|       4.0|        4|
|(16,[1,2,4,8,9,12...|       2.0|        2|
|[1.0,0.0,0.0,1.0,...|       1.0|        1|
|(16,[0,3,7,8,9,12...|       1.0|        1|
|(16,[1,2,5,6,8,9,...|       2.0|        2|
|(16,[0,3,6,7,8,9,...|       1.0|        1|
|(16,[1,2,6,8,9,12...|       2.0|        2|
|(16,[2,5,7,8,11,1...|       4.0|        4|
|(16,[5,6,7,8,10,1...|       5.0|        3|
|(16,[2,9],[1.0,1.0])|       7.0|        7|
|(16,[2,5,7,8,11,1...|       4.0|        4|
|(16,[2,8,9,12,13,...|       7.0|        3|
|(16,[1,2,4,8,9,12...|       2.0|        2|
+--------------------+----------+---------+

Evaluating a Model (We got 90% Accuracy)

%scala

val evaluator = new MulticlassClassificationEvaluator()
  .setLabelCol("trueLabel")
  .setPredictionCol("prediction")
  .setMetricName("accuracy")
val accuracy = evaluator.evaluate(prediction)

Output:

evaluator: org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator = MulticlassClassificationEvaluator: uid=mcEval_0fb4d9356960, metricName=accuracy, metricLabel=0.0, beta=1.0, eps=1.0E-15
accuracy: Double = 0.9090909090909091

Identify the Type of animal (7 Types) based on the available attributes

Data Set Information

Attribute Information

Technology Used

Challenges

Introduction

Free Account creation in Databricks

Creating a Spark Cluster

Basics about Databricks notebook

Loading Data into Databricks Environment

Download Data

Defining User Define Schema to Load Data in Dataframe

Collecting all String Columns into an Array

StringIndexer encodes a string column of labels to a column of label indices.

Example of StringIndexer

Define the Pipeline

Print Schema

Split the Data

Prepare the Training Data

VectorAssembler()

Train a Classification Model

Prepare the Testing Data

Test the Model

Evaluating a Model (We got 90% Accuracy)

By Bhavesh

Tags

Identify the Type of animal (7 Types) based on the available attributes

Data Set Information​

Attribute Information​

Technology Used

Challenges

Introduction

Free Account creation in Databricks

Creating a Spark Cluster

Basics about Databricks notebook

Loading Data into Databricks Environment

Download Data

Defining User Define Schema to Load Data in Dataframe

Collecting all String Columns into an Array

StringIndexer encodes a string column of labels to a column of label indices.

Example of StringIndexer

Define the Pipeline

Print Schema

Split the Data

Prepare the Training Data

VectorAssembler()

Train a Classification Model

Prepare the Testing Data

Test the Model

Evaluating a Model (We got 90% Accuracy)

By Bhavesh

Tags

Data Set Information

Attribute Information