A mushroom, or toadstool, is the fleshy, spore-bearing fruiting body of a fungus, typically produced above ground on soil or on its food source.
Problem Statement or Business Problem
In this project, looking at the various properties of a mushroom, we will predict whether the mushroom is edible or poisonous.
Attribute Information or Dataset Details:
To be more understandable, let’s write properties one by one.
- classes: edible=e, poisonous=p
- cap-shape: bell=b,conical=c,convex=x,flat=f, knobbed=k,sunken=s
- cap-surface: fibrous=f,grooves=g,scaly=y,smooth=s
- cap-color: brown=n, buff=b, cinnamon=c, gray=g,green=r, pink=p, purple=u, red=e,white=w,yellow=y
- bruises: bruises=t,no=f
- odor: almond=a, anise=l, creosote=c, fishy=y, foul=f, musty=m, none=n, pungent=p,spicy=s
- gill-attachment: attached=a,descending=d,free=f,notched=n
- gill-spacing: close=c,crowded=w,distant=d
- gill-size: broad=b,narrow=n
- gill-color: black=k,brown=n,buff=b,chocolate=h,gray=g, green=r, orange=o, pink=p,purple=u,red=e,white=w,yellow=y
- stalk-shape: enlarging=e,tapering=t
- stalk-root: bulbous=b, club=c, cup=u, equal=e, rhizomorphs=z, rooted=r,missing=?
- stalk-surface-above-ring: fibrous=f,scaly=y,silky=k,smooth=s
- stalk-surface-below-ring: fibrous=f,scaly=y,silky=k,smooth=s
- stalk-color-above-ring: brown=n, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e,white=w,yellow=y
- stalk-color-below-ring: brown=n, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e,white=w,yellow=y
- veil-type: partial=p,universal=u
- veil-color: brown=n,orange=o,white=w,yellow=y
- ring-number: none=n,one=o,two=t
- ring-type: cobwebby=c, evanescent=e, flaring=f, large=l, none=n, pendant=p, sheathing=s,zone=z
- spore-print-color: black=k, brown=n, buff=b, chocolate=h, green=r, orange=o, purple=u,white=w,yellow=y
- population: abundant=a, clustered=c, numerous=n, scattered=s, several=v, solitary=y
- habitat: grasses=g, leaves=l, meadows=m, paths=p, urban=u, waste=w, woods=d
Also, following image shows mushroom parts as we mentioned above. (Image Credit goes to Infovisual)
- Cap: The cap is the top of the mushroom (and often looks sort of like a small umbrella). Mushroom caps can come in a variety of colors but most often are brown, white, or yellow.
- Gills, Pores, or Teeth: These structures appear under the mushroom’s cap. They look similar to a fish’s gills.
- Ring: The ring (sometimes called the annulus) is the remaining structure of the partial veil after the gills have pushed through.
- Stem or Stipe: The stem is the tall structure that holds the cap high above the ground.
- Volva: The volva is the protective veil that remains after the mushroom sprouted up from the ground. As the fungus grows, it breaks through the volva.
- Spores: Microscopic seeds acting as reproductive agents; they are usually released into the air and fall on a substrate to produce a new mushroom.
Technology Used
- Apache Spark
- Spark SQL
- Apache Spark MLLib
- Scala
- DataFrame-based API
- Databricks Notebook
Introduction
Welcome to this project on predict whether mushroom is edible or poisonous in Apache Spark Machine Learning using Databricks platform community edition server which allows you to execute your spark code, free of cost on their server just by registering through email id.
In this project, we explore Apache Spark and Machine Learning on the Databricks platform.
I am a firm believer that the best way to learn is by doing. That’s why I haven’t included any purely theoretical lectures in this tutorial: you will learn everything on the way and be able to put it into practice straight away. Seeing the way each feature works will help you learn Apache Spark machine learning thoroughly by heart.
We’re going to look at how to set up a Spark Cluster and get started with that. And we’ll look at how we can then use that Spark Cluster to take data coming into that Spark Cluster, a process that data using a Machine Learning model, and generate some sort of output in the form of a prediction. That’s pretty much what we’re going to learn about the predictive model.
In this project, we will be performing prediction where mushroom are edible or poisonous.
We will learn:
Preparing the Data for Processing.
Basics flow of data in Apache Spark, loading data, and working with data, this course shows you how Apache Spark is perfect for a Machine Learning job.
Learn the basics of Databricks notebook by enrolling in Free Community Edition Server
Define the Machine Learning Pipeline
Train a Machine Learning Model
Testing a Machine Learning Model
Evaluating a Machine Learning Model (i.e. Examine the Predicted and Actual Values)
The goal is to provide you with practical tools that will be beneficial for you in the future. While doing that, you’ll develop a model with a real use opportunity.
I am really excited you are here, I hope you are going to follow all the way to the end of the Project. It is fairly straight forward fairly easy to follow through the article we will show you step by step each line of code & we will explain what it does and why we are doing it.
Free Account creation in Databricks
Creating a Spark Cluster
Basics about Databricks notebook
Loading Data into Databricks Environment
Download Data
Load Data in Dataframe using User-defined Schema
Scala
x
36
1
%scala2
3
val mushroom = sqlContext.read.format("csv")4
.option("header", "true")5
.option("inferSchema", "true")6
.option("delimiter", ",")7
.load("/FileStore/tables/mushrooms-1.csv")8
9
mushroom.show()10
11
+-----+--------+----------+--------+-------+----+--------------+-----------+--------+---------+----------+---------+---------------------+---------------------+-------------------+-------------------+--------+---------+----------+--------+---------------+----------+-------+12
|class|capshape|capsurface|capcolor|bruises|odor|gillattachment|gillspacing|gillsize|gillcolor|stalkshape|stalkroot|stalksurfaceabovering|stalksurfacebelowring|stalkcolorabovering|stalkcolorbelowring|veiltype|veilcolor|ringnumber|ringtype|sporeprintcolor|population|habitat|13
+-----+--------+----------+--------+-------+----+--------------+-----------+--------+---------+----------+---------+---------------------+---------------------+-------------------+-------------------+--------+---------+----------+--------+---------------+----------+-------+14
| p| x| s| n| t| p| f| c| n| k| e| e| s| s| w| w| p| w| o| p| k| s| u|15
| e| x| s| y| t| a| f| c| b| k| e| c| s| s| w| w| p| w| o| p| n| n| g|16
| e| b| s| w| t| l| f| c| b| n| e| c| s| s| w| w| p| w| o| p| n| n| m|17
| p| x| y| w| t| p| f| c| n| n| e| e| s| s| w| w| p| w| o| p| k| s| u|18
| e| x| s| g| f| n| f| w| b| k| t| e| s| s| w| w| p| w| o| e| n| a| g|19
| e| x| y| y| t| a| f| c| b| n| e| c| s| s| w| w| p| w| o| p| k| n| g|20
| e| b| s| w| t| a| f| c| b| g| e| c| s| s| w| w| p| w| o| p| k| n| m|21
| e| b| y| w| t| l| f| c| b| n| e| c| s| s| w| w| p| w| o| p| n| s| m|22
| p| x| y| w| t| p| f| c| n| p| e| e| s| s| w| w| p| w| o| p| k| v| g|23
| e| b| s| y| t| a| f| c| b| g| e| c| s| s| w| w| p| w| o| p| k| s| m|24
| e| x| y| y| t| l| f| c| b| g| e| c| s| s| w| w| p| w| o| p| n| n| g|25
| e| x| y| y| t| a| f| c| b| n| e| c| s| s| w| w| p| w| o| p| k| s| m|26
| e| b| s| y| t| a| f| c| b| w| e| c| s| s| w| w| p| w| o| p| n| s| g|27
| p| x| y| w| t| p| f| c| n| k| e| e| s| s| w| w| p| w| o| p| n| v| u|28
| e| x| f| n| f| n| f| w| b| n| t| e| s| f| w| w| p| w| o| e| k| a| g|29
| e| s| f| g| f| n| f| c| n| k| e| e| s| s| w| w| p| w| o| p| n| y| u|30
| e| f| f| w| f| n| f| w| b| k| t| e| s| s| w| w| p| w| o| e| n| a| g|31
| p| x| s| n| t| p| f| c| n| n| e| e| s| s| w| w| p| w| o| p| k| s| g|32
| p| x| y| w| t| p| f| c| n| n| e| e| s| s| w| w| p| w| o| p| n| s| u|33
| p| x| s| n| t| p| f| c| n| k| e| e| s| s| w| w| p| w| o| p| n| s| u|34
+-----+--------+----------+--------+-------+----+--------------+-----------+--------+---------+----------+---------+---------------------+---------------------+-------------------+-------------------+--------+---------+----------+--------+---------------+----------+-------+35
only showing top 20 rows36
Print Schema of Dataframe
Scala
1
31
1
%scala2
3
mushroom.printSchema();4
5
Output:6
7
root8
|-- class: string (nullable = true)9
|-- capshape: string (nullable = true)10
|-- capsurface: string (nullable = true)11
|-- capcolor: string (nullable = true)12
|-- bruises: string (nullable = true)13
|-- odor: string (nullable = true)14
|-- gillattachment: string (nullable = true)15
|-- gillspacing: string (nullable = true)16
|-- gillsize: string (nullable = true)17
|-- gillcolor: string (nullable = true)18
|-- stalkshape: string (nullable = true)19
|-- stalkroot: string (nullable = true)20
|-- stalksurfaceabovering: string (nullable = true)21
|-- stalksurfacebelowring: string (nullable = true)22
|-- stalkcolorabovering: string (nullable = true)23
|-- stalkcolorbelowring: string (nullable = true)24
|-- veiltype: string (nullable = true)25
|-- veilcolor: string (nullable = true)26
|-- ringnumber: string (nullable = true)27
|-- ringtype: string (nullable = true)28
|-- sporeprintcolor: string (nullable = true)29
|-- population: string (nullable = true)30
|-- habitat: string (nullable = true)31
Statistics of Data
Scala
1
16
1
%scala2
3
mushroom.describe().show()4
5
Output:6
7
+-------+-----+--------+----------+--------+-------+----+--------------+-----------+--------+---------+----------+---------+---------------------+---------------------+-------------------+-------------------+--------+---------+----------+--------+---------------+----------+-------+8
|summary|class|capshape|capsurface|capcolor|bruises|odor|gillattachment|gillspacing|gillsize|gillcolor|stalkshape|stalkroot|stalksurfaceabovering|stalksurfacebelowring|stalkcolorabovering|stalkcolorbelowring|veiltype|veilcolor|ringnumber|ringtype|sporeprintcolor|population|habitat|9
+-------+-----+--------+----------+--------+-------+----+--------------+-----------+--------+---------+----------+---------+---------------------+---------------------+-------------------+-------------------+--------+---------+----------+--------+---------------+----------+-------+10
| count| 8124| 8124| 8124| 8124| 8124|8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124| 8124|11
| mean| null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null| null| null| null| null| null|12
| stddev| null| null| null| null| null|null| null| null| null| null| null| null| null| null| null| null| null| null| null| null| null| null| null|13
| min| e| b| f| b| f| a| a| c| b| b| e| ?| f| f| b| b| p| n| n| e| b| a| d|14
| max| p| x| y| y| t| y| f| w| n| y| t| r| y| y| y| y| p| y| t| p| y| y| w|15
+-------+-----+--------+----------+--------+-------+----+--------------+-----------+--------+---------+----------+---------+---------------------+---------------------+-------------------+-------------------+--------+---------+----------+--------+---------------+----------+-------+16
Create Temporary View so we can perform Spark SQL on Data
Scala
1
1
%scala2
3
mushroom.createOrReplaceTempView("MushroomData")4
Spark SQL
Scala
1
1
%sql2
3
select * from MushroomData;
Exploratory Data Analysis or EDA
Bruises Counts with Mushroom Types
SQL
1
1
%sql2
3
select count(class), 4
CASE 5
WHEN class = "e" THEN "Edible"6
ELSE "Poisonous"7
END AS CLASSES,8
bruises from MushroomData group by CLASSES, bruises; 
Mushroom Cap Color Quantity
SQL
1
16
1
%sql2
3
select count(capcolor), 4
CASE 5
WHEN capcolor = "n" THEN "Brown"6
WHEN capcolor = "b" THEN "Buff"7
WHEN capcolor = "c" THEN "Cinnamon"8
WHEN capcolor = "g" THEN "Gray"9
WHEN capcolor = "r" THEN "Green"10
WHEN capcolor = "p" THEN "Pink"11
WHEN capcolor = "u" THEN "Purple"12
WHEN capcolor = "e" THEN "Red"13
WHEN capcolor = "w" THEN "White"14
ELSE "Yellow"15
END AS ColorOfCap 16
from MushroomData group by capcolor order by count(capcolor) desc;
Edible and Poisonous Mushrooms Based on Cap Color
SQL
1
20
1
%sql2
3
select count(capcolor),4
CASE 5
WHEN class = "e" THEN "Edible"6
ELSE "Poisonous"7
END AS CLASSES,8
CASE 9
WHEN capcolor = "n" THEN "Brown"10
WHEN capcolor = "b" THEN "Buff"11
WHEN capcolor = "c" THEN "Cinnamon"12
WHEN capcolor = "g" THEN "Gray"13
WHEN capcolor = "r" THEN "Green"14
WHEN capcolor = "p" THEN "Pink"15
WHEN capcolor = "u" THEN "Purple"16
WHEN capcolor = "e" THEN "Red"17
WHEN capcolor = "w" THEN "White"18
ELSE "Yellow"19
END AS ColorOfCap 20
from MushroomData group by capcolor,class order by count(capcolor) desc;
Mushroom Odor and Quantity
SQL
1
15
1
%sql2
3
select count(odor), 4
CASE 5
WHEN odor = "a" THEN "almond"6
WHEN odor = "l" THEN "anise"7
WHEN odor = "c" THEN "creosote"8
WHEN odor = "y" THEN "fishy"9
WHEN odor = "f" THEN "foul"10
WHEN odor = "m" THEN "musty"11
WHEN odor = "n" THEN "none"12
WHEN odor = "p" THEN "pungent"13
ELSE "spicy"14
END AS odor 15
from MushroomData group by odor order by count(odor) desc; 
Edible and Poisonous Mushrooms Based on Odor
SQL
1
19
1
%sql2
3
select count(odor), 4
CASE 5
WHEN class = "e" THEN "Edible"6
ELSE "Poisonous"7
END AS CLASSES,8
CASE 9
WHEN odor = "a" THEN "almond"10
WHEN odor = "l" THEN "anise"11
WHEN odor = "c" THEN "creosote"12
WHEN odor = "y" THEN "fishy"13
WHEN odor = "f" THEN "foul"14
WHEN odor = "m" THEN "musty"15
WHEN odor = "n" THEN "none"16
WHEN odor = "p" THEN "pungent"17
ELSE "spicy"18
END AS odor 19
from MushroomData group by odor, class order by count(odor) desc; 
Mushroom Population Type Percentage
SQL
1
12
1
%sql2
3
select count(population), 4
CASE 5
WHEN population = "a" THEN "abundant"6
WHEN population = "c" THEN "clustered"7
WHEN population = "n" THEN "numerous"8
WHEN population = "s" THEN "scattered"9
WHEN population = "v" THEN "several"10
ELSE "solitary"11
END AS Population 12
from MushroomData group by Population; 
Edible & Poisonous Mushroom Population Type Percentage
SQL
1
16
1
%sql2
3
select count(population), 4
CASE 5
WHEN class = "e" THEN "Edible"6
ELSE "Poisonous"7
END AS CLASSES,8
CASE 9
WHEN population = "a" THEN "abundant"10
WHEN population = "c" THEN "clustered"11
WHEN population = "n" THEN "numerous"12
WHEN population = "s" THEN "scattered"13
WHEN population = "v" THEN "several"14
ELSE "solitary"15
END AS Population 16
from MushroomData group by Population, class; 
Mushroom Habitat Type Percentage
SQL
1
13
1
%sql2
3
select count(habitat), 4
CASE 5
WHEN habitat = "g" THEN "grasses"6
WHEN habitat = "l" THEN "leaves"7
WHEN habitat = "m" THEN "meadows"8
WHEN habitat = "p" THEN "paths"9
WHEN habitat = "u" THEN "urban"10
WHEN habitat = "w" THEN "waste"11
ELSE "wood"12
END AS Habitat13
from MushroomData group by habitat
Edible & Poisonous Mushroom Habitat Type Percentage
SQL
1
17
1
%sql2
3
select count(habitat), 4
CASE 5
WHEN class = "e" THEN "Edible"6
ELSE "Poisonous"7
END AS CLASSES,8
CASE 9
WHEN habitat = "g" THEN "grasses"10
WHEN habitat = "l" THEN "leaves"11
WHEN habitat = "m" THEN "meadows"12
WHEN habitat = "p" THEN "paths"13
WHEN habitat = "u" THEN "urban"14
WHEN habitat = "w" THEN "waste"15
ELSE "wood"16
END AS Habitat17
from MushroomData group by habitat, class
