Hadoop Archives - Projects Based Learning

Installing Apache Flume on Ubuntu

Bigdata HadoopApache Flume, Bigdata, Flume, Hadoop, Linux

System Requirements:Java Runtime Environment - Java 1.8 or laterMemory - Sufficient memory for configurations used by sources, channels or sinksDisk Space - Sufficient disk space for configurations used by channels or sinksDirectory Permissions - Read/Write permissions for directories used by agentThe first step is to create a folder Flume:Make flume directory in /home/dataengineer/mkdir flumecd flume We need to go to the website https://flume.apache.org/ and click on download. A new webpage will get open click on apache-flume-1.11.0-bin.tar.gz A new webpage will get open https://www.apache.org/dyn/closer.lua/flume/1.11.0/apache-flume-1.11.0-bin.tar.gz and copy the link shown to you. Type the below commandwget https://dlcdn.apache.org/flume/1.11.0/apache-flume-1.11.0-bin.tar.gz You will be able to…

Installing Apache Sqoop on Ubuntu

Bigdata HadoopApache Hadoop, Apache Sqoop, Bigdata, Hadoop

Step 1) Create a Sqoop directory by using the command mkdir sqoop so that we can download Apache Sqoop.Step 2) Download the stable version of Apache Sqoop (ie Apache Sqoop 1.4.7 in the year 2022) Website URL https://archive.apache.org/dist/sqoop/1.4.7/wget https://archive.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gzStep 3) Unzip the downloaded file using the tar commandtar -xvzf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gzStep 4) Edit the .bashrc file by using the commandnano .bashrcStep 5) Enter the following commands below in bashrc file and save itexport SQOOP_HOME="/home/dataengineer/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0"export PATH=$PATH:$SQOOP_HOME/bin Step 6) Execute the below command on the command prompt so bashrc gets activated.source ~/.bashrcStep 7) Check the installed sqoop version using the below commandsqoop version…

Apache Hive Installation Steps on Ubuntu

Bigdata HadoopApache Hadoop, Apache Hive, Bigdata, Hadoop, Linux

With this tutorial, we will learn the complete process to install Apache Hive 3.1.2 on Ubuntu 20.The Apache Hive data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage. A command line tool and JDBC driver are provided to connect users to Hive.Steps for Installing Hadoop on UbuntuStep 1 - Create a directory for example $mkdir /home/bigdata/apachehive Step 2 - Move to hadoop directory $cd /home/bigdata/apachehive Step 3 - Download Apache Hive (Link will change with respect to country so please get the download link from…

Apache Hadoop 3.3.1 Installation Steps on Ubuntu (Part 1)

Bigdata HadoopApache Hadoop, Bigdata, Hadoop, Linux

With this tutorial, we will learn the complete process to install Hadoop 3.3.1 on Ubuntu 20.Supported Java VersionsApache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only)Please compile Hadoop with Java 8. Compiling Hadoop with Java 11 is not supported: HADOOP-16795 - Java 11 compile support OPENApache Hadoop from 3.0.x to 3.2.x now supports only Java 8Apache Hadoop from 2.7.x to 2.10.x support both Java 7 and 8Required software for Linux include: Java must be installed. Recommended Java versions are described at HadoopJavaVersions. ssh must be installed and sshd must be running to use the Hadoop scripts that…

Apache Hadoop 3.3.1 Installation Steps on Ubuntu (Part 2)

Bigdata HadoopApache Hadoop, Bigdata, Hadoop, Linux

Use the following property in the respective filesFile: nano etc/hadoop/core-site.xml: <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration> File: nano etc/hadoop/hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> File: nano etc/hadoop/mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.application.classpath</name> <value> $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value> </property> </configuration> File: nano etc/hadoop/yarn-site.xml <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.env-whitelist</name> <value> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE, HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME*</value> </property> </configuration> Now check that you can ssh to the localhost without…

Top 1000+ Big Data Interview Question and Answers

Bigdata HadoopApache Hadoop, Apache Hive, Apache Pig, Apache Spark, Apache Sqoop, Bigdata, Hadoop, MapReduce

With more companies turning to big data to run their business, the demand for talent is at an all-time high. What does that mean for you? It just translates to better opportunities if you want to get employed in any of the big data-related fields. In the era of big data, companies are turning more and more towards using big data to operate their operations. It means better prospects for employment in any big data-related organization. There is a huge demand for talent in the big data era, with more and more companies utilizing big data to run their operations.…

Apache Hadoop 3.3.0 Single Node Installation on Windows 10 Part 1

Bigdata HadoopApache Hadoop, Bigdata, Hadoop

Download Link for Apache Hadoop 3.3.0 URL : https://hadoop.apache.org/releases.html Click on the Binary it will open a new website https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz (This link may change based on your location) Download link for Java SE Development Kit 8 https://www.oracle.com/in/java/technologies/javase/javase-jdk8-downloads.html Register or Login If you have already registered the Download will begin We will have Below files in Download Folder Installing and Configuring Java Step 1: Create a Empty Folder Java in C Drive Step 2: Go to the Download location Step 3: Double Click on the Setup file Click on Next Click on Next Click on Next Click on Change Make Sure you change…

Apache Hadoop 3.3.0 Single Node Installation on Windows 10 Part 2

Bigdata HadoopApache Hadoop, Bigdata, Hadoop

We have downloaded hadoop installation files We need to move (that is cut and paste) From: Downloads Location To: C:\hadoop-3.3.0.tar In C Drive Extract the hadoop-3.3.0.tar files in C Drive using extraction software (WinZip, WinRar or 7Zip) Now we will have the following in C Drive Now Open Folder C:\hadoop-3.3.0\etc\hadoop We need to edit 5 files File C:/Hadoop-3.3.0/etc/hadoop/core-site.xml, paste below xml paragraph and save this file. <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration> C:/Hadoop-3.3.0/etc/hadoop/mapred-site.xml, paste below xml paragraph and save this file. <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> Create folder "data" under "C:\Hadoop-3.3.0" 1) Create folder "datanode" under "C:\Hadoop-3.3.0\data" 2) Create…

Tags