Hadoop Archives - Projects Based Learning

How to Run Apache Druid on Docker Desktop (Windows OS) – A Step-by-Step Guide

Bigdata HadoopApache Druid, Apache Hadoop, Apache Hive, Apache Kafka, Bigdata, Hadoop

Apache Druid is a real-time analytics database designed for fast slice-and-dice analytics on large datasets. Running Druid on Docker Desktop in Windows OS enables data engineers and analysts to spin up a full Druid cluster with minimal configuration. In this blog, we'll walk through how to get Apache Druid running locally using Docker.PrerequisitesBefore starting, ensure your system meets the following requirements:Windows 10/11 with WSL 2 enabledDocker Desktop installed and runningMinimum 8GB RAM (16GB recommended for better performance)Git Bash or PowerShell for command-line executionStep 1: Clone the Apache Druid GitHub RepositoryApache Druid provides a quickstart Docker Compose setup in its GitHub…

Running Hive on Windows Using Docker Desktop: Everything You Need to Know

Bigdata HadoopApache Hadoop, Apache Hive, Bigdata, Hadoop

Apache Hive is a powerful data warehouse infrastructure built on top of Apache Hadoop, providing SQL-like querying capabilities for big data processing. Running Hive on Docker simplifies the setup process and ensures a consistent environment across different systems. This guide will walk you through setting up Apache Hive on Docker Desktop on a Windows operating system.PrerequisitesBefore you start, ensure you have the following installed on your Windows system:Docker Desktop (with WSL 2 backend enabled)At least 8GB of RAM for smooth performanceStep 1: Pull the Required Docker ImagesPull the 4.0.1 image from Hive DockerHub (Latest April 2025)docker pull apache/hive:4.0.1This image comes…

The roadmap for becoming a Data Engineer

Bigdata HadoopApache Cassandra, Apache Druid, Apache Flume, Apache Hadoop, Apache Hive, Apache Kafka, Apache Spark, Apache Spark Machine Learning, Bigdata, Hadoop

The roadmap for becoming a Data Engineer typically involves mastering various skills and technologies. Here's a step-by-step guide:Step 1: Learn the FundamentalsProgramming Languages: Start with proficiency in languages like Python, SQL, and possibly Scala or Java.Database Knowledge: Understand different database systems (SQL and NoSQL) and their use cases.Data Structures and Algorithms: Gain a solid understanding of fundamental data structures and algorithms.Mathematics and Statistics: Familiarize yourself with concepts like probability, statistics, and linear algebra.Step 2: Acquire Big Data TechnologiesApache Hadoop: Learn the Hadoop ecosystem tools like HDFS, MapReduce, Hive, and Pig for distributed data processing.Apache Spark: Master Spark for data processing,…

Installing Apache Druid on the Local Machine

Bigdata HadoopApache Druid, Apache Hadoop, Apache Hive, Apache Kafka, Bigdata, Hadoop

Apache Druid is a real-time analytics database designed for fast slice-and-dice analytics ("OLAP" queries) on large data sets. Most often, Druid powers use cases where real-time ingestion, fast query performance, and high uptime are important.Druid is commonly used as the database backend for GUIs of analytical applications, or for highly-concurrent APIs that need fast aggregations. Druid works best with event-oriented data.Common application areas for Druid include:Clickstream analytics including web and mobile analyticsNetwork telemetry analytics including network performance monitoringServer metrics storageSupply chain analytics including manufacturing metricsApplication performance metricsDigital marketing/advertising analyticsBusiness intelligence/OLAP Prerequisites You can follow these steps on a relatively modest…

Installing Single Node Kafka Cluster

Bigdata HadoopApache Hadoop, Apache Kafka, Bigdata, Hadoop, Kafka

In this tutorial, we will set up a single-node Kafka Cluster and run it using the command line.Step 1) Let’s start getting the Kafka binary, you can download the Kafka binary from the below linkhttps://kafka.apache.org/Step 2) Click on Download button Click on the binary download to get the download started Kafka is download in the Downloaded folder Moving the Kafka download to the Kafka Directory (ie /home/dataengineer/kafka) Step 3) Unzip Kafkatar -xvzf kafka_2.12-3.6.0.tgz Step 4) START THE KAFKA ENVIRONMENTNOTE: Your local environment must have Java 8+ installed.Apache Kafka can be started using ZooKeeperKafka with ZooKeeperRun the following commands in order…

Installing Apache Flume on Ubuntu

Bigdata HadoopApache Flume, Bigdata, Flume, Hadoop, Linux

System Requirements:Java Runtime Environment - Java 1.8 or laterMemory - Sufficient memory for configurations used by sources, channels or sinksDisk Space - Sufficient disk space for configurations used by channels or sinksDirectory Permissions - Read/Write permissions for directories used by agentThe first step is to create a folder Flume:Make flume directory in /home/dataengineer/mkdir flumecd flume We need to go to the website https://flume.apache.org/ and click on download. A new webpage will get open click on apache-flume-1.11.0-bin.tar.gz A new webpage will get open https://www.apache.org/dyn/closer.lua/flume/1.11.0/apache-flume-1.11.0-bin.tar.gz and copy the link shown to you. Type the below commandwget https://dlcdn.apache.org/flume/1.11.0/apache-flume-1.11.0-bin.tar.gz You will be able to…

Installing Apache Sqoop on Ubuntu

Bigdata HadoopApache Hadoop, Apache Sqoop, Bigdata, Hadoop

Step 1) Create a Sqoop directory by using the command mkdir sqoop so that we can download Apache Sqoop.Step 2) Download the stable version of Apache Sqoop (ie Apache Sqoop 1.4.7 in the year 2022) Website URL https://archive.apache.org/dist/sqoop/1.4.7/wget https://archive.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gzStep 3) Unzip the downloaded file using the tar commandtar -xvzf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gzStep 4) Edit the .bashrc file by using the commandnano .bashrcStep 5) Enter the following commands below in bashrc file and save itexport SQOOP_HOME="/home/dataengineer/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0"export PATH=$PATH:$SQOOP_HOME/bin Step 6) Execute the below command on the command prompt so bashrc gets activated.source ~/.bashrcStep 7) Check the installed sqoop version using the below commandsqoop version…

Apache Hive Installation Steps on Ubuntu

Bigdata HadoopApache Hadoop, Apache Hive, Bigdata, Hadoop, Linux

With this tutorial, we will learn the complete process to install Apache Hive 3.1.2 on Ubuntu 20.The Apache Hive data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage. A command line tool and JDBC driver are provided to connect users to Hive.Steps for Installing Hadoop on UbuntuStep 1 - Create a directory for example $mkdir /home/bigdata/apachehive Step 2 - Move to hadoop directory $cd /home/bigdata/apachehive Step 3 - Download Apache Hive (Link will change with respect to country so please get the download link from…

Apache Hadoop 3.3.1 Installation Steps on Ubuntu (Part 1)

Bigdata HadoopApache Hadoop, Bigdata, Hadoop, Linux

With this tutorial, we will learn the complete process to install Hadoop 3.3.1 on Ubuntu 20.Supported Java VersionsApache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only)Please compile Hadoop with Java 8. Compiling Hadoop with Java 11 is not supported: HADOOP-16795 - Java 11 compile support OPENApache Hadoop from 3.0.x to 3.2.x now supports only Java 8Apache Hadoop from 2.7.x to 2.10.x support both Java 7 and 8Required software for Linux include: Java must be installed. Recommended Java versions are described at HadoopJavaVersions. ssh must be installed and sshd must be running to use the Hadoop scripts that…

Apache Hadoop 3.3.1 Installation Steps on Ubuntu (Part 2)

Bigdata HadoopApache Hadoop, Bigdata, Hadoop, Linux

Use the following property in the respective filesFile: nano etc/hadoop/core-site.xml: <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration> File: nano etc/hadoop/hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> File: nano etc/hadoop/mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.application.classpath</name> <value> $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value> </property> </configuration> File: nano etc/hadoop/yarn-site.xml <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.env-whitelist</name> <value> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE, HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME*</value> </property> </configuration> Now check that you can ssh to the localhost without…