KSU Hadoop on Pi (SmallData)

Hadoop 2.7.3 Raspberry Pi 3 Cluster 4 Nodes

As a Professor and Infrastructure Engineer, I kept hearing the term BigData and Hadoop around campus. In fact, we ordered 4 Dell C4130 Servers for our setup. I didn’t have an extra $100,000 laying around, so the next best thing was the Raspberry Pi. I wanted to be able to learn Hadoop in my own environment and teach others how to do the same.

This tutorial was written as detailed as reasonably possible. I did not create the original tutorial, however, it is a combination of data gathered and recompiled for my own environment. Basic Linux and networking skills are needed. If you find an error, please let me know. It took three attempts to get it just right and document at the same time.

In this tutorial, we will install and configure a Hadoop cluster using Raspberry Pi’s. Our cluster will consist of four nodes (one master and three slaves). Since Raspberries have a low price and Hadoop is open source, together they offer a great resource to learn Hadoop.

Items in blue are code to be copied. Items in red are to be changed for your environment. Items in purple are for slave nodes only.

Materials

4x Raspberry PI 3 Model (4 Core 1Ghz CPU, 1GB RAM)
4x 16gb MicroSDHC cards (Sandisk UHS-I 10x)
4x micro USB to USB for power
4x CAT5e network cables (3feet if possible)
1x Powered USB Hub
1x 5 or more port switch
1x Dogbone case for 4x Raspberry Pi’s (Any case will do)
Rasbpian Jessie Lite (Linux 4.1.17-v7+)
Oracle Java java version 1.8.0_65
Hadoop 2.7.2 (ARM native compiled from source)

For my setup, I ordered the dogbone case, usb hub, microUSB to USB cables, and ethernet cables from Amazon. Since Microcenter is local, I drove over and purchased 4 Raspberry Pi 3’s for $29 each and 4 16gb MicroSDHC cards for $7.99 each. I already owned a 24 port non-managed switch and a 48 port Cisco switch. I also had a spare monitor, keyboard, and HDMI to DVI cable. Once I setup the Raspberry Pi’s using the monitor, I would ssh into each one from a remote computer. The total cost of this project was under $200.

Install Raspbian

Download Raspbian Jessie Lite: https://www.raspberrypi.org/downloads/raspbian/

Write to SD card (use any tool of choice) for windows I use: https://sourceforge.net/projects/win32diskimager/

Plugin in SD card and fire up your PI.

For initial configuration (raspi-config)

Expand filesystem
Memory Split Choose 16MB to give as much RAM as possible for Hadoop
Change the hostname (rpi0)
Turn on SSH
Change the root password

Reboot

Basic Pre-installation Configuration

Change Password, Update, and Install Java, rsync, and ant

passwd

sudo apt-get update && sudo apt-get install oracle-java8-jdk && sudo apt-get install rsync && sudo apt-get install ant

Verify that it was installed correctly:

java –version

Run update-alternatives, ensure jdk-8-oracle-*** is selected if you have Java 7

sudo update-alternatives --config java

Setup Connectivity

sudo nano /etc/hosts

10.0.0.100 rpi0
10.0.0.101 rpi1
10.0.0.102 rpi2
10.0.0.103 rpi3

sudo nano /etc/dhcpcd.conf

At the bottom of the file add:

static ip_address=interface eth0
static ip_address=10.0.0.100/24
# static ip_address=10.0.0.101/24
# static ip_address=10.0.0.102/24
# static ip_address=10.0.0.103/24
static routers=10.0.0.1
static domain_name_servers=10.0.0.1

Restart & SSH to new IP

sudo shutdown -r

Login locally or remote

ssh pi@10.0.0.100

Configure Hadoop User

sudo addgroup hadoop

sudo adduser --ingroup hadoop hduser

sudo adduser hduser sudo

logout

Log in as the newly created hduser. From hence forth everything will be executed as hduser. Create SSH pairs keys with blank password. This will enable nodes to communicate with each other in the cluster.

mkdir ~/.ssh

ssh-keygen -t rsa -P ""

cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys

SSH to your node and answer yes when prompted to trust certificate key – otherwise Hadoop will fail to login later.

ssh rpi0

exit

Install Hadoop 2.7.3 for Raspberry PI (ARM)

Ensure you have logged out as hduser and logged in as pi user. (for sudo command below to work properly, as we did above)

Install Hadoop

cd ~/

sudo mkdir /opt (directory may already exist)

cd /opt/span>

sudo wget http://www-us.apache.org/dist/hadoop/common/stable/hadoop-2.7.3.tar.gz

sudo tar xvzf hadoop-2.7.3.tar.gz

sudo mv hadoop-2.7.3 hadoop

sudo chown -R hduser:hadoop hadoop

Give access to hduser

sudo chown -R hduser:hadoop /opt/hadoop/

Add environment variables

sudo nano ~/.bashrc

Add the following to the bottom of the file

# — HADOOP ENVIRONMENT VARIABLES START — #
export JAVA_HOME=$(readlink -f /usr/bin/java | sed “s:bin/java::”)
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS=”-XX:-PrintWarnings -Djava.library.path=$HADOOP_HOME/lib”
# — HADOOP ENVIRONMENT VARIABLES END — #

Check to see if the Hadoop variable has been set

source ~/.bashrc

hadoop version

Setting up the Hadoop Daemon configurations

Add environment variable to hadoop-env.sh

cd /opt/hadoop/etc/hadoop

sudo nano hadoop-env.sh

Add this:

export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
export HADOOP_HEAPSIZE=250

Edit Config Files (Only change what’s in red to match your hostname)

cd opt/hadoop/etc/hadoop

sudo nano core-site.xml

<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hdfs/tmp</value>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://rpi0:9000/</value>
</property>

</configuration>

sudo nano hdfs-site.xml

<property>
<name>dfs.replication</name>
<value>3</value>
</property>

<property>
<name>dfs.webhdfs.enable</name>
<value>true</value>
</property>

<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>

</configuration>

sudo nano yarn-site.xml

<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>rpi0:8025</value>
</property>

<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>rpi0:8035</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>rpi0:8050</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>4</value>
</property>

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>768</value>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>64</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>256</value>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>4</value>
</property>

<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
</property>

</configuration>

sudo nano mapred-site.xml.template

Save As: mapred-site.xml

<property>
<name>mapredreduce.job.tracker</name>
<value>rpi00:9001</value>
</property>

<property>
<name>mapred.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapreduce.map.memory.mb</name>
<value>256</value>
</property>

<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx204m</value>
</property>

<property>
<name>mapreduce.map.cpu.vcores</name>
<value>2</value>
</property>

<property>
<name>mapreduce.reduce.memory.mb</name>
<value>128</value>
</property>

<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx102m</value>
</property>

<property>
<name>mapreduce.reduce.cpu.vcores</name>
<value>2</value>
</property>

<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>128</value>
</property>

<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx102m</value>
</property>

<property>
<name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
<value>1</value>
</property>

<property>
<name>mapreduce.job.maps</name>
<value>4</value>
</property>

<property>
<name>mapreduce.job.reduces</name>
<value>4</value>
</property>

</configuration>

Hadoop Data File System (HDFS)

As part of the Hadoop installation HDFS is automatically added. A tmp folder needs to be created within HDFS to keep the temporary test data.

sudo mkdir /opt/hdfs/tmp

sudo chown hduser:hadoop /opt/hdfs/tmp

sudo chmod 750 /opt/hdfs/tmp

Add the slaves file

sudo nano /opt/hadoop/etc/hadoop/slaves

add :

rpi1
rpi2
rpi3

Copy SD Card

Use Win32imager or any other software of your choice to clone the sd-card. After clone ensure to configure each node with new hostname and ip-address.

Update Datanodes one at a time

Leave rpi0 off until all datanodes are setup

Plugin in SD card in (rpi1, then rpi2, then rpi3) and fire up your Pi’s.

sudo raspi-config

Change the hostname (rpi1, rpi2, rpi3)

Change the IP Address

sudo nano /etc/dhcpcd.conf

Comment out the IP for rpi0 and uncomment the IP for rpi1. Repeat on rpi2, rpi3

interface eth0
#static ip_address=10.0.0.100/24
static ip_address=10.0.0.101/24
# static ip_address=10.0.0.102/24
# static ip_address=10.0.0.103/24
static routers=10.0.0.1
static domain_name_servers=10.0.0.1

Reboot – This concludes the setup of the slaves

Add the master file to the master rpi0 only

sudo nano /opt/hadoop/etc/hadoop/masters

add :

rpi0

Logon with ssh to all nodes from namenode (rpi1). Enter Yes when you get the “Host key verification failed message”. This is important in order for namenode to be able to communicate with the other nodes without entering a password.

Format the namenode (rpi0) and start the services

hdfs namenode -format

That will format the master as a proper namenode. Finally, start the services. Previously, Hadoop used the start-all.sh script for this, but it has been deprecated and the recommended method now is to use the start-<service>.sh scripts individually. In each node, go to /opt/hadoop/etc/hadoop/ and execute the following:

start-dfs.sh

start-yarn.sh

Verify the cluster

There are several ways to confirm that everything is running properly, for example, you can point your browser to http://rpi0:50070 (the http address configured earlier) or to http://rpi0:8088. There you can see status reports of the cluster.

Alternatively, you can also check the Java processes in the nodes. In the master node you should see at least 3 processes: NameNode, SecondaryNameNode and ResourceManager. In the slaves you should see only two: NodeManager and DataNode. My favorite way is to use the hdfs report command. This is the output report of my configuration:

hdfs dfsadmin -report

Configured Capacity: 182240477184 (169.72 GB)
Present Capacity: 153073315840 (142.56 GB)
DFS Remaining: 153073020928 (142.56 GB)
DFS Used: 294912 (288 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

————————————————-

Live datanodes (4):

These strings copy the basic configuration to from the master to the slave nodes. You only need to do this if you make changes on the master to any of the xml files and want to replicate those changes to the slave nodes.

sudo rsync -avxP --rsync-path='/usr/bin/sudo /usr/bin/rsync' /opt/hadoop/etc/hadoop pi@rasp1:/opt/hadoop/etc/hadoop

sudo rsync -avxP --rsync-path='/usr/bin/sudo /usr/bin/rsync' /opt/hadoop/etc/hadoop pi@rasp1:/opt/hadoop/etc/hadoop

sudo rsync -avxP --rsync-path='/usr/bin/sudo /usr/bin/rsync' /opt/hadoop/etc/hadoop pi@rasp1:/opt/hadoop/etc/hadoop

References

Verduco, A. (2016, March 23). Building a Hadoop cluster with Raspberry Pi – developerWorks Recipes. Retrieved April 26, 2017, from https://developer.ibm.com/recipes/tutorials/building-a-hadoop-cluster-with-raspberry-pi/#r_overview

Widriksson, J. (2016). Raspberry PI 2 Hadoop 2 Cluster – Jonas Widriksson. Retrieved April 26, 2017, from http://www.widriksson.com/raspberry-pi-2-hadoop-2-cluster/

Cybersecurity & Information Technology Professor

Hadoop 2.7.3 Raspberry Pi 3 Cluster 4 Nodes

Leave a Reply

Cybersecurity & Information Technology Professor

KSU Hadoop on Pi (SmallData)

Hadoop 2.7.3 Raspberry Pi 3 Cluster 4 Nodes

Related articles

KSU Kali on AWS

KSU Theoretical Project

KSU Parrot Security OS

Leave a Reply