Providing Elasticity To DataNode Storage of Hadoop Using LVM

Hey Everyone!!!

Here we are increasing the storage of datanodes to increase the size of cluster and for that we don’t have to shut or reboot any of nodes.

we are using Logical volume Management (LVM) in each node to increase the size individually and it will automatically increase the size of cluster on fly

Before going further lets get overview of LVM

LVM

Logical Volume Management, is a storage device management technology that gives users the power to pool and abstract the physical layout of component storage devices for easier and flexible administration.

The Physical Volume and Volume group are just one of the processes that must be followed to create the end result, which is the Logical Volume.

Physical Volume: Physical volumes are regular storage devices. LVM writes a header to the device to allocate it for management

volume Group: LVM combines physical volumes into storage pools known as volume groups.

Logical Volume:A volume group can be sliced up into any number of logical volumes. Logical volumes are functionally equivalent to partitions on a physical disk, but with much more flexibility

Pre-requisite::

• Hadoop master-data nodes should be pre-configured

Our hadoop setup is having a Master node and 4 live Datanodes is connected.

The IP Address of Datanode on which we are configuring is 13.233.62.71

To List all the storage device attach to datanode cmd is :

fdisk -l

This disk does not have any partition, to Creates partition please refer this blog, here i have explained in detail.

Way To Increase or Decrease The Size Of Static Partition in Linux

As we are set with our partition, now to create Logical volume follow some steps

1. Create Physical Volume

pvcreate <partition_name>

NOTE: to list all the physical volume command is :

pvdisplay

2. Create volume Group

vgcreate <name_of_vg> <pv’s>

to see the list of volume group :

vgdisplay

3 Create logical volume

lvcreate — size <Enter_size_of _Lv> — name <LV_name> <vg_name>

To see the info of newly created LV :

lvdisplay vg_name/lv_name

4. Format the newly created Logical Volume

mkfs.ext4 <path_of_LV>

5. mount the Logical volume

mount <path_of_LV> <folder>

NOTE: Always mount the the LV to the Folder which is contributed to the master node

To check the mount point cmd is :

df -h

The datanode having IP Address (13.233.62.71) is sharing 476MiB of storage which is a Logical volume

This is the basic LVM is Setup in this Datanode

Now here is the fun part …….

whenever datanode wants to share node Storage to the Master Node the only thing Datanode have to do And that is extend the size of Logical Volume, AS the lv increases the size of cluster automatically.

Lets check it it works or not , for that we are going to extend the size of LV And gonna check ,Is Size of cluster increases on fly??

6. Extend the Logical Volume

lvextend — size +<Enter_Size> <Path_of_Lv>

here we have incresed our LV to 1GiB

7. Format the increased partition

resize2fs <path_of_Lv>

Now Lets check the cluster Info

Conclusion : we can see the before we extended the LV the configured Storage is just 477MiB and as we Extended LV in DataNode the size of cofigured storage is also increases

— — — — — — — - — — — — !!! Thank you !!! — — — — — — — — — — — —

This article comes under ARTH — the School of Technologies #Task 7.1 Special Thanks to @vimal daga sir