Storage: Configure Disks, Partitions and Disk Mount to fit with Hadoop

Disk configuration

Namenode

RAID1 is recommended for the namenode’s disks, to protect against corruption of its metadata.

DataNodes

• JBOD(No RAID) is recommended.
• One disk for the operation system specifically, others for Hadoop data storage.
• Partition looks like following. 
  
• Partition the remaining Data Storage disks.
  1. Create Partition

     fdisk /dev/sdb
     

  2. Press n to create a new partition.
  3. One partition table for each disk. Using the default in the remaining options.
  4. Press w to write table to disk.
  5. Format the disk.

     mkfs -t ext4 /dev/sdb1
     

  6. Repeat the above steps for other Data Storage Disk.
• Mounting disk with noatime and nodiratime. As Hadoop (HDFS) manages the metadata (inode) of its filesystem with NameNode, any access time information kept by Hadoop is independent of the atimeattribute of individual blocks. So, the access timestamps in DataNode's local filesystem makes no sense here.
  1. Create the mount points

     mkdir -p /mnt/d0/data
     mkdir -p /mnt/d1/data
     

  2. Appending the following rules to the file /etc/fstab.

     /dev/sdb1 /mnt/d0/data ext4 defaults,noatime,nodiratime 0 0
     /dev/sdc1 /mnt/d1/data ext4 defaults,noatime,nodiratime 0 0
     

  3. Mounting the disk.

     mount /dev/sdb1
     mount /dev/sdc1
     

• Reduce the reserved blocks on the disks

  tune2fs -m 1 /dev/sdb1
  tune2fs -m 1 /dev/sdc1
  

  Note: Do not reduce the reserved blocks on the disks hosting the operating system.

Configure hdfs-site.xml

Note: assume the ssh key has been distributed to the Datanodes

• Creating folders to presist HDFS data

  mkdir /mnt/d0/data/dfs
  mkdir /mnt/d1/data/dfs
  

• Changing the owner to hdfs

  chown hdfs:hdfs -R /mnt/d0/data/dfs
  chown hdfs:hdfs -R /mnt/d1/data/dfs
  

• Add the following property

  hadoop@master1$ vi $HADOOP_HOME/conf/hdfs-site.xml
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/mnt/d0/data/dfs,/mnt/d1/data/dfs,...,/mnt/dn/data/dfs</value>
  </property>
  

• Redo the previous step on all the Datanodes.
• Restarting HDFS, on Namenode host

  hdfs@namenode$ $HADOOP_HOME/bin/stop-dfs.sh
  hdfs@namenode$ $HADOOP_HOME/bin/start-dfs.sh
  

  Note: To do this step, it required the ssh key has been distributed across the Clusters.

Conigure mapred-site.xml

• Creating the folder for MapReuce to cache intermediate data.

  mkdir /mnt/d0/data/mapred
  mkdir /mnt/d1/data/mapred
  

• Changing the owner to hdfs

  chown mapred:hadoop -R /mnt/d0/data/mapred
  chown mapred:hadoop -R /mnt/d1/data/mapred
  

• Add the following property

  hadoop@master1$ vi $HADOOP_HOME/conf/hdfs-site.xml
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/mnt/d0/data/mapred,/mnt/d1/data/mapred,...,/mnt/dn/data/mapred</value>
  </property>
  

• Redo the previous step on all the Datanodes.
• MapReduce restart.