CDH 5 on VMware with Local Attached Storage

Use LBT (load-based teaming) and NIOC for QoS—the VMware recommended configuration option.

Various functional components in the vSphere cluster need guaranteed bandwidth and have different traffic patterns and loads.

This requires allocation of shares to various port groups based on role.

When using network-based storage (as with Isilon use cases), you must design appropriate port groups and configure QoS to ensure efficient service).

Disk multipathing policy uses round robin (RR). The storage array vendor might have specific recommendations. Not every vendor supports RR, so use an appropriate DMP algorithm.

This has little impact on this deployment model.

Each provisioned disk is either -

• mapped to one vSphere datastore (which in turn contains one VMDK or virtual disk) or
• mapped to one raw device mapping (RDM)

Set up virtual disks in “independent persistent” mode for optimal performance. Eager Zeroed Thick virtual disks provide the best performance.

Partition alignment at the VMFS layer depends on the storage vendor. Misaligned storage impacts performance.

Disable SIOC, and disable storage DRS.

Power Policy is an ESXi parameter. The balanced mode may be the best option. Evaluate your environment and choose accordingly. In some cases, performance might be more important than power optimization.

Avoid memory and CPU over-commitment, and use large pages for Hypervisor.

For network tuning, enable advanced features such as TSO, LRO, scatter gather, and so on

interrupt coalescing.

Minimize unnecessary virtual hardware devices. Choose the appropriate virtual hardware version; check the latest version and understand its capabilities.

VMXNET3 is supported in RHEL 6.x and CentOS 6.x with the installation of VMware tools.

• Tune the MTU size for jumbo frames at the guest level as well as ESXi and switch level.
• Enable TCP segmentation offload (TSO) at the ESXi level (should be enabled by default). This can be leveraged only by VMXNET3 drivers at the Guest layer.
• Similarly, other offload features can be leveraged only when using the VMXNET3 driver.
• Use regular platform tuning parameters, such as ring buffer size. However, RSS and RPS tuning must be specific to the VMXNET3 driver.

Use the PVSCSI storage adapter. This provides the best performance characteristics (reduced CPU utilization and increased throughput), and is optimal for I/O-intensive guests (as with Hadoop).

• Tune queue depth in the guest OS SCSI driver.
• Disk partition alignment -- Typically if VMFS is already aligned, this is not necessary (TBD for Linux).

The I/O scheduler used for the OS disks might need to be different if using VMDKS. Instead of using CFQ, use deadline or noop elevators. This varies and must be tested. Any performance gains must be quantified appropriately; for example, 1-2% improvement vs. 10-20% improvement).

Disable or minimize anonymous paging by setting vm.swappiness=0 or 1.

Consider using Virtual NUMA (vNUMA). This exposes the NUMA architecture to the guest OS so that the guest OS can be tuned to leverage NUMA during scheduling. Virtual Hardware version 8 or later is required to leverage vNUMA.

Referring to the HDFS-side HVE JIRA (https://issues.apache.org/jira/browse/HADOOP-8468), following are considerations for HVE:

Virtualized platform has following genes that shouldn't been ignored by hadoop topology in scheduling tasks, placing replica, do balancing or fetching block for reading:

1. VMs on the same physical host are affected by the same hardware failure. In order to match the reliability of a physical deployment, replication of data across two virtual machines on the same host should be avoided.

2. The network between VMs on the same physical host has higher throughput and lower latency and does not consume any physical switch bandwidth.

Thus, we propose to make hadoop network topology extendable and introduce a new level in the hierarchical topology, a node group level, which maps well onto an infrastructure that is based on a virtualized environment.

The following diagram illustrates the addition of a new layer of abstraction (in red) called NodeGroups. The NodeGroups represent the physical hypervisor on which the nodes (VMs) reside.

All VMs under the same node group run on the same physical host. With awareness of the node group layer, HVE refines the following policies for Hadoop on virtualization:

Replica Placement Policy

• No duplicated replicas are on the same node or nodes under the same node group.
• First replica is on the local node or local node group of the writer.
• Second replica is on a remote rack of the first replica.
• Third replica is on the same rack as the second replica.
• The remaining replicas are located randomly across rack and node group for minimum restriction.

Replica Choosing Policy

The HDFS client obtains a list of replicas for a specific block sorted by distance, from nearest to farthest: local node, local node group, local rack, off rack.

Balancer Policy

• At the node level, the target and source for balancing follows this sequence: local node group, local rack, off rack.
• At the block level, a replica block is not a good candidate for balancing between source and target node if another replica is on the target node or on the same node group of the target node.

HVE typically supports failure and locality topologies defined from the perspective of virtualization. However, you can use the new extensions to support other failure and locality changes, such as those relating to power supplies, arbitrary sets of physical servers, or collections of servers from the same hardware purchase cycle.

Using Cloudera Manager, configure the following in safety valves:

• HDFS
  • hdfs core-site.xml:

    <property>
        <name>net.topology.impl</name>
    <value>org.apache.hadoop.net.NetworkTopologyWithNodeGroup</value>
    </property>
    <property>
        <name>net.topology.nodegroup.aware</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.block.replicator.classname</name>
    <value>org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithNodeGroup</value>
    </property>

• In mapred-site.xml, add the following properties and values:

  <property>
      <name>mapred.jobtracker.nodegroup.aware</name>
      <value>true</value>
  </property>
  <property>
      <name>mapred.task.cache.levels </name>
          <value>3</value>
  </property>

Establish the Topology:

• Create a topology data file; for example:

  192.168.x.1 /rack1/nodegroup1
  192.168.x.2 /rack1/nodegroup1
  192.168.x.3 /rack2/nodegroup1
  192.168.x.4 /rack2/nodegroup2
  192.168.x.5 /rack3/nodegroup1
  192.168.x.6 /rack3/nodegroup2

• Create a topology script; the following is from http://ofirm.wordpress.com/2014/01/09/exploring-the-hadoop-network-topology/)

  #!/bin/bash
  HADOOP_CONF=/usr/local/hadoop-1.2.1/conf
  echo `date` input: $@ >> $HADOOP_CONF/topology.log
  while [ $# -gt 0 ] ; do
    nodeArg=$1
    exec< ${HADOOP_CONF}/topology.data
    result=""
    while read line ; do
      ar=( $line )
      if [ "${ar[0]}" = "$nodeArg" ] ; then
        result="${ar[1]}"
      fi
    done
    shift
    if [ -z "$result" ] ; then
  #        echo -n "/default/rack "
       echo -n "/rack01"
    else
      echo -n "$result "
    fi
  done

• Place the topology data file and script in the same location on each node in your cluster. Specify the location using set topology.script.file.name (in conf/hadoop-site.xml).
  • The file must be an executable script or program.
  • You can also set topology.script.file.name using Cloudera Manager.