What does it mean to virtualize the Hadoop File System? Tom Phelan Chief Architect for BlueData

It is HDFS …

Unless it is not

Outline There are questions to be answered … Three “What”’s: What is HDFS? What does it mean to virtualize HDFS? What are the different methods of virtualization? Instances Advantages and considerations And a “When”: When to choose HDFS storage virtualization?

Before we can virtualize it, we need to understand what “it” is. What is HDFS? Before we can virtualize it, we need to understand what “it” is.

It is a distributed file system built with NameNodes and DataNodes HDFS It is a distributed file system built with NameNodes and DataNodes Source: David Engfer via slidershare.net http://image.slidesharecdn.com/introtohadoop-javamug-110414122200-phpapp01/95/intro-to-the-hadoop-stack-april-2011-javamug-14-728.jpg?cb=1302793500

HDFS Implementation HDFS Implementation hadoop-hdfs.jar org.apache.hadoop.fs.FileSystem org.apache.hadoop.hdfs.FileSystem org.apache.hadoop.hdfs.DistributedFileSystem

It is a stack of Java code used by Hadoop applications to access data. HDFS Implementation It is a stack of Java code used by Hadoop applications to access data. YARN Hadoop Distributed File System API/Java Class HDFS Implementation Distributed File System Client Protocol at TCP/IP level – “over the wire” HDFS Implementation HDFS Implementation

HDFS Layers of Potential Virtualization Generic Java Classes Java class org.apache.hadoop.fs.FileSystem HDFS over the wire protocol Java class org.apache.hadoop.hdfs.DFSClient

HDFS Implementation Host Host Wire Protocol HDFS Implementation NameNode Resource Manager HDFS Implementation Host DataNode Node Manager App HDFS Impl DFSClient Local Disk

The virtualization of either the HDFS Implementation or the Protocols HDFS Virtualization The virtualization of either the HDFS Implementation or the Protocols

Outline There are questions to be answered … Three “What”’s: What is HDFS? What does it mean to virtualize HDFS? What are the different methods of virtualization? Instances Advantages and considerations And a “When”: When to choose HDFS storage virtualization?

HDFS Virtualization Methods Virtualize the HDFS Implementation Implement one of the Hadoop Compatible File System (HCFS) Protocols Implement a HCFS via the over-the-wire protocol (hdfs.DFSClient) Implement a HCFS via the FileSystem protocol (fs.FileSystem)

Virtualize the HDFS Implementation This is the only method of HDFS virtualization that requires Hadoop compute virtualization. Simple. Install a Hadoop distro into a cluster of virtualized compute nodes and run the HDFS services in the cluster storing data on vdisks/vmdks. Instances of this type of HDFS virtualization include: VMware BDE Apache OpenStack Sahara Cloudera Director Hortonworks Cloudbreak

Virtualize the HDFS Implementation HOST VM NameNode Resource Manager HOST VM HOST VM Node Manager App HDFS Impl DFSClient Node Manager App HDFS Impl DFSClient Local Disk DataNode DataNode Local Disk Local Disk Local Disk

Virtualize the HDFS Implementation Advantages: Simple No new Java code Compute/data locality Considerations: Requires data ingest time The clusters become stateful

HDFS Virtualization Methods Virtualize the HDFS Implementation Implement a Hadoop Compatible File System – HCFS Implement a HCFS via the over-the-wire protocol (hdfs.DFSClient) Implement a HCFS via the FileSystem protocol (fs.FileSystem)

Implement a HCFS via the over-the-wire protocol Use the unmodified hadoop-hdfs jar fs.defaultfs hdfs://1.2.3.4:8020/path Instance: EMC Isilon

Implement a HCFS via the over-the-wire protocol Host Storage Service Local Disk NameNode Resource Manager Host Host Node Manager App HDFS Impl DFSClient Node Manager App HDFS Impl DFSClient Local Disk DataNode DataNode Local Disk

Implement a HCFS via the over-the-wire protocol Advantages: Multi-protocol No new Java code Enterprise storage services Considerations: Open source / proprietary No compute / data locality

HDFS Virtualization Methods Virtualize the HDFS Implementation Implement a Hadoop Compatible File System – HCFS Implement a HCFS via the over-the-wire protocol (hdfs.DFSClient) Implement a HCFS via the FileSystem protocol (fs.FileSystem)

Implement a HCFS via the FileSystem Java classes Write the java code that implements the class, build a jar file, put the jar file in the YARN services class path edit the core-site.xml file Instances: S3 and S3a/S3n – org.apache.hadoop.fs.FileSystem https://github.com/Aloisius/hadoop-s3a GlusterFS - org.apache.hadoop.fs.FilterFileSystem https://github.com/gluster/glusterfs-hadoop Tachyon – org.apache.hadoop.fs.FileSystem https://github.com/amplab/tachyon Apache Ignite – org.apache.hadoop.fs.AbstractFileSystem https://github.com/apache/ignite

Implement a HCFS via the FileSystem Java classes Host NameNode Resource Manager Storage Service Host Host Node Manager Node Manager Local Disk Storage Service Storage Service DataNode App DataNode Local Disk App HDFS Impl CustomFS Impl HDFS Impl CustomFS Impl DFSClient DFSClient

Implement a HCFS via the FileSystem Java classes Host Local Disk Storage Service NameNode Resource Manager Host Host Node Manager Node Manager Local Disk Storage Service App DataNode Local Disk DataNode App Storage Service CustomFS Impl HDFS Impl HDFS Impl CustomFS Impl DFSClient DFSClient

Implement a HCFS via the FileSystem Java classes Advantages: Open source / proprietary Multiple file access protocols supported Considerations: These are file systems New Java code Possibly no compute / data locality May lag latest HDFS feature set

HDFS Virtualization Is there another way?

HDFS Virtualization Virtualize the HDFS Implementation Implement a Hadoop Compatible File System – HCFS Implement a HCFS via the over-the-wire protocol Implement a HCFS via the FileSystem Java classes Virtualize the Hadoop Compatible File System Protocol

Virtualize the Hadoop Compatible File System Protocol Translate the Hadoop File System Calls into native calls to the BackEnd File systems Insert intelligent caching layer Instance: BlueData EPIC software – org.apache.fs.FileSystem

Virtualize the Hadoop Compatible File System Protocol Host Storage Service Host NameNode Resource Manager Local Disk Local Disk Host Host Node Manager DTAP Service Node Manager DTAP Service App App Local Disk DataNode Local Disk DTAP Impl HDFS Impl DataNode HDFS Impl DTAP Impl Local Disk DFSClient Local Disk DFSClient

Application is cache aware HDFS mem cache Page Cache DataNode page HDFS Implementation DFSClient Application is cache aware

Extend mem cache to any File System or Object storage Page Cache HDFS GlusterFS Object Store page DTAP Service DTAP FileSystem Implementation Application is cache unaware

Virtualize the Hadoop Compatible File System Protocol Advantages: Not a file system Transparent in memory cache write back, read ahead Supports multiple protocols Supports compute / data locality Considerations: New Java code Open source / proprietary May lag latest HDFS feature set

Let’s Review

Outline There are questions to be answered … Three “What”’s: What is HDFS? What does it mean to virtualize HDFS? What are the different methods of virtualization? Instances Advantages and considerations And a “When”: When to choose HDFS storage virtualization?

A Few Words about Performance Performance measurements are an art as well as a science Bottlenecks in applications Bottlenecks in infrastructure network CPU disk Configuration is key block size distro security

Virtualize the HDFS Implementation Performance – VMware BDE Source of graph: VMware Technical Paper – Virtualized Hadoop Performance with VMware vSphere 6 on High Performance Servers

Implement a HCFS via the over-the-wire protocol Performance – Isilon http://stefanradtke.blogspot.com/2015/05/comparing-hadoop-performance-on-das-and.html Source of graph: Stefan Radtke blog post

Implement a HCFS via the FileSystem Java classes Performance – Tachyon Source of graph: Haoyuan Li https://spark-summit.org/2014/wp-content/uploads/2014/07/Tachyon-Further-Improve-Sparks-Performance-Haoyuan-Li.pdf

Performance – BlueData Virtualize the Hadoop Compatible File System Protocol Performance – BlueData Source of Graph: BlueData customer proof-of-concept results

Virtualized HDFS solutions provide good performance Even with remote storage Even in virtualized environments

When it comes to Hadoop storage virtualization, speed is not the whole story Other factors to consider when implementing a virtualized HDFS option: Use of a virtualized compute environment Open source / proprietary solution Required Hadoop File System features Lifespan of Hadoop cluster

When it comes to Hadoop storage virtualization, speed is not the whole story Other factors to consider when selecting storage: Data accessibility Hadoop File System protocol NFS, object store, other protocols Enterprise storage services data protection geographical replication offline backup

Consider a Virtualized HDFS Solution When any of the following are true: Hadoop and non-Hadoop applications are required to access the same data Do not want to replicate the data Enterprise storage data services required Need to run Hadoop in a virtual compute environment

Volume, Velocity, Variety Hadoop File System Volume, Velocity, Variety Virtualization

Q & A twitter: @tapbluedata email: tap@bluedata.com www.bluedata.com Visit our booth in the Expo