1. © Hortonworks Inc. 2011 MapReduce over snapshots HBASE-8369 Enis Soztutar Enis [at] apache [dot] org @enissoz Page 1

2. © Hortonworks Inc. 2011 About Me Page 2 Architecting the Future of Big Data In the Hadoop space since 2007 Committer and PMC Member in Apache HBase and Hadoop Working at Hortonworks as member of Technical Staff Twitter: @enissoz

3. © Hortonworks Inc. 2011 Snapshots Currently a snapshot is a bunch of reference files together with some metadata A table snapshot can contain –Table descriptor –List of regions –References to files in the regions –References to WALs for regionservers Current snapshot impl is flush based –Forces flush to all regions, so that in-memory data is written to disk Page 3 Architecting the Future of Big Data

4. © Hortonworks Inc. 2011 MR over Snapshots Idea is do scans on the client side bypassing region servers Use snapshots since they are immutable Similar to short circuit hdfs reads TableSnapshotInputFormat works similar to TableInputFormat TableMapReduceUtil methods to configure the job Page 4 Architecting the Future of Big Data

5. © Hortonworks Inc. 2011 Deployment Options HBase online Take snaphot while HBase is running Run MR job over the snapshot HBase offline Take snapshot while HBase is running Export Snapshot using ExportSnapshot to a different hdfs Run MR job over snapshot with or without HBase running Page 5 Architecting the Future of Big Data

6. © Hortonworks Inc. 2011 TableSnapshotInputFormat Gets a Scan representing the query Restore the snapshot to a temporary directory For each region in the snapshot: –Determine whether the region should be scanned (falls between scan start row and stop row) –Create one split per region in the scan range ( # of map tasks) –Each RecordReader will open the region (Hregion) as in HRegionServer –An internal RegionScanner is used for running the scan Page 6 Architecting the Future of Big Data

7. © Hortonworks Inc. 2011 API Page 7 Architecting the Future of Big Data

8. © Hortonworks Inc. 2011 Timeline Will (hopefully) be committed to trunk next week or so Interest in bringing this to 0.94 and 0.96 bases as well Will come in HDP-2.1, which will be based on 0.96 line Page 8 Architecting the Future of Big Data

9. © Hortonworks Inc. 2011 Security Aspects HBase user owns the files in filesystem Snapshot files are also owned by the HBase user Mapreduce job should be able to read the files in the snapshot + actual data files HDFS only has posix-like perms based on user/group/other –User running MR job has to be either the HBase user, or have group perms –HDFS does not have ACLs, so there is no easy way to grant read access at filesystem layer Idea: similar to current short circuit impl, we can implement a FD transfer –User will submit jobs under her own user credentials –Ask HBase daemons to open the files, and pass a handler / token Page 9 Architecting the Future of Big Data

10. © Hortonworks Inc. 2011 Performance ScanTest: Scan : open a scanner, do full table scan SnapshotScan : open a client-side scanner, do full table scan ScanMR : parallel full table scan from MR SnapshotScanMR : do full table scan 8 Region servers, 6 disks each HBase trunk Hadoop-2.2 (HDP-2.0.7.0-12) Load data with IntegrationTestBulkLoad –Evenly distributed rows, created as bulk loaded hfiles. 3 column families # store files per region varies 3,6,9, and 12 (1,2,3,4 file per store) Data sizes: 6.6G, 13.2G, 19.8G, 26.4G Page 10 Architecting the Future of Big Data

11. © Hortonworks Inc. 2011 Scan speed Page 11 Architecting the Future of Big Data

12. © Hortonworks Inc. 2011 API We do not want to limit snapshot scanning only to MapReduce Allow client side scanners over snapshot files Page 12 Architecting the Future of Big Data

13. © Hortonworks Inc. 2011 ResultScanner is main scan API Page 13 Architecting the Future of Big Data

14. © Hortonworks Inc. 2011 API (caution: not final yet) Page 14 Architecting the Future of Big Data

15. © Hortonworks Inc. 2011 To the future and beyond HBASE-8691 High-Throughput Streaming Scan API Can we bypass regionservers without taking snapshots? Bypass memstore data, or stream memstore data, but read directly from hfiles Secure reading from snapshots Keep up with the updates at –https://issues.apache.org/jira/browse/HBASE-8369 Page 15 Architecting the Future of Big Data

16. © Hortonworks Inc. 2011 Thanks Questions? Architecting the Future of Big Data Page 16 Enis Söztutar enis [ at ] apache [dot] org @enissoz