Hadoop IT Services Hadoop Users Forum CERN October 7 th,2015 CERN IT-D*

Hadoop A framework for large scale data processing Distributed storage and processing Shared nothing architecture – scales horizontally Optimized for high throughput on sequential data access 2 Interconnect network MEMORY CPU Disks MEMORY CPU Disks MEMORY CPU Disks MEMORY CPU Disks MEMORY CPU Disks MEMORY CPU Disks Node 1Node 2Node 3Node 4 Node 5 Node X

How Hadoop Can Help You Parallel processing of large amounts of data Perform analytics on a big scale Dealing with diverse data: structured, semi- structured, unstructured ‘Cold’ storage / Archives Performance is usually suboptimal for Random reads and real-time access ‘Small’ datasets 3

There are already interesting use cases of WLCG grid monitoring Data Transfers etc. Atlas Events Indexing CASTOR log aggregation Data Warehousing Logging/time series data IT monitoring 4

Hadoop Service in IT Setup and run the infrastructure Provide consultancy Build the community Joint work IT-DB and IT-DSS 5

Hadoop Clusters in IT (Oct 2015) lxhadoop (22 nodes) general purpose cluster (mainly used by ATLAS) stable software setup recent hardware analytix (56 nodes) for analysis of monitoring data varied hardware specifications the biggest in terms of number of nodes hadalytic (17 nodes) general purpose cluster with additional services recent hardware 6

Many Configuration Options Hadoop is a platform Many components and key decisions in the implementation Rapidly evolving field Examples Data access: domain specific language or SQL Many components and data formats Data loading and unloading tools 7

Currently available components 8 HDFS Hadoop Distributed File System Hbase NoSql columnar store YARN Cluster resource manager MapReduce Hive SQL Pig Scripting Flume Log data collector Sqoop Data exchange with RDBMS Zookeeper Coordination Impala SQL Spark Large scale data proceesing

Software version policy Align to CDH distributions 9 lxhadoop (22 nodes) analytix (56 nodes) hadalytic (17 nodes) CDH HDFS HBase Hive Pig Spark Impala Sqoop

Maintenance activities Actions Upgrades to a newer CDH Frequency Typically twice a year Impact Downtime 1-3 hours 10

Recent activities (last 3 months) Hadoop Tutorials – during summer Deployment of Coudera Impala component Monitoring of hanging HBase region servers Self-service Oracle2Hadoop integration (work in progress) Building a database of users’ data sources 11

Contact points Service is available in SNOW SE: Hadoop Service FE: Hadoop Components FE: Hadoop Core E-group: Show up on the Wednesday’s meeting Analytic Working Group Hadoop User Forum 12

How to Learn More Hadoop tutorials at CERN, summer 2015 Introduction to Hadoop (Architecture, HDFS, MapReduce, Spark) SQL on Hadoop (Hive, Impala) NoSQL on Hadoop (HBase) We plan to do more/repeats in the future 13

Future plans Infrastructure HDFS backups Rolling upgrades Support from Cloudera? Users community Write a Knowledge Base (SNOW) New features/technology testing Kudu – a new columnar file system from Cloudera Tachyon – in-memory file system 14