ETRI Site Introduction 2009. 6. 8 Han Namgoong, nghan@etri.re.kr

ETRI Government sponsored Research Institute  3,000 staffs, 500M USD (year 2009)  focus on technologies of broadcasting, software and contents, IT convergence, and convergence components and materials

ETRI Cluster Topology (1/2)

ETRI Cluster Topology (2/2) Key Masters Server Pool +Agent +10,000Nodes Monitoring +Provisioning Proxy +DHCP +Agents +256 nodes +Provisioning Server +LVS +DB +40 Group Masters Cluster Master Database File System Group +Global Service Dispatcher +Disaster Recovery +100 Data Centers Distributed Procesing

Video based Internet Application Services ETRI Cluster Software Stack and Services (2009.6.8) Video based Internet Application Services UGC Search Service IPTV Service e-Learning Service Internet Services Common Components Video Management Components Security Cluster Management Production Tagging Store Retrieval Delivery Large Scale Parallel Processing Large Scale Data Mgmt. User/ service authen. Cluster Orchestration Service Data Management Job Partition and Merge Distributed Data Store Device/ kernel authen. Distributed Job Scheduling Provisioning Data Access and Recovery Global File System Service Mgmt. File Metadata Management File Store And Replication File Remote Backup/Archiving Resources Monitoring Platform OS and HW Low Power OS Node Manager Low Power HW

Monitoring Tool for Large Cluster System Research Topics (2009.6.8) Monitoring Tool for Large Cluster System - current monitoring SW  heavy overhead cpu/memory  small/light monitoring tool 2. Management of Big Video Feature Data - Google YouTube(2006) * Upload : 70,000 per day, Viewing : 100 Million play per day - Keyword based Retrieval (vague, imprecise,..) - Content based Retrieval (not simple interface/slow result)  Integrated Query(Keyword + Content based) 3. Elimination of Duplicated Video Data - Lots of same video files occupied storage spaces.  File (NOT data) deduplication is strongly required.

Cloud stack (OSS) for evaluation Schedule (2009.6.8) Phase 1 : 2009.9.1 ~ 2010.2.28 Cloud stack (OSS) for evaluation - System management/Monitoring tool - Middleware(Web/AP/DB server) - Linux(CentOS,..) - Virtualization(Xen, KVM) - Distributed file-system/DB (Hadoop, Hbase) - Authentication(OpenLDAP) Evaluation point - Error recovery procedure, configuration, structure - Add resource(planned, unexpected) - Remove resource by degrade of load, and Migration - Overhead of virtualization, distributed file-system, distributed DB - Authentication between systems Source : Tomomi Suzuki, Status report of Cloud Computing activity, Japan OSS Promotion Forum, 2009.6.4

Schedule (2009.6.8) 2. Phase 2 : 2010.3.1 ~ 2011.2.28 Selection of Requirements Develop , Test and Deployment - Monitoring Tool for Large Cluster System - Management of Big Video Feature Data - Elimination of Duplicated Video Data Distributed Processing - Fail-over of task execution node and job manage node - Distributed task processing based on data location - Configurable job scheduling : 9 policies …….. Distributed Data Management based on Hadoop/Hbase - Multi dimensional map model - Support a composite row key - Column group based storage model - Distribute partitions splited by a composite row key - Data access control by user and privilege management ………

Parallel Processing Model Plans, Expectations (1/3) Category Hadoop/MapReduce What Expectations Parallel Processing Model - Map/Reduce Programming Model - I/O Source : HDFS, LFS, Hbase - Map/Reduce Programming Model I/O Source : + new-FS, new-DB Enlargement of parallel processing target Cluster Size - Thousand nodes - Manually configure Thousand nodes Automatically configure Easy to manage parallel processing cluster Job Control - None - Execution control based on user Access control to parallel processing cluster Job Scheduling - Direct Priority, FIFO - Priority management by job - 9 configurable scheduling policies - Priority management by job, Group and user Support of various jobs Task Distribution - Consideration of data location and node position - Consideration of data location, node position and node resource Increase of node utilization High Availability - Fail-over of task execution node Fail-over of task execution node Fail-over of job manage node - Increase availability Reduction of Job execution time

Plans, Expectations (2/3) Category Hbase What Expectations Data Model - Multi dimensional map - Row key : single field Multi dimensional map Row key : composite field Easy to construct key Video Manage - None High dimensional index manage k-NN search Provide large scale video content based retrieval Data Storage Model Column oriented Per column Column oriented Per column group Performance enhancement Data Distribution Distribute partitions splited by row key - Distribute partitions splited by row key - Distribute clusters by high dimensional index Performance enhancement of key-based/content based retrieval Access Control - None User management Privilege management of table/column Provide data security High Availability Fail-over of partition management node  serial processing log file and parallel recovery Fail-over of partition management node  parallel processing log file and parallel recovery Fail-over of master node - Increase availability - Reduction of down time Query Language - Use in shell - Use in application Easy to develop application

Plans, Expectations (3/3) Category Function What OSCAR Cluster Orchestration Structure Hierarchical Flat Scalability Automatic Reconfiguration Pxe+DHCP Availability Independent HA Tool Activ-active(2 head node) Management Interface Web X-GUI, Command(C3) Communication XML XDR, XML IP Management Server Configuration DHCP auto/static Maximum Nodes 10,000 per data center / Max. 1,000,000 Oscar 440 Load Balancing Front-end LVS, Back-end new-DP Front-end PBS, TORQUE, MAUI Service Management Node Reconfiguration By Load Balancing Yes None Master Master Node Configuration Hierarchy (Key Master, Cluster Master, Group Master) Head node Resources Monitoring Monitoring Tool Proprietary Ganglia Provisioning Provisioning (image) OS imaging Provisioning (streaming) SW streaming SW tar/rpm

Thank you