Learning Google foster@hf.webex.com

Reference http://zh.wikipedia.org/zh/MapReduce Engineering Case Study training by Peter Xiao and Stanley Huang: https://go.webex.com/go/ldr.php?AT=pb&SP=MC&rID=4363322&rKey=5c29baa6dc43ce4c http://zh.wikipedia.org/zh/MapReduce http://code.google.com/appengine/ http://www.codechina.org/doc/google/gfs-paper/guarantees-by-gfs.html http://royal.pingdom.com/2008/04/11/map-of-all-google-data-center-locations/

Agenda Google Overview Data Center Google File System Map Reduce Big Table Google App Engine -Demo

Google Overview Why named google: typo from word "googolplex" , mathematical term for a 1 followed by 100 zeros Mission:Organize the world’s information and make it universally accessible and useful Infrastructure: Three layer stack

Google Data Center - Overview Total of 35 data centers globally 19 in US 12 in Europe 3 in Asia 1 in South America. 1 in Russia

Google Data Center – Cost and Scale According to Google’s earning report, 1.9 billion for 2006 2.4 billion for 2007 4 data centers proposed in 2007, each costs $600 million Power consumption 50-100+ Mega Watts per major data center Size No standard physical size Google’s data center at Dalles,Oregon(俄勒冈州 ) 2000 square foot administration building 1600 square foot “transient employee dormitory” 1800 square foot facility for cooling towers (Estimate of power consumption is 103 mage watts)

Google Data Center – Interior & Exterior Google data center at Dalles,Oregon

Google Data Center – Hardware and Software Google customizes commodity hardware to minimize energy consumption Google web servers Google Ethernet switches Google builds in-house software to achieve high performance and scalability Google web servers (GWS) Google Front End (WFE) Google File System (GFS) Google MapReducer Google BigTable

Google Data Center – Service Availability All 32 data centers reached 4 nines of uptime

Google Data Center vs Cisco/CSG’s

Architecture Foundation - overview Architecture serve the Mission organize the world’s information and make it universally accessible and useful Storage For Raw data Google file system Distributed On Thousands of Machines on Tans of Data Centers Backend computing K-V Relational Data storage Map Reduce Big Table

Architecture Foundation - GFS Background Typical way to store persistent file data; Local Disk, NFS, Storage GFS is a scalable(~100TB) distributed file system. on top of thousands of machine Goal: Performance and Scalability for Large file and Concurrent visit Workflow

Architecture Foundation – GFS –Cont’ Dive in further Hadoop: Java based open-source software for reliable, scalable, distributed computing. HDFS: Hadoop File System, a similar distributed file system as GFS GFS is good in handing large file with Appending(no random editing) write and tons of read KFS:（KOSMOS DISTRIBUTED FILE SYSTEM），一个类似GFS、Hadoop中HDFS 的一个开源的分布式文件系统 Where we are Distributed storage concept has been used in Queue&Dispatch service(WAPI2.0) and Search Farm

Architecture Foundation - MapReduce Background Typical way to do computing:local CPU, parallel computing in application level MapReduce is a programming Model for computing large data set by distribution Goal:computing Terabytes of data on thousands of machines for performance Example: Google Page rank; 1 Terabytes file, Calculate count of every word Workflow Pseudo Code Map(Stirng key,String value) //key: document name //Value: document contents For each word w in value Emitlntermediate(w,”1”); Reduce(String key,Merator values) //key: a word //value: a list of counts Int result=0; For each v in values; Result += ParseInt(v); Emit(AsString(result));

Architecture Foundation – MR – cont’ Dive in further MapReduce is good for Simple large computing work Hadoop map reduce provide similar functionality Where we are Some computing happened in Oracle DB layer Many computing happened parallel in application layer, example: Search Farm, Activity Server etc.

Architecture Foundation – BigTable Background BigTable is a scalable, distributed, multi-dimensional K-V store Goal: High performance One way search in large data volume Example: Google earth, grab all geographic data based on location Workflow

Architecture Foundation – BigTable – cont’ Dive in further Cassandra is Open Source implementation for Big Table concept Data Store design is all about: CAP (Data Consistency, Availability, Partition) Which one you focus? Depend on what value/user experience you try to provide. Where we are Cassandra with modification has been used in WAPI 2.0 for User Wall/Feed Memcached used from WAP2.0

Google App Engine – Development and deployment deom An experienced developer can develop and deploy a “Hello word” application to App Engine within 1-2 hours Create and App Engine account Download App Engine SDK or Eclipse with App Engine plug-in Develop “Hello World” application Deploy application Access “Hello World” Application via http://appID.appspot.com/appname

Thanks！