The Storage B. Ramamurthy C B. Ramamurthy1

Topics for discussion On chip memory On board memory System memory Off system/online storage/ secondary memory File system abstraction Offline/ tertiary memory RAID: Redundant Array of Inexpensive Disks NAS: Network Accessible Storage SAN: Storage area networks DB and DBMS: Data base and DB management systems Distributed file system Google file system Hadoop file system C B. Ramamurthy2

Data and Computation Continuum Compute intensive Ex: computation of digits of PI Data intensive Ex: analyzing web logs C B. Ramamurthy3

More dimensions C B. Ramamurthy4 Data scale Compute scale Payroll KMGT MFLOPS GFLOPS TFLOPS PFLOPS P Digital Signal Processing Weblog Mining Business Analytics Realtime Systems Massively Multiplayer Online game (MMOG) Other variables: Communication Bandwidth, ?

Solution Processing Granularity Pipelined Instruction level Concurrent Thread level Service Object level Indexed File level Mega Block level Virtual System Level Data size: small Data size: large

On chip memory Registers Cache Buffers (instruction pipeline) Characteristics: volatile C B. Ramamurthy6

On board memory Cache – Instructions cache – Data cache – Translation look aside buffers (TLB) Characteristics: content addressable, set- associative organization C B. Ramamurthy7

System memory RAM : Random access memory: main memory Read and write possible volatile ROM: Read only memory: boot programs for operating systems Flash memory: Erasable/writable non-volatile memory SDRAM: synch dynamic RAM others EAROM C B. Ramamurthy8

Off-system storage (Earlier Lectures covered these) Off system/online storage/ secondary memory File system abstraction Offline/ tertiary memory RAID: Redundant Array of Inexpensive Disks NAS: Network Accessible Storage SAN: Storage area networks C B. Ramamurthy9

Database and Database Management System Data source Transactional Data base server Relational db or similar foundation Tables, rows, result set, SQL ODBC: open data base connectivity Very successful business model: Oracle, DB2, MySQL, and others Persistence models: EJB, DAO, ADO (I am not going to expand the abbreviation.. ) C B. Ramamurthy10

Distributed file system(DFS) A dedicated server manages the files for an compute environment For example, nickelback,cse.buffalo.edu is your file server and that is why we did not want you to run your user applications on this machine. DFS addresses various transparencies: location transparency, sharing, performance etc. Examples: NFS, NFS+, AFS (Andrew FS)… (you will study these in Distributed Systems course) C B. Ramamurthy11

On to Google File Internet introduced a new challenge in the form web logs, web crawler’s data: large scale “peta scale” But observe that this type of data has an uniquely different characteristic than your transactional or the “order” data on amazon.com: “write once” ; so is HIPPA protected healthcare and patient information; Google exploited this characteristics in its Google file system: S. GhemavatGoogle file system: S. Ghemavat C B. Ramamurthy12

Hadoop File System (HFS) Hadoop file system is a reverse engineered version of the GFS : this is my first opinion on HFS HFS is a distributed file system for large scale data Data throughput is more important than latency Batch computing than interactive time shared computing C B. Ramamurthy13

Cat Bat Dog Other Words (size: TByte) map split combine reduce part0 part1 part2 MapReduce