August 28, 2003APAN, Logistical Networking WS DiDaS Distributed Data Storage Ludek Matyska Masaryk University, Institute of Comp. Sci. and CESNET, z.s.p.o

August 28, 2003APAN, Logistical Networking WS Outline Motivation Infrastructure Applications Future extensions

August 28, 2003APAN, Logistical Networking WS Motivation Increased need for network storage –Computational Grids –Data Grids –Temporary Data Deposits –Transient Caches –Video deposits National Library Requirements –Distribution of digitized content

August 28, 2003APAN, Logistical Networking WS Requirements Transparent Location independent –Good geographical distribution Providing support for –Access quality (e.g. Streaming) –Reliability (no single point of failure)

August 28, 2003APAN, Logistical Networking WS Infrastructure Data depots –Control: Personal computer with Linux –Storage: RAID of IDE disks –Capacity  1,5 TB each –Number: 7 (total capacity  10 TB) Connectivity –Directly to the backbone –100 Mb/s or 1 Gb/s

August 28, 2003APAN, Logistical Networking WS

August 28, 2003APAN, Logistical Networking WS

August 28, 2003APAN, Logistical Networking WS Data Layer IBP (70% capacity) –General use GridFTP servers (30% capacity) –Grid support –Computer independent temporary data storage –Comparison with IBP based solution

August 28, 2003APAN, Logistical Networking WS Traffic optimisation Network traffic cost function Inter-depots topology known Instrumented clients –Measurement from depot to client –Simultaneous data transfer and measurements Real-time transfer rate prediction –Choose depot –Decision between point and multipoint transfers

August 28, 2003APAN, Logistical Networking WS Applications National Technical Library Video Streaming Nonspecific Users

August 28, 2003APAN, Logistical Networking WS National Technical Library Requirements –Program of content digitalisation –Data stored on the central tape robot –Not optimised for distribution Danger of overload –Model data: old cartographic maps

August 28, 2003APAN, Logistical Networking WS National Technical Library DiDaS role –Cache like storage –Load balancing optimisation –Data transfer reliability (multistreaming)

August 28, 2003APAN, Logistical Networking WS Video Streaming Permanent storage Specific clients QoS requirements (pre-caching) Replica management –Not yet implemented

August 28, 2003APAN, Logistical Networking WS Nonspecific Users Temporary data deposits Provide data for load balancing –Transfer outside of DiDaS core Access reliability –Automatic replica generation –Transparent multi-access –Ability to react on connectivity loss

August 28, 2003APAN, Logistical Networking WS Future work New clients development –support for new application areas Extended and transparent replica management Full instrumentation –Data for Load balancing Replica creation/deletion User access optimisation

August 28, 2003APAN, Logistical Networking WS Thank you for your interest