Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 1 Untertitel durch Klicken bearbeiten Volker Guelzow DESY & HTW-Berlin Hamburg, Sept. 24th, 2015 Big Data Management - Motor der Wissenschaft -

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 2 Introduction What is „Big Data“, what DESY? > Volume -> PB/year > Velocity -> data ingest, analysis less time critical (compared f.i. to stock exchange) > Variety -> well structured data (compared to social media data) > Veracity -> high (compared to social media data) > Value -> high because the science is in the data

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 3 > Data Sources for DESY

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 4 Particle Physics needs

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 5 Higgs Discovery

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 6 Higgs Discovery # of Analysis Jobs

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 7 Completed WLCG Jobs per Tier-2 Site DESY Atlas CMS

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 8 Higgs Discovery Data Volume

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 9 Data Requirements from Photon DESY

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 10 PETRA III 1010 Köcherfliege (Limnephilus flacivornis) Kopf + Thorax Courtesy: Dr. F. Beckmann

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 11 Investigation of a van Gogh painting

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 12 Other examples  3D-time dependent  Pattern recognition  Fast image analyis (f.i. through neural networks)  Virtual reality (Cave )

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 13 New Challenges TypeFrame size Frame rate Peak rate Avail. Pilatus 6M 2463 x 2527 x 425 Hz4.6Gb/sNow AGIPD ( Module) 128 x 512 x 2 x 352 x 14bit 4.5 MHz (10 Hz) 6.1 Gb/s2015 Eiger1k x 1k x 22 kHz30 Gb/snow Lambda3 x 1536 x 512 x 22 kHz60 Gb/snow Percival (1S) 4k x 4k x 2120 Hz60 Gb/s2015 Percival (4S) 8k x 8k x 2120 Hz240 Gb/s Late 2015

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 14 Methods are different for light sources

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 15 The DESY ICT Eco-System Detect or Centr al Stora ge Dat a Analys is Loc al Research er Remot e Research er Loc al Cac he Onl ine Archi ve, Outs ide Simulati ons Grid NAF HPC Farm Cloud Visual izatio n Data&Metadata Management Software Data Policies Technical Infrastructure User Management/AAI networks

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 16 Hard- & Software technology evolution

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 17 Untertitel durch Klicken bearbeiten Electronic systems market value in 2014 was ~1.5 Trillion $ 10 biggest segments Moderate growth rates Maturing markets HEP is here ~15M$ out of 52B$ From Bernd Panzer, Cern CAGR = Compound Annual Growth Rate End-Use Markets

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 18 Processors INTEL, Qualcomm, Samsung, AMD, IBM Graphics INTEL, Nvidia, AMD Hard Disk Drives Western Digital, Seagate, Toshiba DRAM memory Samsung, SK Hynix, Micron NAND Flash memory Samsung, Toshiba, SanDisk, Micron, Hynix, INTEL Solid State Disks Samsung, INTEL, SanDisk, Toshiba, Micron FPGA Xilinx, Altera (currently being bought by INTEL) Tape Storage HP, Fuji, IBM, SpectraLogic, ORACLE Only a few large companies are dominating the various components markets Market Dominance Few companies capable of large scale investments, majority fabless companies Favour evolutionary (adiabatic) changes of technology Clear bias against ‘disruptive’ new technologies (memristor, holographic storage, DNA storage,quantum computing, non-volatile memory, etc.)

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 19 Hardware technology perspective > 20% increase of compute power/year per $ > 15% increase of disk capacity/year per $ > Tape will still improve very much but the role will change > Only a few vendors, this is risky > Evolution, no disruptive changes > Application development for multicore/GPU‘s needed > A rapid network development to Tbit/s

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 20 dCache – OwnCloud Data Management Spinning Disks Tape, Blue-ray … Unlimited hierarchical Storage Space NFS 4.1 CDMI WEB 2.0 dCache SSD’S

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 21 dCache Big Data Cloud LOFAR antenna Huge amounts of data X-FEL (Free Electron Lasers) Fast Ingest

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 22 Towards the DESY Strategy 2 Examples: > Speed > HNSciCloud

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 23 The Speed Project for Beamlines: Cooperation DESY/IBM

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 24 The Architecture

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 25 April 28, 2015 The HNSciCloud Project European Open Science Cloud Pilot Project > Bring together the stakeholders  Research Infrastructures (ESFRI, etc.)  Research Organisations (WLCG tier-1 etc.)  European e-Infrastructures (GEANT, EGI, PRACE, EUDAT, OpenAIRE)  Commercial cloud service providers (Helix Nebula, etc.)  End-users including the long-tail of science > Deliver the pilot  Technical architecture for the hybrid cloud  Security model compatible with EU data protection legislation  Assemble and deploy a 5% scale prototype  Verify the business model to ensure it can be sustained beyond the pilot  Governance structure avoiding monopoly of any research group or service provider > Roadmap for full-scale implementation > Today still more cost effective to operate our own facilities, but this situation is expected to change -> spot market > Hybrid model gives us flexibility  Does not save staff effort as we still need to operate services there, as well as maintaining in-house services

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 26 April 28, 2015 Project Consortium > Includes buyers and experts in the preparation, execution and promotion of the procurement > Idea to use EC procurement calls to co-fund exploratory joint-procurement of cloud services > Proposal submitted on April 14 > Duration:30 month EGI.eu – Integration with e-infrastructures TRUST-IT - Comms/Dissem. CERN, DESY, EMBL-EBI, KIT, IN2P3, INFN, PIC, SARA/Nikhef, STFC BuyersExpertsConsortium Sub-contractor experts: Strategic Blue - Cloud Financial Broker Pinsent Mason – Cloud legal advisor Trento Network - EC PCP Legal Advisor The buyers are public organisations that commit to contribute to a joint procurement

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 27 … and a lot more to do > AAI: access to federated resources (adapting existing solutions from other projects like EduGain based solutions) > Security, policies, life cycle management (like „how long? Ownership? Who has access? What kind of data?..) > Portals for scientific and industrial users (like access to resources, virtual accounting, industrial usage,..) > Open Access

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 28 The DESY Big Data strategy in a nutshell (1) > Develop high sophisticated „Big Data“ management solutions fully alined to the research topics at DESY > … fitting on site experiments and off site experiments > Close cooperation with the scientists, directly in scientific projects > Find solutions in cooperation with other Lab‘s > Apply for third party funding > Cooperation with industry > Assure 24x7 operation > Serve Eu-Xfel (and others) on a full cost model

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 29 The DESY Big Data strategy in a nutshell (2) > Share resources as much as possible between communities > Prepare for a hybrid model „data on site – compute partially on the spot market“ > Development of data management software (-> dCache) > Development of data portals -> Gamma Portal > Extend DESY Data Cloud > Provide excellent analysis facilities, local and Cloud based > Define Data Policies with experiments > Offer Long Term Data Preservation > Continously upgrade networking

Big Data Minds 2015 | DESY-IT | V. Gülzow | Seite 30 Final No Computing - no Science