Microsoft Research Microsoft Research Jim Gray Distinguished Engineer Microsoft Research San Francisco SKYSERVER.

Slides:

Advertisements

Similar presentations

Closing the Gap Between Global Environmental Sensing Needs and Cyber Infrastructure Tools Jim Gray Jeff Burch Mark Ellisman Miron Livny David Maidment.

Advertisements

Data Challenges I'm Struggling With Jim Gray, Microsoft Research 1.Sneakernet is probably the best way to moving WAN data at 1GBps File transfer efforts.

Trying to Use Databases for Science Jim Gray Microsoft Research

Online Science -- The World-Wide Telescope Archetype

Gigabyte Bandwidth Enables Global Co-Laboratories Prof. Harvey Newman, Caltech Jim Gray, Microsoft Presented at Windows Hardware Engineering Conference.

World Wide Telescope mining the Sky using Web Services Information At Your Fingertips for astronomers Jim Gray Microsoft Research Alex Szalay Johns Hopkins.

1 Online Science the New Computational Science Jim Gray Microsoft Research Alex Szalay Johns Hopkins.

1 Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research Talk at

1 Experience Building The World Wide Telescope aka: The Virtual Observatory Jim Gray Alex Szalay.

1 Online Science -- The World-Wide Telescope as an Archetype Jim Gray Microsoft Research Collaborating with: Alex Szalay, Peter Kunszt, Ani

1 Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research

Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research

1 Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research

Universal Access to All Internet Archive: Non-Profit Library.

The Australian Virtual Observatory e-Science Meeting School of Physics, March 2003 David Barnes.

Astronomy Data Bases Jim Gray Microsoft Research.

31/03/00 CMS(UK)Glenn Patrick What is the CMS(UK) Data Model? Assume that CMS software is available at every UK institute connected by some infrastructure.

Windows Azure for SharePoint people Dennis – Solution Architect Microsoft Windows Azure.

Scientific Collaborations in a Data-Centric World Alex Szalay The Johns Hopkins University.

Modeling and Maintaining Virtualized Services Microsoft System Center Virtual Machine Manager 2012 (c) 2011 Microsoft. All rights reserved.

DVDZone2.com From Linux to Windows 2003 Gregory Bronchart [web-o-net] Fabrice Cornet [BrainSys]

Mission Critical Messaging Platform Roni Havas Unified Communications Solution Specialist Specialists Technology Unit – EPG - Microsoft Israel

Development of China-VO ZHAO Yongheng NAOC, Beijing Nov

Enterprise CAL Overview. Different Types of CALs Standard CAL base A component Standard CAL is a base CAL that provides access rights to basic features.

Summary Role of Software (1 slide) ARCS Software Architecture (4 slides) SNS -- Caltech Interactions (3 slides)

DEV392: Extending SharePoint Products And Technologies Through Web Parts And ASP.NET Clint Covington, Program Manager Data And Developer Services - Office.

Business Intelligence in the 2007 Microsoft Office System Rob Gray Product Marketing Manager SharePoint Technologies.

SDSS Web Services Tamás Budavári Johns Hopkins University Coding against the Universe.

Copyright © 2006 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill Technology Education Copyright © 2006 by The McGraw-Hill Companies,

Presentation to Baltimore-Washington DC Metro MPUG Chapter September 22, 2009.

Copyright © 2010 Platform Computing Corporation. All Rights Reserved.1 The CERN Cloud Computing Project William Lu, Ph.D. Platform Computing.

Scientific Data Infrastructure in CAS Dr. Jianhui Scientific Data Center Computer Network Information Center Chinese Academy of Sciences.

Supported by the National Science Foundation’s Information Technology Research Program under Cooperative Agreement AST with The Johns Hopkins University.

Business Solutions Using Microsoft ® Office SharePoint ® Server ROADSHOW.

1 The Terabyte Analysis Machine Jim Annis, Gabriele Garzoglio, Jun 2001 Introduction The Cluster Environment The Distance Machine Framework Scales The.

1 Managing Data for the World Wide Telescope aka: The Virtual Observatory Jim Gray Alex Szalay SLAC Data Management Workshop.

1 The World Wide Telescope an Archetype for Online-Science Jim Gray (Microsoft) Alex Szalay (Johns Hopkins University) Microsoft Academic Days in Silicon.

Public Access to Large Astronomical Datasets Alex Szalay, Johns Hopkins Jim Gray, Microsoft Research.

The Data Avalanche Jim Gray Microsoft Research Talk at HP Labs/MSR: Research Day July 2004.

Michael Woods Sr. Technical Product Manager.

EScience May 2007 From Photons to Petabytes: Astronomy in the Era of Large Scale Surveys and Virtual Observatories R. Chris Smith NOAO/CTIO, LSST.

NVO Review -- San Diego Jan The VO compared to Other O‘s Jim Gray Microsoft T HE US N ATIONAL V IRTUAL O BSERVATORY.

Center for Computational Visualization University of Texas, Austin Visualization and Graphics Research Group University of California, Davis Molecular.

Grids 2003 The Great Academia/Industry Grid Debate Dan Fay | Microsoft Research Grid, grid, everywhere a Grid Blocking out the scenery, breaking my mind.

Cloud Roadshow. Advanced SharePoint add-in Development.

1 Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research

1 Where The Rubber Meets the Sky Giving Access to Science Data Jim Gray Microsoft Research Alex.

Microsoft Research San Francisco (aka BARC: bay area research center) Jim Gray Researcher Microsoft Research Scalable servers Scalable servers Collaboration.

Dan Fay Technical Computing Microsoft

Building Enterprise Applications Using Visual Studio®

5/9/2018 7:28 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS.

Scalable Web Apps Target this solution to brand leaders responsible for customer engagement and roll-out of global marketing campaigns. Implement scenarios.

Data Platform and Analytics Foundational Training

Introduction to R Programming with AzureML

Scalable Web Apps Target this solution to brand leaders responsible for customer engagement and roll-out of global marketing campaigns. Implement scenarios.

Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research

Jim Gray Alex Szalay SLAC Data Management Workshop

BARC Scaleable Servers

DeFacto Planning on the Powerful Microsoft Azure Platform Puts the Power of Intelligent and Timely Planning at Any Business Manager’s Fingertips Partner.

Rick, the SkyServer is a website we built to make it easy for professional and armature astronomers to access the terabytes of data gathered by the Sloan.

Intranet web banner units

Jim Gray Researcher Microsoft Research

Microsoft Virtual Academy

XtremeData on the Microsoft Azure Cloud Platform:

Intranet web banner units

Quasardb Is a Fast, Reliable, and Highly Scalable Application Database, Built on Microsoft Azure and Designed Not to Buckle Under Demand MICROSOFT AZURE.

Jim Gray Microsoft Research

Internet Vocabulary Terms

Microsoft Data Insights Summit

Microsoft Virtual Academy

Presentation transcript:

Microsoft Research Microsoft Research Jim Gray Distinguished Engineer Microsoft Research San Francisco SKYSERVER

Microsoft Research Organization goal: Advance state of the art More than 700 staff, 55 areas Labs in US, Europe, Asia Internationally recognized teams University organizational model Open research environment Close ties to universities Close working relations with development.

My Research Goal Information at your fingertips Bring all scientific literature and data online Focus on large database issues, and scalable servers. Experiments & Instruments Simulations facts answers questions ? Literature Other Archives facts

World Wide Telescope World Wide Telescope Premise: Most Astronomy data is online The Internet is the worlds best telescope It has data on every part of the sky In every measured spectral band: As deep as the best instruments It is up when you are up. The seeing is always great (no working at night, no clouds no moons no..). Its a smart telescope: links data with literature.

SkyServer.SDSS.org SkyServer.SDSS.org Built with Johns Hopkins U. SkyServer.SDSS.org A modern archive Raw data in file servers Catalog data (derived objects) in Database 10 billon records, 2 TB Also used for education 150 hours of online Astronomy Interesting things Based on Web Services Spatial data search Cloned by other surveys (a design template)

Service Oriented Architecture Data Federations of Web Services Massive datasets live near their owners: Near instrument software pipeline, apps Near data knowledge and curation Each Archive publishes a web service Schema: documents the data Methods on objects (queries) Uniform access to multiple Archives A common global schema Scientists get personalized extracts DB

2MASS INT SDSS FIRST SkyQuery Portal Image Cutout SkyQuery Structure Each SkyNode publishes Schema Web Service Data Query Web Service Portal Plans Query (2 phase) Integrates answers Is itself a web service

Federation: SkyQuery.Net SkyQuery.Net Combines 15 archives Send query to portal, portal joins data from archives. Problem: want to do multi-step data analysis (not just single query). Solution: Allow personal databases on portal Problem: some queries are monsters Solution: batch scheduler on portal server, Deposits answer in personal db.

Current Status: CERN Pasadena Multi Stream tpc/ip 7.1 Gbps ~900 MBps New speed Single Stream tpc/ip 6.5 Gbps ~800 MBps File Transfer Speed ~450 MBps mbps per second 0 1,000 2,000 3,000 4,000 5,000 6,000 7,

Challenge: Move Data from CERN to Remote 1GBps Disk-to-Disk Disk-to-Disk gigabyte / second data rates gigabyte / second data rates 80TB/day 80TB/day 30 petabytes by petabytes by exabyte by exabyte by 2014 ~5 GBps CERN Filter Tier 2 Tier 3 Tier 1 … INP3RALINFNFNAL Tier 2 Institute Tier 2 Institute Tier 4 Experiment ~1 GBps ~PBps.1 GBps Physics data cache ~1 GBps Workstations OC192 = 9.9 Gbps Graphics courtesy of Harvey Caltech

Summary Microsoft Research is active inside and outside Microsoft. World Wide Telescope is coming Exemplifies service oriented architecture Built with web services and databases Has interesting spatial database algorithms 10Gbps Networking is coming, x-64 is coming and we are investing to make them real. Details on my website:

© 2003 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.