IB in the Wide Area How can IB help solve large data problems in the transport arena.

Slides:



Advertisements
Similar presentations
1 Use Cases Application provisioning (version control) Workload management/load-balancing (server consolidation) Data Federation/sharing E-utilities (provisioning.
Advertisements

System Center 2012 R2 Overview
PNFS, 61 th IETF, DC1 pNFS: Requirements 61 th IETF – DC November 10, 2004.
XSEDE 13 July 24, Galaxy Team: PSC Team:
Scale-out Central Store. Conventional Storage Verses Scale Out Clustered Storage Conventional Storage Scale Out Clustered Storage Faster……………………………………………….
Optimizing the User Experience Throughout the Infrastructure Consolidation Process Dan Smith, Enterprise Solutions Manager, GTSI Chris Theon, Practice.
Network-Attached Storage
 Introduction Originally developed by Open Software Foundation (OSF), which is now called The Open Group ( Provides a set of tools and.
Business Continuity and DR, A Practical Implementation Mich Talebzadeh, Consultant, Deutsche Bank
SERVER LOAD BALANCING Presented By : Priya Palanivelu.
Grids and Grid Technologies for Wide-Area Distributed Computing Mark Baker, Rajkumar Buyya and Domenico Laforenza.
World Wide Web Caching: Trends and Technology Greg Barish and Katia Obraczka USC Information Science Institute IEEE Communications Magazine, May 2000 Presented.
STORAGE Virtualization
Module – 7 network-attached storage (NAS)
Reducing Risk with Cloud Storage. Dell Storage Forum 2011 Storage 2 Dells’ Definition of Cloud Demand driven scalability: up or down, just happens Metered:
Office 365: Efficient Cloud Solutions Wednesday March 12, 9AM Chaz Vossburg / Gabe Laushbaugh.
Windows ® Powered NAS. Agenda Windows Powered NAS Windows Powered NAS Key Technologies in Windows Powered NAS Key Technologies in Windows Powered NAS.
MaxTronic International Co., Ltd. May 2013 Focused on Data Protection.
To provide the world with a next generation storage platform for unstructured data, enabling deployment of mobile applications, virtualization solutions,
Distributed Systems Early Examples. Projects NOW – a Network Of Workstations University of California, Berkely Terminated about 1997 after demonstrating.
Object-based Storage Long Liu Outline Why do we need object based storage? What is object based storage? How to take advantage of it? What's.
SANPoint Foundation Suite HA Robert Soderbery Sr. Director, Product Management VERITAS Software Corporation.
Module 11: Designing Files Services and DFS in Windows Server® 2008
Module 12: Designing High Availability in Windows Server ® 2008.
Page 1 of John Wong CTO Twin Peaks Software Inc. Mirror File System A Multiple Server File System.
© Oxford University Press 2011 DISTRIBUTED COMPUTING Sunita Mahajan Sunita Mahajan, Principal, Institute of Computer Science, MET League of Colleges, Mumbai.
Designing Authentication for a Microsoft Windows 2000 Network Designing Authentication in a Microsoft Windows 2000 Network Designing Kerberos Authentication.
Protect Your Business-Critical Data in the Cloud with SoftNAS, a Full-Featured, Highly Available Solution for the Agile Microsoft Azure Platform MICROSOFT.
Moving Large Amounts of Data Rob Schuler University of Southern California.
Storage Tank in Data Grid Shin, SangYong(syshin, #6468) IBM Grid Computing August 23, 2003.
Introduction to DFS. Distributed File Systems A file system whose clients, servers and storage devices are dispersed among the machines of a distributed.
10 1 Chapter 10 Distributed Database Management Systems Database Systems: Design, Implementation, and Management, Sixth Edition, Rob and Coronel.
Oracle's Distributed Database Bora Yasa. Definition A Distributed Database is a set of databases stored on multiple computers at different locations and.
1 Public DAFS Storage for High Performance Computing using MPI-I/O: Design and Experience Arkady Kanevsky & Peter Corbett Network Appliance Vijay Velusamy.
Fast Crash Recovery in RAMCloud. Motivation The role of DRAM has been increasing – Facebook used 150TB of DRAM For 200TB of disk storage However, there.
WHAT OUR CUSTOMERS ARE SAYING “After thorough market research and a review process, Qorus Breeze Proposals stood out from the competitors because of its.
Distributed Data Management Graeme Kerr Oracle in R&D Programme.
Access and Information Protection Product Overview Andrew McMurray Technical Evangelist – Windows
OpenField Consolidates Stadium Data, Provides CRM and Analysis Functions for an Intelligent, End-to-End Solution COMPANY PROFILE : OPENFIELD Founded by.
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. NAS versus SAN NAS – Architecture to provide dedicated file level access.
 Introduction  Architecture NameNode, DataNodes, HDFS Client, CheckpointNode, BackupNode, Snapshots  File I/O Operations and Replica Management File.
Features Of SQL Server 2000: 1. Internet Integration: SQL Server 2000 works with other products to form a stable and secure data store for internet and.
Roadmap to Next Generation Internet: Indian Initiatives Subbu C-DAC, India.
To provide the world with a next generation storage platform for unstructured data, enabling deployment of mobile applications, virtualization solutions,
Flight is a SaaS Solution that Accelerates the Secure Transfer of Large Files and Data Sets Into and Out of Microsoft Azure Blob Storage MICROSOFT AZURE.
Data Evolution: 101. Parallel Filesystem vs Object Stores Amazon S3 CIFS NFS.
Tackling I/O Issues 1 David Race 16 March 2010.
1 CEG 2400 Fall 2012 Network Servers. 2 Network Servers Critical Network servers – Contain redundant components Power supplies Fans Memory CPU Hard Drives.
1 Copyright © 2012 Tata Consultancy Services Limited Windchill Architecture.
An Introduction to GPFS
Introduction to Data Analysis with R on HPC Texas Advanced Computing Center Feb
GPFS Parallel File System
1 © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events.
AuraPortal Cloud Helps Empower Organizations to Organize and Control Their Business Processes via Applications on the Microsoft Azure Cloud Platform MICROSOFT.
Reducing Risk with Cloud Storage
Meemim's Microsoft Azure-Hosted Knowledge Management Platform Simplifies the Sharing of Information with Colleagues, Clients or the Public MICROSOFT AZURE.
Unit OS8: File System 8.6. Quiz
Brookhaven National Laboratory Storage service Group Hironori Ito
Geographically distributed storage over multiple universities
Gregory Kesden, CSE-291 (Storage Systems) Fall 2017
Gregory Kesden, CSE-291 (Cloud Computing) Fall 2016
Module – 7 network-attached storage (NAS)
Voice Analytics on Microsoft Azure Allows Various Customers to Get the Most Out of Conversations with Clients Through Efficient Content Analysis MICROSOFT.
Dell Data Protection | Rapid Recovery: Simple, Quick, Configurable, and Affordable Cloud-Based Backup, Retention, and Archiving Powered by Microsoft Azure.
Media365 Portal by Ctrl365 is Powered by Azure and Enables Easy and Seamless Dissemination of Video for Enhanced B2C and B2B Communication MICROSOFT AZURE.
Consistency and Replication
Research Opportunities in IP Wide Area Storage
BluSync by ParaBlu Offers Secure Enterprise File Collaboration and Synchronization Solution That Uses Azure Blob Storage to Enable Secure Sharing MICROSOFT.
Outline Review of Quiz #1 Distributed File Systems 4/20/2019 COP5611.
Presentation transcript:

IB in the Wide Area How can IB help solve large data problems in the transport arena

Enhanced Capabilities Near real-time access of operational data across the global filesystem –Rapid access to data means faster, more comprehensive, more accurate analysis –Reach back/forward to any data in enterprise to enhance change/pattern, I&W and critical node analysis Enables Real-time (Within Seconds) Ingestion of huge data sets Enhanced situational awareness –Enables dynamic updates to users and analysts –Search within theater data streams Upgraded local storage/caching in reduced footprint

1. Global Situational Awareness – requires dynamic access from very large, distributed data centers 2. Latest imagery streams into global services 3. OTD and Commercial applications used to visualize new imagery 4. Cross-node dynamic data sharing for collaboration

Why do we need a scalable Global File System Must support 100s of users working on thousands of large (GB and TB) file sets –There is nothing like this today with the needed information assurance features –The closest is what some in the HPC are doing on clusters –Must work on multiple transport and storage protocols to get to end users IP, IB, FC, NFS, CIFS, etc Global – Must work over long distances and multiple data centers

Storage-wise… The earlier ‘truths’ drive demand for the following capabilities 1.Capability: disaster recovery Data replication for disaster recovery 2.Capability: performance Geographic locality of data for performance reasons Caching, prefetching 3.Capability: wide area accessability All clients have access to the same datasets (+/- security reqmts) In a nutshell, you’d like to be able to inject raw information into the system from anywhere, and you’d like to provide access to both the raw information and processed data to consumers located anywhere. And, you want to be sure the data is protected and continuously accessible. That’s it!

Storage-wise… This suggests, in turn… –a data store which is globally accessible to facilitate availability and disaster recovery –a globally coherent, application level name space That means that a file identifier used by an application located anywhere in the system will always resolve to the same underlying data. This is important to enable global access to data, to simplify disaster recovery, to support highly available systems and for a host of other reasons. The idea is to make providers and consumers of information truly portable

There are at least three ways to achieve a globally coherent file-level namespace: 1.Brute force: Create a single, globally flat file system 2.Create a single global block-level name space, then force a strong binding to a consistent file-level name space among a set of distributed, federated filesystems 3.Create a consistent file-level name space among a set of distributed, federated filesystems

A globally coherent file-level namespace file system site 1 f/s 1f/s ‘n’ … f/s 1f/s ‘n’ BV site ‘n’ site 1 … site ‘n’ site 1 … site ‘n’ File I/O block I/O As long as: coherence is maintained between the federated filesystems, and as long as each f/s has access to the global data pool, there is no need to enforce a strict binding between block-level I/O and file-level I/O. Coherence at the block I/O level is neither necessary nor sufficient. Global file system Federated file system Block approach

An Idea of a Wide and Distributed Data Center Construct Core Node Components InfiniBand WAN GATEWAY ENCRYPTOR (10 Gbps) STORAGE ARRAY (Petabytes) NetworkStorageController LAN with Local Servers, Apps, & Nets InfiniBand WAN OC-192 (10 Gbps) Core VLH Other Users