Presentation is loading. Please wait.

Presentation is loading. Please wait.

Company Proprietary Karl Kortkamp, SE, SGI New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts.

Published byEmily Grasty Modified over 10 years ago

Similar presentations

Presentation on theme: "Company Proprietary Karl Kortkamp, SE, SGI New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts."— Presentation transcript:

1 Company Proprietary Karl Kortkamp, SE, SGI New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts

2 Company Proprietary Karl Kortkamp, SE, SGI New Mexico Computing Applications Center HPC for New Mexico

3 Company Proprietary Karl Kortkamp, SE, SGI Overall Top-level Diagram 10 Gig-E Encanto Altix ICE System 14,336 Compute Cores 1,792 Nodes 28 Racks UNM Altix ICE Exemplar 176 Compute Cores 22 Nodes 1 Rack NMSU Altix ICE Exemplar 176 Compute Cores 22 Nodes 1 Rack New Mexico Tech Altix ICE Exemplar 176 Compute Cores 22 Nodes 1 Rack

4 Company Proprietary Karl Kortkamp, SE, SGI Main System Encanto Overview Major Components 172 TFLOPS Altix ICE 8200 Cluster 1792 Nodes, 3,584 Sockets, 14,336 Cores 28.7 TB memory (2GB/core) 172 TB Parallel File Subsystem (Lustre) 20 OSS Service Nodes 20 TB NAS & Archive Storage Subsystem 1 System Admin Controller 28 Rack Leader Controllers 1 Batch Service Node 5 Login Service Nodes Lambda Rail Connectivity 36 Racks – 28 Compute – 5 Storage – 3 Support

5 Company Proprietary Karl Kortkamp, SE, SGI Encanto Top-level Diagram 33 IB Lines 1 Lamba Rail Fiber 120 Lambda Rail Gateway 5 Login Service Nodes IB DDR Gig-E10 Gig-E Login Service Nodes Login Service Nodes Login Service Nodes Login Service Nodes Qty 5 1 1 System Admin Controller Batch Node 20 Lustre-Based Parallel Storage Sub-system 172 TB Usable 5 2 4 NAS CUBE / Archive Storage Sub-system NAS: 20 TB Usable Arch: 40TB Uncompressed 10 Altix ICE 8200 Cluster 14,336 Compute Cores 1,792 Nodes 28 Racks

6 Company Proprietary Karl Kortkamp, SE, SGI NMCAC Qwest Fiber Connections to New Mexico GigaPop 505 Marquette Albuquerque, NM Lambda Rail Point of Presence 4100 Sara Road Rio Rancho, NM Encanto Supercomputer Existing National Lamba Rail Fiber Ring

7 Company Proprietary Karl Kortkamp, SE, SGI 1 3 Encanto Lambda-Rail Router Diagram Lamba Rail Fiber IB DDR Gig-E10 Gig-E 10 Future Expansion 2 1 2 1 2 1 2 1 2 1 Gateway/Router Cisco 6509 10-GigE Router 5 Login Service Node Altix ICE 8200 Cluster 14,336 Compute Cores 1,792 Nodes 28 Racks 5 1 1 1 1 1

8 Company Proprietary Karl Kortkamp, SE, SGI Login Node NMCAC Switch Owned by NMCAC Router between Fibre & Encanto Redundant AC Power Routing module to 5 Login Nodes Load-leveling Module NMCAC Switch Owned by NMCAC Redundant DC Power Cisco 6509 Switch Qwest Geomax Cisco 6509 Switch Cisco 6513 Switch UNM Switch Owned by UNM Connected to NLR Connects to NMSU Connects to NMTECH Redundant DC Power UNM NMTECH NMSU National Lamba Rail Encanto External Network Diagram 505 Marquette Site4100 Sara Road Site

9 Company Proprietary Karl Kortkamp, SE, SGI … Encanto Lustre-Based Storage System 10 Lustre Storage Units Lustre Storage Unit 1 1 20 SAS IB DDR Gig-EHeartbeat FC4 Altix ICE 8200 Cluster 14,336 Compute Cores 1,792 Nodes 28 Racks Meta-data Servers IS220 6015B IS4500 OST XE310 Node OSS 1 1 XE310 Node IS4500 OST XE310 Node OSS 1 1 XE310 Node IS4500 OST XE310 Node OSS 1 1 XE310 Node IS4500 OST XE310 Node OSS 1 1 XE310 Node Place OSS XE310 servers on top of the following ICE racks (5, 6, 7, 8, 9, 10, 11, 12, 24, 25)

10 Company Proprietary Karl Kortkamp, SE, SGI Object Storage Servers Encanto Lustre Storage Unit IS4500 Dual-controller 2GB Cache FC 1 1 Object Storage Targets 12x ( 4 + 1 ) LUNs + 4 Hot Spares Across 4 FC Loops FC RAID Drive Tray Loop FC RAID Drive Tray Loop FC RAID Drive Tray Loop FC RAID Drive Tray Loop Altix XE 310 Node 2x3 GHz quad-core 16GB memory FC 1 IB Altix XE 310 Node 2x3 GHz quad-core 16GB memory FC 1 IB 1 1 64x 400GB 15K rpm Fibre Channel drives

11 Company Proprietary Karl Kortkamp, SE, SGI OSSs Lustre Storage Cabling Details OSTs IS4500 Dual-controller FC FC RAID Drive Tray FC4 FC RAID Drive Tray FC4 FC RAID Drive Tray FC4 FC RAID Drive Tray FC4 Altix XE 310 Node FC 1 IB Altix XE 310 Node FC IS4500 Dual-controller FC FC RAID Drive Tray FC4 FC RAID Drive Tray FC4 FC RAID Drive Tray FC4 FC RAID Drive Tray FC4 Altix XE 310 Node FC Altix XE 310 Node FC 1 IB 1 1 IB DDR Heartbeat FC4 Gig-E

12 Company Proprietary Karl Kortkamp, SE, SGI Altix 450 Combined NAS & Archive Storage Subsystem Altix 450 Blade Enclosure Atempo Archive Server SL500 Dual-LT03 Tape Drives FC Control LineData Lines Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Altix 450 Blade Enclosure NAS Head Node IS4000 RAID-controller FC 4 IB NUMAlink 12 11 SATA RAID Drive Tray FC SATA RAID Drive Tray FC SATA RAID Drive Tray FC

13 Company Proprietary Karl Kortkamp, SE, SGI Altix ICE Rack 014 (Front) Altix ICE Rack 013 (Front) Altix ICE Rack 015 (Front) Altix ICE Rack 016 (Front) Altix ICE Rack 006 (Front) Altix ICE Rack 005 (Front) Altix ICE Rack 007 Altix ICE Rack 008 (Front) Altix ICE Rack 010 (Front) Altix ICE Rack 009 Altix ICE Rack 011 (Front) Altix ICE Rack 012 (Front) Archive Rack {Front) 028 Altix ICE Rack {Front) 027 Altix ICE Rack {Front) 026 Altix ICE Rack {Front) 025 Altix ICE Rack {Front) 020 Altix ICE Rack {Front) 019 Altix ICE Rack {Front) 018 Altix ICE Rack {Front) 017 Altix ICE Rack {Front) 024 Altix ICE Rack {Front) 023 Altix ICE Rack {Front) 022 Altix ICE Rack {Front) 021 Altix ICE Rack (Front) Storage Rack 001 (Front) Storage Rack 002 (Front) Storage Rack 003 (Front) Storage Rack 004 (Front) Storage Rack 005 Encanto Floor Plan {Front) SWITCH Rack Altix ICE Rack 002 (Front) Altix ICE Rack 001 (Front) Altix ICE Rack 003 (Front) Altix ICE Rack 004 (Front) ADMIN Rack (Front) Lustre IB connections Chilled Water & Condensate Indicates overhead cable trellis XE310

14 Company Proprietary Karl Kortkamp, SE, SGI Encanto Compute Rack (28) Rack Leader Ctrl – 6015B Lustre OSS I/O Service Nodes (XE310 servers) are placed on top of the following ICE racks (5, 6, 7, 8, 9, 10, 11, 12, 24, 25) Altix ICE 8200 Blade Enclosure (16 blades) Altix ICE Rack (Front)

15 Company Proprietary Karl Kortkamp, SE, SGI Encanto Admin Rack Sys Admin Ctrl – 6015B Login Node – 6015B Batch Node – 6015B Login Node – 6015B Administrator Console Altix 450 Blade Enclosure IS4000 Drive Tray (16 drives) NAS CUBE & Archive Main Cluster Service Nodes IS4000 Controller Altix 450 Blade Enclosure ADMIN Rack (Front)

16 Company Proprietary Karl Kortkamp, SE, SGI MDS Node – 6015B IS220 (12 drives) Encanto Switch Rack {Front) Support ICE Rack Sys Admin Ctrl (Test) – 6015B Login Node (Test) – 6015B Rack Leader Ctrl (Test) – 6015B Administrator Console (Test) Lustre Meta Data Server Test ICE System Altix ICE 8200 Blade Enclosure (Test) (16 blades) Cisco 6509 10-GigE Switch

17 Company Proprietary Karl Kortkamp, SE, SGI Encanto Luster Storage Rack (5) IS4500 Controller IS4500 Drive Tray (16 drives) IS4500 Controller Lustre Storage Unit Storage Rack (Front) Lustre Storage Unit

18 Company Proprietary Karl Kortkamp, SE, SGI Encanto Archive Library Cartridge Rack (Front) Archive Rack STK SL500 Cartridge Expansion Unit Archive & Backup Cartridge Storage Units

19 Company Proprietary Karl Kortkamp, SE, SGI Major Components 2.1 TFLOPS Altix ICE 8200 Cluster 22 Nodes, 44 Sockets, 176 Cores 352 TB memory (2GB/core) 16 TB NAS Storage Subsystem 1 System Admin Controller 1 Rack Leader Controller 1 Login Service Node Lambda Rail Connectivity 1 Consolidated Rack Exemplar System Overview

20 Company Proprietary Karl Kortkamp, SE, SGI Exemplar Top-level Diagram 33 IB Lines 1 Lamba Rail Fiber Lambda Rail Gateway 33 IB DDR Gig-E10 Gig-E Login Service Node System Admin Controller 4 Altix ICE 8200 Cluster 176 Compute Cores 22 Nodes 1 Rack 2 NAS Storage Subsystem 16 TB Raw

21 Company Proprietary Karl Kortkamp, SE, SGI Altix 450 NAS Head Node Exemplar NAS Storage Subsystem 4 IB IS4000 RAID-controller FC 11 SATA RAID Drive Tray FC SATA RAID Drive Tray FC

22 Company Proprietary Karl Kortkamp, SE, SGI Exemplar System Rack Rack Leader Controller – 6015B Altix ICE 8200 Blade Enclosure (11 blades) Administrator Console Altix 450 NAS Storage CUBE IS4000 Controller and 16 Drives Altix ICE 8200 Blade Enclosure (11 blades) Sys Admin Controller – 6015B Login Node – 6015B Drive Tray (16 Drives) Altix ICE Cluster

Download ppt "Company Proprietary Karl Kortkamp, SE, SGI New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts."

Similar presentations

Factor each trinomial:

Factor each trinomial:
MicroTerabyte Leveraging InfiniBand to Build a Powerful, Scalable Oracle Database and Application Platform Brian Dougherty Chief Architect, CMA.

MicroTerabyte Leveraging InfiniBand to Build a Powerful, Scalable Oracle Database and Application Platform Brian Dougherty Chief Architect, CMA.
Hadoop at ContextWeb February 2009. 2 ContextWeb: Traffic Traffic – up to 6 thousand Ad requests per second. Comscore Trend Data:

Hadoop at ContextWeb February ContextWeb: Traffic Traffic – up to 6 thousand Ad requests per second. Comscore Trend Data:
Copyright © 2003 Pearson Education, Inc. Slide 1 Computer Systems Organization & Architecture Chapters 8-12 John D. Carpinelli.

Copyright © 2003 Pearson Education, Inc. Slide 1 Computer Systems Organization & Architecture Chapters 8-12 John D. Carpinelli.
Labour quality and ICT capital skill complementarity: International comparisons Mary OMahony NIESR.

Labour quality and ICT capital skill complementarity: International comparisons Mary OMahony NIESR.
Properties Use, share, or modify this drill on mathematic properties. There is too much material for a single class, so you’ll have to select for your.

Properties Use, share, or modify this drill on mathematic properties. There is too much material for a single class, so you’ll have to select for your.
Multiplication X 1 1 x 1 = 1 2 x 1 = 2 3 x 1 = 3 4 x 1 = 4 5 x 1 = 5 6 x 1 = 6 7 x 1 = 7 8 x 1 = 8 9 x 1 = 9 10 x 1 = 10 11 x 1 = 11 12 x 1 = 12 X 2 1.

Multiplication X 1 1 x 1 = 1 2 x 1 = 2 3 x 1 = 3 4 x 1 = 4 5 x 1 = 5 6 x 1 = 6 7 x 1 = 7 8 x 1 = 8 9 x 1 = 9 10 x 1 = x 1 = x 1 = 12 X 2 1.
Division ÷ 1 1 ÷ 1 = 1 2 ÷ 1 = 2 3 ÷ 1 = 3 4 ÷ 1 = 4 5 ÷ 1 = 5 6 ÷ 1 = 6 7 ÷ 1 = 7 8 ÷ 1 = 8 9 ÷ 1 = 9 10 ÷ 1 = 10 11 ÷ 1 = 11 12 ÷ 1 = 12 ÷ 2 2 ÷ 2 =

Division ÷ 1 1 ÷ 1 = 1 2 ÷ 1 = 2 3 ÷ 1 = 3 4 ÷ 1 = 4 5 ÷ 1 = 5 6 ÷ 1 = 6 7 ÷ 1 = 7 8 ÷ 1 = 8 9 ÷ 1 = 9 10 ÷ 1 = ÷ 1 = ÷ 1 = 12 ÷ 2 2 ÷ 2 =
Jeopardy Q 1 Q 6 Q 11 Q 16 Q 21 Q 2 Q 7 Q 12 Q 17 Q 22 Q 3 Q 8 Q 13

Jeopardy Q 1 Q 6 Q 11 Q 16 Q 21 Q 2 Q 7 Q 12 Q 17 Q 22 Q 3 Q 8 Q 13
Jeopardy Q 1 Q 6 Q 11 Q 16 Q 21 Q 2 Q 7 Q 12 Q 17 Q 22 Q 3 Q 8 Q 13

Jeopardy Q 1 Q 6 Q 11 Q 16 Q 21 Q 2 Q 7 Q 12 Q 17 Q 22 Q 3 Q 8 Q 13
Multiplication Facts Review. 6 x 4 = 24 5 x 5 = 25.

Multiplication Facts Review. 6 x 4 = 24 5 x 5 = 25.
1 1  1 =.

1 1  1 =.
FACTORING ax2 + bx + c Think “unfoil” Work down, Show all steps.

FACTORING ax2 + bx + c Think “unfoil” Work down, Show all steps.
Storage Procurements Some random thoughts on getting the storage you need Martin Bly Tier1 Fabric Manager.

Storage Procurements Some random thoughts on getting the storage you need Martin Bly Tier1 Fabric Manager.
C1 Sequences and series. Write down the first 4 terms of the sequence u n+1 =u n +6, u 1 =6 6, 12, 18, 24.

C1 Sequences and series. Write down the first 4 terms of the sequence u n+1 =u n +6, u 1 =6 6, 12, 18, 24.
Lustre WAN @ 100GBit Testbed Michael Kluge michael.kluge@tu-dresden.de Robert Henschel, Stephen Simms {henschel,ssimms}@indiana.edu.

Lustre 100GBit Testbed Michael Kluge Robert Henschel, Stephen Simms
Computing Infrastructure

Computing Infrastructure
Storage.

Storage.
The Linux Storage People Simple Fast Massively Scalable Network Storage Coraid EtherDrive ® Storage.

The Linux Storage People Simple Fast Massively Scalable Network Storage Coraid EtherDrive ® Storage.
Chapter 1: Introduction to Scaling Networks

Chapter 1: Introduction to Scaling Networks

Similar presentations

Ads by Google