Download presentation
Presentation is loading. Please wait.
Published byEzra Hart Modified over 9 years ago
1
Using Grid Computing David Groep, NIKHEF 2002-07-15
2
Physics @ CERN LHC particle accellerator operational in 2007 5-10 Petabyte per year 150 countries > 10000 Users lifetime ~ 20 years level 1 - special hardware 40 MHz (40 TB/sec) level 2 - embedded level 3 - PCs 75 KHz (75 GB/sec) 5 KHz (5 GB/sec) 100 Hz (100 MB/sec) data recording & offline analysis The Grid, But Why?
3
CPU & Data Requirements Estimated CPU Capacity at CERN 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 5,000 1998199920002001200220032004200520062007200820092010 year K SI95 Moore’s law – some measure of the capacity technology advances provide for a constant number of processors or investment Jan 2000: 3.5K SI95 LHC experiments Other experiments < 50% of the main analysis capacity will be at CERN Estimated CPU capacity required at CERN
4
More Reasons Why ENVISAT 3500 MEuro programme cost 10 instruments on board 10 instruments on board 200 Mbps data rate to ground 200 Mbps data rate to ground 400 Tbytes data archived/year 400 Tbytes data archived/year ~100 `standard’ products ~100 `standard’ products 10+ dedicated facilities in Europe 10+ dedicated facilities in Europe ~700 approved science user projects ~700 approved science user projects 3500 MEuro programme cost 10 instruments on board 10 instruments on board 200 Mbps data rate to ground 200 Mbps data rate to ground 400 Tbytes data archived/year 400 Tbytes data archived/year ~100 `standard’ products ~100 `standard’ products 10+ dedicated facilities in Europe 10+ dedicated facilities in Europe ~700 approved science user projects ~700 approved science user projects
5
And More … For access to data –Large network bandwidth to access computing centers –Support of Data banks replicas (easier and faster mirroring) –Distributed data banks For interpretation of data –GRID enabled algorithms BLAST on distributed data banks, distributed data mining Bio-informatics
6
Common Ground Large amounts of data Distributed, ad-hoc user community Problems are distributable Need for resources grows faster than market Network grows faster than the application needs Willingness to share resources … … if security and integrity is guaranteed
7
The One-Liner Resource sharing and coordinated problem solving in dynamic multi-institutional virtual organisations
8
What is Grid computing? Dependable, consistent and pervasive access Combining resources from various organizations `Virtual Organizations’ – user-based view on Grid Technical challenges: –transparent decisions for the user –uniformity in access methods –secure & crack resistant –authentication, authorization, accounting (AAA) "a
9
Globus Project started 1997 de facto-standard Reference implementation of Gridforum standards Large community effort Basis of several projects, including EU-DataGrid Toolkit `bag-of-services' approach Successful test beds, with single sign-on, etc… Grid Middleware
10
In The Beginning Distributed Computing –synchronous processing High-Throughput Computing –asynchronous processing On-Demand Computing –dynamic resources Data-Intensive Computing –databases Collaborative Computing –science Ian Foster and Carl Kesselman, editors, “The Grid: Blueprint for a New Computing Infrastructure,” Morgan Kaufmann, 1999
11
Grid Architecture Applications Grid Services GRAM Grid Security Infrastructure (GSI) Grid Fabric CondorMPIPBSInternetLinux Application Toolkits DUROCMPICH-G2Condor-G GridFTPMDS SUN VLAM-G Make all resources talk standard protocols Promote interoperability of application toolkit, similar to interoperability of networks by Internet standards ReplicaSrv
12
OGSA: new directions Looks superficially like `web services’ Based on common standards: –WSDL –SOAP –UDDI Adds: –Transient services –State of distributed activities –Workflow, videoconf, distributed data analysis Management of service instances Grid Security Infrastructure
13
EU DataGrid HEP, EO, Bio Applications Grid Services GRAM Grid Fabric CondorPBSInternetLinux Application Toolkits MPICH-G2Condor-G GridFTPMDS SUN ResourceBroker Data Replicas Databases Mass storage Fabric&Network
14
Looking for Resources Resource Brokerage based on matchmaking (Condor) Information Services Mesh –Meta-computing directory –Replica Catalogues DataGrid http://marianne.in2p3.fr/http://marianne.in2p3.fr/
15
Submitting a Job
16
Locating a Replica Grid Data Mirror Package Moves data across sites Replicates both files and individual objects Catalogue used by Broker Replica Location Service (giggle) Read-only copies “owner” by the Replica Manager. http://cmsdoc.cern.ch/cms/grid
17
Sending Your Data Tape robots, disks, etc. share GridFTP interface Supports single-sign-on and confidentiality Optimize for high-speed >1Gbit/s networks In the future: automatic optimizations, bandwidth reservations, directory-enabled networking, …
18
Grid-enabled Databases? SpitFire uniform access to persistent storage on the Grid Multiple roles support Compatible with GSI (single sign-on) though CoG Uses standard stuff: JDBC, SOAP, XML Supports various back-end data bases http://hep-proj-spitfire.web.cern.ch/hep-proj-spitfire/
19
DataGrid Test Bed 1 DataGrid TB1: –14 countries –21 major sites –Growing rapidly Submitting Jobs: –Login only once, run everywhere –Cross administrative boundaries in a secure and trusted way –Mutual authorization
20
DutchGrid Platform Amsterdam Utrecht KNMI Delft Leiden Nijmegen Enschede DutchGrid: –Test bed coordination –PKI security Participation by NIKHEF: FOM, VU, UvA, Utrecht, Nijmegen KNMI, SARA AMOLF DAS-2 (ASCI): TUDelft, Leiden, VU, UvA, Utrecht Telematics Institute
21
And now for some Technical Details For Users
22
Resources Current startup-resources to be (ab)used: –NIKHEF: Several Globus test machines (try them now from your desk!) 50x2 CPU’s D0 cluster 2x10x2 (=40) CPU’s LHCb at NIKHEF(WCW) &VU 10x2 CPU’s Alice NIKHEF(WCW) ca. 4x2 CPU’s Alice Utrecht ca. 10x2 CPU’s D0 Nijmegen Lots of disk & dedicated 1.3TByte cache server –DAS-II: 200 dual-PIII’s systems & some disk (~2TByte) Spread over 5 locations (NIKHEF is one!) –SARA: tape robot (>200TByte), some clusters –More systems (NCF) to come this year …
23
Start using the grid All the necessary “client tools” are on all Linux and Solaris systems You just need: –Credentials/tokens for the Grid (see next slides) –Authorization to use resources (you get all NIKHEF resources by default) –Information on which resources to use effectively
24
Your Grid Credentials You will use resources across several domains –You may not care about security and authorization –But the remote site admin will ! All communications are authenticated using X.509 “Public Key” Certificates The technology used to secure credit card transactions on the web ( https://…… ) Uniquely binds name/affiliation to a digital token
25
Certification Authorities CA’s act as trusted third parties Remote sites trust the CA for a proper binding They will not do authentication again, so only authorization left. CA’s are highly valuable: crack one to impersonate others on the Grid (and abuse resources) Registration Authorities do in-person ID checks
26
CA’s in DataGrid 10 National CA’s (one per EU country) Each one has a detailed policy and practice statement NIKHEF operates the CA for DutchGrid See http://www.dutchgrid.nl/ca http://www.dutchgrid.nl/ca Get a “certificate” from the DutchGrid CA before you can start using the Grid It’s valuable, protect it with a pass phrase One cert valid for all DataGrid sites
27
The Proxy A `proxy certificate’ is a limited-lifetime delegation without a pass phrase to protect it Implements the single sign-on for Grid Valid for 12 hours (by default) Use it to: –Run your jobs –Get access to your data Get it, by running grid-proxy-init
28
Now see for yourself
29
Getting a Certificate Initialize your environment for the Grid Use the Globus local guide from http://www.dutchgrid.nl/Support/ http://www.dutchgrid.nl/Support/ Send the result to ca@nikhef.nl you will be contacted by phone Put the certificate (sent by mail) in your $HOME/.globus/usercert.pem Or use the Web at http://certificate.nikhef.nl/userhelp.html http://certificate.nikhef.nl/userhelp.html
30
Using the Grid Request authorization: grid.support@nikhef.nl grid.support@nikhef.nl Look what is out there using grid-info-search or http://marianne.in2p3.fr/datagrid/giis/giis-browse.html http://marianne.in2p3.fr/datagrid/giis/giis-browse.html Try some local hosts: –bilbo, kilogram, triangel kilogram:davidg:1009$ globus-job-run dommel.wins.uva.nl /usr/ucb/quota -v Disk quotas for random (uid 12xxx): Filesystem usage quota limit timeleft files quota limit timeleft /home/random 13067 1500000 2000000 0 0 0 kilogram:davidg:1010$ Start running your analysis/MC/other jobs
31
grid-proxy-init kilogram:davidg:1003$ grid-proxy-init Your identity: /O=dutchgrid/O=users/O=nikhef/CN=David Groep Enter GRID pass phrase for this identity: PassPhrase Creating proxy.................................... Done Your proxy is valid until Wed Sep 26 05:50:53 2001
32
GridFTP Universal high-performance file transfer Extends the FTP protocol with: –Single sign-on ( GSI, GSSAPI, RFC2228 ) –Parallel streams for speed-up –Striped access (ftp from multiple sites to be faster) Clients: gsincftp, globus-url-copy.
33
What’s Next? Some of the nice user-features to come: –Finding data files by characteristics (give me all golden decay’s) –Moving your job to where the data is –Automatic partitioning of jobs –Support true-interactive work –Better network utilisation (faster access to data) –……… If you are in the DataGrid project, ask your WP leader for authorization in TB1
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.