Status of IU’s E10000 and recent activities with Sun Sun NDA Visit June Craig Stewart, Ph.D. Please cite as: Stewart, C.A. Status of IU’s E10000 and recent activities with Sun. Presented at Indiana University Wrubel Computing Center, 14 June 2000, Bloomington, IN. Available from

IU's Goal “To be a leader in absolute terms in information technology” - IU President Myles Brand, 1996

IU in a nutshell Founded in 1820 $2B Annual Budget 8 campuses >90,000 students 3,900 faculty 878 degree programs; >1,000 majors; > 60 programs ranked within top 20 of their type nationally University highly regarded as research and teaching institution

in a nutshell Academic programs in IT through computer science, library and information sciences, engineering and technology, and most notably through new School of Informatics CIO and Vice President for Information Technology Michael A. McRobbie –Office of the Vice President for Information Technology and University Information Technology Services (UITS): ~$70M annual budget –Technology services offered university-wide – UITS comprises ~500 FTE staff, organized into four technology divisions (Teaching & Learning Information Technology,Telecommunications, University Information Systems, Research and Academic Computing) and crosscutting units (e.g. finance, planning, HR)

IU IT Strategic Plan 10 recommendations, 68 Actions covering all campuses and all IT areas Completed in May 1998, approved and implementation commenced January 1999 Total cost of implementation: $205M over 5 years A unique charter for Information Technology at a large university that sets the strategic course for the next five years

Two new initiatives IPCRES: the Indiana Pervasive Computing Research Initiative –IPCRES Laboratories (3 at IUPUI; 3 at IUB) –Established by $30M grant from Lilly Endowment School of Informatics –One of the first such schools at a major University –The School's mission is to educate students broadly in the technical, psychological and social aspects of information technology – Includes the already-successful New Media program – Offers B.S. in Informatics, graduate programs in Chemical Informatics, Bioinformatics, HCI, Health Informatics

Other IT highlights Network operations center for Abilene Internet2 network Lead US Institution on TransPAC High-Speed connection to APAN Major reengineering of University Information Systems underway Major Distributed Education Initiative underway Recognized leader in use of IT in education Recognized leader in support (IU leveraged support model).

Advanced Computation Accomplishments Three HPC systems: –IBM RS/6000 SP processors, 147 GFLOPS (largest supercomputer in Indiana) –64-processor Sun E10000 –64-processor Parallel PC Cluster (Compaq) Prominently involved in Grid projects: CCAT, CCA Forum, Globus, iGrid, GOIN99 Expanded support for High Performance Computing

Sun E10000 Purchase negotiated as competitive upgrade to IU’s SGI Origin2000, and part of a purchase that also included equipment to completely refurbish IU’s messaging infrastructure Announced by Sun CEO Scott McNealy at IU Business Conference on 2/23/2000, simultaneous with announcement of Sun’s partnership with IU as a Sun Center of Excellence.

Sun E10000 timeline 3/27 Hardware arrives 3/30 System turned over to IU 3/30-4/16 Software installation & testing 4/17 first availability for users (in hardy user mode) 4/24-4/27 Training by Sun 5/24-5/25 Training by KAI representatives 6/15 our 64P O2000 to be decommissioned

Software summary HPC 3.1, Workshop 6, PBS 2.3 beta KAI suite: compilers (currently recommended as default), KAP/Pro Toolset IMSL Fortran & libraries, LAPACK, NAG Pallas tools Radiance The usual suite of GNU Unix tools

Performance comparisons

Performance comparisons, cont'd.

Successes with E10000 System stood up in good order and extremely reliable (only 2 hardware failures thus far) Excellent performance for some large codes managed by sophisticated users (e.g. does some hard problems well) Good response by Sun in regards to problem with PBS and Sun MPI libraries (although significant issues remain). Papakhian, Burks (IU) and Duncan (Sun) plan a paper for SuPerG. IU’s E10000 is 447th on current Top500 list

Problems to date Compilers (OpenMP, MPI, general robustness) 64-bit performance questions Performance obtained by naïve users Lack of control of /tmp’s use of swap DFS clients for Solaris 7 just now available

Concerns for future Sun security notification policy (we need to know about exposures when Sun knows, not when a fix is available) Effects of purchase of KAI by Intel (in our view, this places some added pressure on Sun to increase the rate of improvement in Sun compilers) Fixes for current software issues Availability of DFS clients MPI/Compiler issues PBS support Maintenance

Detail on Software issues Most of what we need has been promised in previous nondisclosures or other communications. Current issues include: – /tmp’s use of swap prevents us from using PFS –Need to get current versions in synch We cannot implement a new OS version (e.g. Solaris 8) until a DFS client is available for it. Compilers –OpenMP and Interval arithmetic for Fortran & C –Better performance, robustness, responsiveness to bugs PBS –MPI/PBS issues not yet completely resolved –PBS support should be a high priority; Sun seems to be moving in the right direction on this

Self-Maintenance Issues IU has had successful self-maintenance agreements with three different vendors: Intel, IBM, SGI. Sun hpc.com model for ‘capacity on demand’ may contain the key elements of the required business model. At Sun’s HPC Government/Higher Ed Summit several Sun leaders emphasized their understanding that the research HPC market is different than the commercial market. IU’s success with self-maintenance makes it a good test case for Sun. This is IU’s #1 priority issue in regards to HPC as we move forward in our relationship with Sun.

Sun/IU relationship history & ongoing events IU site licenses: Solaris, related products, Star Office (obtained prior to Star Division’s acquisition by Sun) IU participation in Sun HPC Consortium Purchase of E3500 as Web data publishing/exchange server Flybase (

Sun/IU current & pending activities Sarpa visit -- two proposals pending –Leake: Task-Based, Proactive Retrieval and Communication (IU, NW, Sun) –Meglicki: Remote HPSS mover (IU, NCSA, Sun) SunRay Pilot Key labs programs -- UITS Student technology centers to be refurbished with Sun WSs, 7 other IU orders AEG application: WeBWorKs *Just approved* Finalization of Sun TREC:HPC-Grid CE agreement (Cheminformatics?) pending

In Summary So far, so good. IU/Sun partnership in HPC area off to an excellent start IU/Sun partnership is being broadened and deepened successfully –Relationship in HPC is strengthening – Building researcher-to-researcher relationships –IT relationship being broadened in several directions, particularly in regards to teaching, learning, and student use

References IU IT Strategic Plan: IPCRES: IU University Information Technology Services:

Except where otherwise noted, the contents of this presentation are © by the Trustees of Indiana University. This content is released under the Creative Commons Attribution 3.0 Unported license ( This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.