1. E. Matias and S. Lomas For the Project Team BigBANGWIDTH CLS IBM University of Alberta University of Western Ontario eScience – The CLS Case Study

2. Agenda What is eScience The Canadian Light Source Facility Traditional User Access Model CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

3. What is eScience? Providing a means for scientists to do better science, faster and at lower cost using advanced networking. In addition to CLS remote access are there other examples? –SETI –Radio Telescopes –Neptune –Grid Computing….

4. Examples…. Neptune Neptune @ University of Victoria Seafloor observatory Remote Telemetry System (800km ring) Joint Canada/US Project

5. Examples …. Grid Computing What is a grid –Clusters of clusters linked together –Generally geographically distributed Some examples: –WestGrid (covering major universities in western Canada is an example) –CERN - LHC (27km accelerator) is an example of an experimental setup where no one county has sufficient processing capabilities

6. Examples …. LHC –LHC at CERN is an example of an grid application where no one county has sufficient processing capabilities –15 million gigabytes of data per year –In 2006 LHC Tier 1 Grid was tested –TRIUMF is a Canadian Tier 1 Centre for LHC Experiments Courtesy TRIUMF

7. Examples….. WestGrid WestGrid and WestGrid II $50M Project Diverse cluster of machines and resources for high-performance computing Example Resources: 1680 processor IBM eServer Blade Centre 256 processor HP XC Cluster 144 processor HP AlphaServer 20 – 4 processor HP ES40s 256 processor SGI Origin Two 64 processor IBM p595 Cray XD1 system 56 processor IBM JS20 30 Terabyte disk storage 360 Terabyte tape storage Partners Institutions: –University of Victoria –University of British Columbia –Simon Fraser University –University of Northern British Columbia –TRIUMF –University of Lethbridge –The Banff Centre –University of Calgary –University of Alberta –Athabasca University –University of Saskatchewan –University of Regina –University of Manitoba –University of Winnipeg –Brandon University Industry Partners: –HP, IBM and SGI

8. Agenda What is eScience The Canadian Light Source Facility Traditional User Access Model CLS Remote Access Project User Configuration Light Paths Lessons Learned Conclusion

9. Canada’s Synchrotron Facility Located on the University of Saskatchewan Campus One of the largest national science projects in a generation Accelerator Complex Consists of: –Linac Accelerator (200 MeV) –Booster (200 MeV to 2.9 GeV) Ring –Storage Ring ( 170.88 m - 2.9 GeV) –Two diagnostic beamlines Seven beamlines in operation or commissioning Six beamlines under construction Six beamlines being proposed (not-funded)

10. Machine Layout

11. Beamlines Operate in parallel harnessing synchrotron light generated by the circulating electron beam Each beamline is tailored to a class of experiments Beam-teams of users provide scientific direction Average beamline contains 50 to 150 motors Data rates are different from beamline to beamline

12. Operate in parallel harnessing synchrotron light generated by the circulating electron beam Each beamline is tailored to a class of experiments Beam-teams of users provide scientific direction Average beamline contains 50 to 150 motors Data rates are different from beamline to beamline CMCF 2 Beamline

13. Applications: Proteins & Disease

14. Applications: Medical Studies

15. Applications: Mining & Petroleum

16. Applications: Better Satellites, Planes and Cars

17. Applications: Nano-Technology

18. Agenda What is eScience The Canadian Light Source Facility Traditional User Access Model CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

19. Traditional User Access Model Academic Users –The user applies for beam-time once every six months –Peer review (Independent Committee) –Safety review (CLS) –Technical feasibility review (CLS) –Time scheduled in 8 hour slices usually multiple shifts Commercial Users –Straight fee-for-service arrangement without peer review A good target for web based automation…..

20. Automation? This sounds like an electronic document management problem? Yes, it is. Good target for outsourcing. The selected platform: –Cronus Technologies “C-Factor” and –Oracle. Hosted off-site by Cronus Managed by our User Services Office

21. Next Project? This time gain the flexibility of SOA Look at developing the infrastructure for remote access and to interweave different users. We have three beamlines that by their nature are good targets for remote access.

22. Agenda What is eScience The Canadian Light Source Facility Traditional User Access Model CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

23. Project Terms of Reference Major project funding from CANARIE with additional funding from: –IBM –Canadian Light Source –Bigbangwidth –University of Western Ontario UP & UML software engineering approach All custom developed software will be open source Project duration 14 months Project is completed before the target beamlines –Therefore concentration is on reusable components Service Oriented Architectures and Web Services

24. Project Delivery Team CANARIE Funded Project Canadian Light Source –Dionisio Medrano (System Analyst)* –Daron Chabot (System Analyst)* –Jason Chan (Intern)* –Elder Matias (Project Leader/Manager) –Michel Fodje (CMCF Beamline Scientist) –Renfei Feng (VESPERS Beamline Scientist) –Jason Cyrenne (Networking) –Bob Harvey (Networking) –Russ Berg (EPICS/CMCF) IBM Canada –Chris Armstrong (System Architect)* –John Haley (System Analyst/Architect)* University of Western Ontario –Marina Fuller (Requirements and Testing)* –Stewart McIntyre (User Champion VESPERS) –Gary Good (System Support) Alberta Synchrotron Institute –Ernst Bergman (User Champion CMCF) Big Bangwidth –Stuart Lomas (Networking) –Steve Hyatt (UCLP WebServices Software) * Full Time

25. High Level Functional Requirements (Long Term)

26. Component Overview Challenge: –Support dynamic reconfiguration –Support on-line changes in a 24/7 environment –Provide flexibility and ease in reconfiguring the environment –Separation of meta data from presentation Solution –XML based configuration information instead of hard-coding

27. Component Overview Challenge: –Web-standards are still immature, –Must have a real-time feel to the user –Diverse client hardware/software independently selected at each university Solution –Thin-client browser (Java Script) –AJAX used to provide real-time like interface with Spring Framework –Identified a single supported browser (Fire-fox) all others at users own risk

28. Component Overview Challenge: –Requires secure data transfer –Real-time performance – Guaranteed Quality of Service –Users located at major research Universities, Institutes in Canada and Australia Solution –LightPath and LighPath Accelerator Technology –CANet4 with International connections

29. Component Overview Challenge: –Robust Reliable –User performance requirements are unknown –Support on-line changes, since the Facility Operates 24/7 with limited outage periods –Common interface presented to the user Solution –Websphere Hosted –Provides Services for Managing Users and Presentation of Data to the User –Spring Framework and Custom Java Classes

30. Component Overview Challenge: –The “glue” that ties things together –Ability to Interface to both Internal and External Services –Reliable, flexible, ability to deal with services connecting and disconnecting gracefully Solution –Provides internal and external services to communicate with other systems, analysis codes etc. –Web-services for diverse and distributed services

31. Component Overview Challenge: –Understand where the system fails –Achieve performance objectives –Allocate resources to performance only where there is a clear measurable benefit Solution –Build in some basic auditing to determine bottlenecks and trace faults

32. Component Overview Challenge: –Several Terra bytes of data –Several megabytes per experiment –Provide good performance Solution –Storage Area Network (SAN) –Light-paths to permit the rapid transfer of data to the user home institution

33. Component Overview Challenge: –Control diverse hardware –Implement motion control and data acquisition algorithms –Support both local and remote access Solution –EPICS (framework extensively used at synchrotrons around the world) –Integrate vendor and other libraries as needed Legacy System?

34. EPICS Overview Application Software EDM Matlab Spec IDL Labview Network Device Drivers Channel Access Server Network Channel Access Network Hardwired Driver VME, Firewire Serial Drive VME, Firewire Scanner State Machine Engine Hardware Controllers Metadata Fault Logging EPICS Provides: Supervisory Control and Data Acquisition (SCADA) or (Distributed Control System) DCS functions Originally written by the US Department of Energy in 1980s Widely used at large scale science facilities around the world Interface Remote Access

35. EPICS Profibus TCP/IP Siemens S7/300 PLC Modbus TCP/IP GPIB RS-232 Channel Access Protocol IOC State Machine Engine CA Single Board Computer CA Operator Workstation User Applications Touch Panels CA Telemecanique Momentum PLC VME IOC CA

36. Logical Component Diagram

37. Layout

38. Remote Beamline Access Prototype Architecture

39. Simplified Business Model for Prototype

40. Application Screen Shots: Meta Data

41. Application Screen Shot: Experiment

42. Agenda The Canadian Light Source Facility Traditional User Access Model What is eScience CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

43. Network Architecture request control Control (UCLP) CANARIE and ORAN Lightpath Network Domain Manager LPA816e VESPERS Beamline Computer CMCF Beamline Computer Canadian Light Source Saskatoon Lightpath Accelerator network. This solution allows any one research workstation, at right, to connect to one of the two beamline computers at the CLS. This solution can be extended to allow multiple connections through each LPA816e Lightpath Director. October, 2006

44. BigBangwidth Lightpath Accelerator BigBangwidth Lightpath Accelerator extends and automates Advanced Networking Lightpaths LPA816e lightpath cross-connect  Software and hardware system  Extends Advanced Network Lightpaths across the LAN “Last Mile Solution” for lightpaths  Interfaces to CANARIE UCLP Provides an automated front end, no IT  Also works without UCLP, in LAN or WAN

45. This graph shows the traffic each second as a stream is recognized and moved from the LAN (blue) to a lightpath (red). Lightpath Accelerator Operation

46. Lightpath Accelerator controls a software virtual cross- connect that commands UCLP. Integrating LPA with UCLP In effect, CA*Net4 is treated as a single lightpath cross-connect real device virtual device domain manager process UCLP commands

47. Agenda The Canadian Light Source Facility Traditional User Access Model What is eScience CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

48. Lessons Learned The technology is powerful but very complex Web-services and the associated technology are not mature this was a problem in this project –Standards are a moving target When evaluating frameworks, the products were evolving faster than the selection process Engaged consultants and collaborators with previous experience (IBM, and BigBangwidth in our case) Concepts are not completely new

49. The End Thank you.