E. Matias and S. Lomas For the Project Team BigBANGWIDTH CLS IBM University of Alberta University of Western Ontario eScience – The CLS Case Study

Agenda What is eScience The Canadian Light Source Facility Traditional User Access Model CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

What is eScience? Providing a means for scientists to do better science, faster and at lower cost using advanced networking. In addition to CLS remote access are there other examples? –SETI –Radio Telescopes –Neptune –Grid Computing….

Examples…. Neptune University of Victoria Seafloor observatory Remote Telemetry System (800km ring) Joint Canada/US Project

Examples …. Grid Computing What is a grid –Clusters of clusters linked together –Generally geographically distributed Some examples: –WestGrid (covering major universities in western Canada is an example) –CERN - LHC (27km accelerator) is an example of an experimental setup where no one county has sufficient processing capabilities

Examples …. LHC –LHC at CERN is an example of an grid application where no one county has sufficient processing capabilities –15 million gigabytes of data per year –In 2006 LHC Tier 1 Grid was tested –TRIUMF is a Canadian Tier 1 Centre for LHC Experiments Courtesy TRIUMF

Examples….. WestGrid WestGrid and WestGrid II $50M Project Diverse cluster of machines and resources for high-performance computing Example Resources: 1680 processor IBM eServer Blade Centre 256 processor HP XC Cluster 144 processor HP AlphaServer 20 – 4 processor HP ES40s 256 processor SGI Origin Two 64 processor IBM p595 Cray XD1 system 56 processor IBM JS20 30 Terabyte disk storage 360 Terabyte tape storage Partners Institutions: –University of Victoria –University of British Columbia –Simon Fraser University –University of Northern British Columbia –TRIUMF –University of Lethbridge –The Banff Centre –University of Calgary –University of Alberta –Athabasca University –University of Saskatchewan –University of Regina –University of Manitoba –University of Winnipeg –Brandon University Industry Partners: –HP, IBM and SGI

Agenda What is eScience The Canadian Light Source Facility Traditional User Access Model CLS Remote Access Project User Configuration Light Paths Lessons Learned Conclusion

Canada’s Synchrotron Facility Located on the University of Saskatchewan Campus One of the largest national science projects in a generation Accelerator Complex Consists of: –Linac Accelerator (200 MeV) –Booster (200 MeV to 2.9 GeV) Ring –Storage Ring ( m GeV) –Two diagnostic beamlines Seven beamlines in operation or commissioning Six beamlines under construction Six beamlines being proposed (not-funded)

Machine Layout

Beamlines Operate in parallel harnessing synchrotron light generated by the circulating electron beam Each beamline is tailored to a class of experiments Beam-teams of users provide scientific direction Average beamline contains 50 to 150 motors Data rates are different from beamline to beamline

Operate in parallel harnessing synchrotron light generated by the circulating electron beam Each beamline is tailored to a class of experiments Beam-teams of users provide scientific direction Average beamline contains 50 to 150 motors Data rates are different from beamline to beamline CMCF 2 Beamline

Applications: Proteins & Disease

Applications: Medical Studies

Applications: Mining & Petroleum

Applications: Better Satellites, Planes and Cars

Applications: Nano-Technology

Agenda What is eScience The Canadian Light Source Facility Traditional User Access Model CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

Traditional User Access Model Academic Users –The user applies for beam-time once every six months –Peer review (Independent Committee) –Safety review (CLS) –Technical feasibility review (CLS) –Time scheduled in 8 hour slices usually multiple shifts Commercial Users –Straight fee-for-service arrangement without peer review A good target for web based automation…..

Automation? This sounds like an electronic document management problem? Yes, it is. Good target for outsourcing. The selected platform: –Cronus Technologies “C-Factor” and –Oracle. Hosted off-site by Cronus Managed by our User Services Office

Next Project? This time gain the flexibility of SOA Look at developing the infrastructure for remote access and to interweave different users. We have three beamlines that by their nature are good targets for remote access.

Agenda What is eScience The Canadian Light Source Facility Traditional User Access Model CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

Project Terms of Reference Major project funding from CANARIE with additional funding from: –IBM –Canadian Light Source –Bigbangwidth –University of Western Ontario UP & UML software engineering approach All custom developed software will be open source Project duration 14 months Project is completed before the target beamlines –Therefore concentration is on reusable components Service Oriented Architectures and Web Services

Project Delivery Team CANARIE Funded Project Canadian Light Source –Dionisio Medrano (System Analyst)* –Daron Chabot (System Analyst)* –Jason Chan (Intern)* –Elder Matias (Project Leader/Manager) –Michel Fodje (CMCF Beamline Scientist) –Renfei Feng (VESPERS Beamline Scientist) –Jason Cyrenne (Networking) –Bob Harvey (Networking) –Russ Berg (EPICS/CMCF) IBM Canada –Chris Armstrong (System Architect)* –John Haley (System Analyst/Architect)* University of Western Ontario –Marina Fuller (Requirements and Testing)* –Stewart McIntyre (User Champion VESPERS) –Gary Good (System Support) Alberta Synchrotron Institute –Ernst Bergman (User Champion CMCF) Big Bangwidth –Stuart Lomas (Networking) –Steve Hyatt (UCLP WebServices Software) * Full Time

High Level Functional Requirements (Long Term)

Component Overview Challenge: –Support dynamic reconfiguration –Support on-line changes in a 24/7 environment –Provide flexibility and ease in reconfiguring the environment –Separation of meta data from presentation Solution –XML based configuration information instead of hard-coding

Component Overview Challenge: –Web-standards are still immature, –Must have a real-time feel to the user –Diverse client hardware/software independently selected at each university Solution –Thin-client browser (Java Script) –AJAX used to provide real-time like interface with Spring Framework –Identified a single supported browser (Fire-fox) all others at users own risk

Component Overview Challenge: –Requires secure data transfer –Real-time performance – Guaranteed Quality of Service –Users located at major research Universities, Institutes in Canada and Australia Solution –LightPath and LighPath Accelerator Technology –CANet4 with International connections

Component Overview Challenge: –Robust Reliable –User performance requirements are unknown –Support on-line changes, since the Facility Operates 24/7 with limited outage periods –Common interface presented to the user Solution –Websphere Hosted –Provides Services for Managing Users and Presentation of Data to the User –Spring Framework and Custom Java Classes

Component Overview Challenge: –The “glue” that ties things together –Ability to Interface to both Internal and External Services –Reliable, flexible, ability to deal with services connecting and disconnecting gracefully Solution –Provides internal and external services to communicate with other systems, analysis codes etc. –Web-services for diverse and distributed services

Component Overview Challenge: –Understand where the system fails –Achieve performance objectives –Allocate resources to performance only where there is a clear measurable benefit Solution –Build in some basic auditing to determine bottlenecks and trace faults

Component Overview Challenge: –Several Terra bytes of data –Several megabytes per experiment –Provide good performance Solution –Storage Area Network (SAN) –Light-paths to permit the rapid transfer of data to the user home institution

Component Overview Challenge: –Control diverse hardware –Implement motion control and data acquisition algorithms –Support both local and remote access Solution –EPICS (framework extensively used at synchrotrons around the world) –Integrate vendor and other libraries as needed Legacy System?

EPICS Overview Application Software EDM Matlab Spec IDL Labview Network Device Drivers Channel Access Server Network Channel Access Network Hardwired Driver VME, Firewire Serial Drive VME, Firewire Scanner State Machine Engine Hardware Controllers Metadata Fault Logging EPICS Provides: Supervisory Control and Data Acquisition (SCADA) or (Distributed Control System) DCS functions Originally written by the US Department of Energy in 1980s Widely used at large scale science facilities around the world Interface Remote Access

EPICS Profibus TCP/IP Siemens S7/300 PLC Modbus TCP/IP GPIB RS-232 Channel Access Protocol IOC State Machine Engine CA Single Board Computer CA Operator Workstation User Applications Touch Panels CA Telemecanique Momentum PLC VME IOC CA

Logical Component Diagram

Layout

Remote Beamline Access Prototype Architecture

Simplified Business Model for Prototype

Application Screen Shots: Meta Data

Application Screen Shot: Experiment

Agenda The Canadian Light Source Facility Traditional User Access Model What is eScience CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

Network Architecture request control Control (UCLP) CANARIE and ORAN Lightpath Network Domain Manager LPA816e VESPERS Beamline Computer CMCF Beamline Computer Canadian Light Source Saskatoon Lightpath Accelerator network. This solution allows any one research workstation, at right, to connect to one of the two beamline computers at the CLS. This solution can be extended to allow multiple connections through each LPA816e Lightpath Director. October, 2006

BigBangwidth Lightpath Accelerator BigBangwidth Lightpath Accelerator extends and automates Advanced Networking Lightpaths LPA816e lightpath cross-connect  Software and hardware system  Extends Advanced Network Lightpaths across the LAN “Last Mile Solution” for lightpaths  Interfaces to CANARIE UCLP Provides an automated front end, no IT  Also works without UCLP, in LAN or WAN

This graph shows the traffic each second as a stream is recognized and moved from the LAN (blue) to a lightpath (red). Lightpath Accelerator Operation

Lightpath Accelerator controls a software virtual cross- connect that commands UCLP. Integrating LPA with UCLP In effect, CA*Net4 is treated as a single lightpath cross-connect real device virtual device domain manager process UCLP commands

Agenda The Canadian Light Source Facility Traditional User Access Model What is eScience CLS Remote Access Project User Configurable Light Paths Lessons Learned Conclusion

Lessons Learned The technology is powerful but very complex Web-services and the associated technology are not mature this was a problem in this project –Standards are a moving target When evaluating frameworks, the products were evolving faster than the selection process Engaged consultants and collaborators with previous experience (IBM, and BigBangwidth in our case) Concepts are not completely new

The End Thank you.