SensorGrid: A new Cyberinfrastructure Integrating Sensor Network and Grid Computing for e-Science Applications Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software Engineering The University of Melbourne, Australia WW Grid

SensorGrid: A new Cyberinfrastructure for Linking the Physical World with the Digital World Dr. Rajkumar Buyya Fellow of Grid Computing Grid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software Engineering The University of Melbourne, Australia gridbus.org/~raj/tut/gridbus.zip WW Grid

3 GRIDS Melbourne The youngest and one of the largest research labs in the CSSE Dept: 2 PostDocs 2 Research Programmers 7 RHD (6 PhD) students ~5 honours/masters projects Funding National and International organizations Australian Research Council Many industries (Sun, StorageTek, Microsoft, IBM) University-wide collaboration: Faculties of Science, Engineering, and Medicine Many national and international collaborations. Academics Industries Software: Our Grid middleware technologies are widely in academic and industrial users. Publication: My research team produces 20% of our Dept’s research output. EducationR & D + Community Services

4 Books at Glance: Co-authored/edited

5 Outline Introduction Technology Trends in Sensors and Grids Sensors and Grids Integration Proposal Grid Computing What is it?, architecture, and technologies NOSA (NICTA Open SensorWeb Architecture) Summary and Conclusion

6

7

8 Internet, Web, and Grid Effect TCP/IP HTML Mosaic XML PHASE 1. Packet Switching Networks 2. The Internet isBorn3. The World Wide Web 4.with XML 5. The Grid 1969: 4 US Universities linked to form ARPANET TCP/IP becomes core protocol HTML hypertext system created 1972: First program created Domain Name System created IETF created (1986) CERN launch World Wide Web 1976: Robert Metcalfe develops Ethernet NCSA launch Mosaic interface The 'Network Effect’ kicks in, and the web goes critical' Number of hosts (millions) Business $$$ ‘Web Services and Grid Effect’

Grid Information Service SensorWeb Service Sensor Network database R2R2 R3R3 RNRN R1R1 R4R4 R5R5 R6R6 Grid Resource Broker The customer can have access to the Grid Resource Broker, and then through Web Service, requests can be sent to obtain real sensor network data.

10 Traditional Devices, Sensors, and their Networks at Glance Traditional Devices (Computers and High End Resources) are: Powerful Connected to Power Grid – so we don’t worry too much about it power consumption Large Storage Space Good for archival and large-scale analysis Connected by High Bandwidth/Speed Network Sensors: Less powerful Scarcity of power (battery operated, or even self-power generated) Less Storage No good for archival Connected by Low Bandwidth/Speed Network But they can sense/smell a phenomena in the physical world.

11 It will be nice to marry them Both of them benefit: Grids: Get Eye to see the world (so that it can sense and assist the us) Sensors: Off load their processing, storage, archival, analysis, etc. requirements to the Grid. Sensors Grids = SensorGrid

12 Outline Introduction Technology Trends in Sensors and Grids Sensors and Grids Integration Proposal Grid Computing What is it?, architecture, and technologies NOSA (NICTA Open SensorWeb Architecture) Summary and Conclusion

13 Grid (computing) Paradigm: Cyberinfrastructure for sharing resources Inspired by Power Grid! * A service-oriented/utility computing paradigm that enables seamless sharing of geographically distributed, autonomous resources. * This was the original aim of building Internet although it ended up in giving birth to !

14 Grids have Emerged as Scalable Cyberinfrastructure for e-Science Applications Grid Resource Broker Resource Broker Application Grid Information Service Grid Resource Broker database R2R2 R3R3 RNRN R1R1 R4R4 R5R5 R6R6 Grid Information Service

15 Type of Services Modern Grids Offer Computational Services – CPU cycles NASA IPG, TeraGrid, I-Grid,… Data Services Data replication, management, secure access-- LHC Grid/Napster Application Services Access to remote software/libraries and license management—NetSolve Information Services Extraction and presentation of data with meaning Knowledge Services The way knowledge is acquired and managed— data mining. Utility Computing Services Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities. Computional Grid Data Grid ASP Grid Information Grid Knowledge Grid Utility Grid

16 Grid Capabilities Security Resource Allocation & Scheduling Data locality Network Management System Management Resource Discovery Uniform Access Computational Economy Application Construction

17 Some Grid Initiatives Worldwide Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch Brazil OurGrid, EasyGrid LNCC-Grid + many others China ChinaGrid – Education CNGrid - application Europe UK eScience EU Grids.. and many more... India I-Grid  Japan NAGERI Korea... N*Grid Singapore NGP USA Globus NASA IPG AccessGrid TeraGrid Cyberinfrasture and many more... Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft -.NET Oracle 10g Infosys – Business Grid StorageTek –Grid.. and many more Public Forums Global Grid Forum Australian Grid Forum Conferences: CCGrid Grid HPDC E-Science billion – 3 yrs 1 billion – 5 yrs 450million – 5 yrs 486million – 5 yrs 1.3 billion (Rs) 27 million 2? billion 120million – 5 yrs

18 The Gridbus Melbourne: Enable Leasing of ICT Services on Demand WWG World Wide Grid!  On Demand Utility Computing Gridbus Distributed Data

19

20 Gridbus Architecture Layer

21 On Demand Assembly of Services: Putting Them All Together Data Source (Instruments/dis tributed sources) Data Replicator (GDMP) ASP Catalogue Grid Info Service Grid Market Directory GSP (Accounting Service) Gridbus GridBank Data GSP (e.g., UofM) PE GSP (e.g., VPAC) PE GSP (e.g., IBM) CPU or PE Grid Service (GS) (Globus) Alchemi GS GTS Cluster Scheduler Grid Service Provider (GSP) (e.g., CERN) PE Cluster Scheduler Job 8 Grid Resource Broker 2 Visual Application Composer Application Code Explore data Results 97 Results+ Cost Info Bill 12 Data Catalogue

22 Alchemi:.NET-based Enterprise Grid Platform & Web Services Internet Alchemi Worker Agents Alchemi Manager Alchemi Users Web Services like Model General Purpose Dedicated/Non-dedicate workers Role-based Security.NET and Web Services C# Implementation GridThread and Job Model Programming Easy to setup and use Widely in use!

23 Some Users of Alchemi Tier TechnologiesTier Technologies, USA Large scale document processing using Alchemi framework CSIROCSIRO, Australia Natural Resource Modeling The Friedrich Miescher Institute (FMI) for Biomedical ResearchThe Friedrich Miescher Institute (FMI) for Biomedical Research, Switzerland Patterns of transcription factors in mammalian genes Satyam Computers Applied Research LaboratorySatyam Computers Applied Research Laboratory, India Micro-array data processing using Alchemi framework The University of Sao PauloThe University of Sao Paulo, Brazil The Alchemi Executor as a Windows Service stochastix GmbHstochastix GmbH, Germany Asynchronous Excel Tasks using ManagedXLL and Alchemi.Net Grid Computing framework. Many users in Universities: See next for an example.

24 Students' project gives old computers new life - 1/25/2005

The Gridbus Grid Service Broker for Data Grid Applications Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, ] And Extends its notion for Data and Service Grids

26 Gridbus Broker Architecture Grid Middleware Gridbus Client Gribus Client Grid Info Server Schedule Advisor Trading Manager Gridbus Farming Engine Record Keeper Grid Explorer GE GIS, NWS TM TS RM & TS Grid Dispatcher RM: Local Resource Manager, TS: Trade Server G G C U Globus enabled node. A L Alchemi enabled node. (Data Grid Scheduler) Data Catalog Data Node Unicore enabled node. $ $ $ App, T, $, Opt (Bag of Tasks Applications)

27 Gridbus Services for eScience applications Application Development Environment: XML-based language for composition of task farming (legacy) applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes that meet user QoS requirements. Hide L ow-Level Grid Middleware interfaces Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

28 Figure 3 : Logging into the portal. Drug Design Made Easy! Click Here for Demo

29 Sample Applications of Gridbus Broker Molecular Docking - WEHI Drug Discovery Brain Activity Analysis – Osaka University Neuroscience studies Natural Language Engineering – Melbourne NLP Indexing of newswire data High Energy Physics – School of Physics/Melbourne Belle experiment data analysis Finance - Portfolio Analysis – U. Comp. Madrid/Spain Investment risk analysis Astronomy – School of and QUT (Queensland University of Technology) Australian Virtual Observatory Spreadsheet Processing Microsoft Excel

30 Outline Introduction Technology Trends in Sensors and Grids Sensors and Grids Integration Proposal Grid Computing What is it?, architecture, and technologies NOSA (NICTA Open SensorWeb Architecture) Summary and Conclusion

31 Sun Proposal: For Integration of Sensors and High End Computers using Network Source: Anil Velluri, Sun

32 Our Proposal: SensorGrid - Integrated sensor network and grid architecture sensor and actuator networks query response command actuate grid actions sense

33 Sensor Web and NOSA Goal: To create Web-based sensor networks by exploiting web-connected sensors (flood gauges, air pollution monitors, mobile heart monitors, satellite- borne earth imaging monitors). This would make all sensors and repositories of sensor data discoverable, accessible, process-able and where applicable controllable via the WWW. OGC (Open Geospatial Consortium) describes five important encoding and service standards for a Sensor Web Implementation (see next slide).

34 What is NOSA? The NICTA Open SensorWeb Architecture (NOSA) project is developing a complete standards compliant platform and middleware for integration of sensor networks with emerging distributed computing platforms such as Grids. It confirms to Web Services standard defined by W3C (World-Wide Web) and SensorML (Sensor Model Language) standard defined by OpenGeospatial Consortium.

35 What are NOSA Benefits? This integration of sensor networks with Grid computing brings out dual benefits: (i) sensor networks can off-load heavy processing activities to the Grid and (ii) Grid-based sensor applications can provide advance services for smart-sensing by deploying scenario- specific operators at runtime. Fundamental services are provided by lower-level layer components whereas components at the higher-level layer provide tools for creation of applications and management of life-cycle of data captured through sensor networks.

36 NICTA Open SensorWeb Architecture (NOSA) Sensor Fabric Simulation Environment Application Services Layer Application Development Layer Applications Layer SensorWeb Fabric or Emulation SensorWeb Core Middleware SensorWeb User-Level Middleware SensorWeb Applications

37 OGC-Sensor Web Enablement Standard Specifications SensorML: XML encoding language for sensors. Used to discover, query and control Web-resident sensors. Observations & Measurements: The general models and an XML encoding for what a sensor observes or measures (The value returned by or derived from a sensor observation -e.g. quantity, count, boolean, category, ordered category, position-). Sensor Collection Service: A service to fetch observations from a sensor or constellation of sensors. Provides real time or archived observed values. Clients can also obtain information that describes the associated sensors and platforms. Sensor Planning Service: A service by which a client can determine collection feasibility for a desired set of collection requests for one or more mobile sensors/platforms, or the client may submit collection requests directly to these sensors/platforms. SPS enables sensor tasking, acquisition requests, processing and simulation requests, and registration for alert notification. Web Notification Service: A service by which a client may conduct a dialog with one or more other services. Provides a means for Sensor Planning Services to alert people, software, or other sensor systems of SPS results or alerts regarding phenomena of interest.

38 A sensor node includes a sensor and radio board. Then the sensor node could send data message through radio to Station which is the gateway connected with the PC. Here is the architecture of the interface connecting sensor network and the real PC.

39 Crossbow Wireless Sensor Network Kit and Testbed Setup in GRIDS Lab Base station MIB510CA Sensor MTS 300

40 SensorWeb Implementation

41 A SensorWeb Collection Client

42 Performance of collecting auto-sending and query data

43 Performance of collecting auto-sending and query data

44 Outline Introduction Technology Trends in Sensors and Grids Sensors and Grids Integration Proposal Grid Computing What is it?, architecture, and technologies NOSA (NICTA Open SensorWeb Architecture) Summary and Conclusion

45 Summary and Conclusion Sensors and Grids two major elements of emerging Cyberinfrastructure that support e*Applications (e- Science, e-Business, e-Health, e-Life). They support creation of smart office, house, and business environments. Current developments in Sensors and Grids is heavily driven by applications and both compliment and need each other. SensorGrid is just emerging and there are many opportunities available for creating many interesting applications in various domains.

46 References Rajkumar Buyya and Srikumar Venugopal, A Gentle Introduction to Grid Computing and Technologies, CSI Communications, pages 9- 19, Computer Society of India, Vol.29, No.1, July 2005.A Gentle Introduction to Grid Computing and Technologies Chen-Khong Tham and Rajkumar Buyya, SensorGrid: Integrating Sensor Networks and Grid Computing, CSI Communications, pages 24-29, Computer Society of India, Vol.29, No.1, July SensorGrid: Integrating Sensor Networks and Grid Computing

47 Any Questions ? Web -