1. Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National Agricultural Research Center Tsukuba Japan

2. Contents Concepts –Remote Procedure Calls (RPC) –Brokers Using a broker –Metadata requests –Data requests –Issues –Linking your own data SOAP New brokers and future enhancements

3. Overall Goal: To help you make “portable” software applications (DSS).

4. Current Situation: Web-based but… Applications closely coupled to databases Limited range of applications at each site Duplicated development at each site

5. Soils Ag and NR Grid – a “Hotel Room” for Agricultural and Natural Resource DSS Room Weather Elevation Maps Connectors for local data DSS User interface in national language

6. New Zealand Soils Weather Elevation Map s

7. 地 気象 地面の高 さ 地図 日本

8. 中国 土壤 气象 海拔 交通图

9. Goal of Ag and NR Grid: To provide this “hotel room” in your country, so you can use models developed by others (and they can use your models).

10. Distributed Computing Concepts Remote Procedure Calls Brokers to handle heterogeneous databases

11. Remote Procedure Calls (RPC) RPCs let a program execute a procedure on a remote computer as if it were local.

12. Remote Procedure Call My Computer remote=findRemote() A=5 B=6 C=remote.add(A,B) Print C... Remote Computer procedure add(x,y) return x+y }...

13. Remote Procedure Calls (RPC) RPCs let a program execute a procedure on a remote computer as if it were local. Newer RPC mechanisms let programs send and receive data objects. RPC mechanisms: CORBA,.NET, Java RMI, SOAP... RMI – only Java programs can use SOAP – used by other languages as well

14. RPC vs Local Calls Latency (delay) –RPCs much slower than local calls –difficult to predict how long call will take May fail –network problems –problems with other computer (“server”) Security issues –firewalls...

15. But... Can distribute processing and data over multiple machines Great when multiple users must share the same dynamic data item or communicate with each other.

16. Mediated Architecture – “Brokers” Broker Domain Databases (Heterogeneous) Applications Consistent data access (using RPC mechanism)

17. MetBroker Korea Japan New Zealand Weather Databases Data Retrieval Applications South Africa USA Data Retrieval Servlet Weather Risk Crop Growth Disease Risk Web Browsers

18. Contents Concepts –Remote Procedure Calls (RPC) –Brokers Using a broker –Metadata requests –Data requests –Issues –Linking your own data SOAP New brokers and future enhancements

19. Data and Metadata Requests Metadata request Metadata result Broker Database (various) Application Standard request DB-specific query DB-specific result Standard result Metadata Data Interaction between applications and brokers uses objects – data packages

20. Example - MetBroker Services Weather Data –Single station –All stations in a region Metadata (data about data) –Database details (coverage of globe, login req.) –Regions within databases –Station details (latitude, longitude, start date, end date, elements recorded, resolution...)

21. Elements of a Station Data Request Weather station Required weather elements Time interval Username/password Resolution Allow summarizing

22. Send Request to MetBroker and Receive Result Data Result Tsuchiura Operated 1976-present 1999/1/3 06:00-1999/2/1 00:00 10.1, -0.2, 0 12.6, 3.8, 0 11.3, -1.5, 0.3... To MetBroker  Data Request amedas 40341 rain, air temperature 1999/1/1-1999/2/1 daily OK to summarize

23. Query Results - Single Station Station & query details rainair temp.solar rad. Station details and data sequences are encapsulated in a single result object

24. Broker Temperature Rainfall Radiation Client Spatial Request

25. Wakayama Rainfall DB AMeDAS Daily max/min temperatures & rainfall for all stations in region Spatial Request Background map dynamically retrieved from http://www.mapblast.com

26. Query results - multiple stations

27. Using Brokers – Implementation Issues

28. RMI Blocked by Firewalls Client Application Java Broker Server Machine Client Application Java Wrapper Servlet HTTP RMI Client-side firewalls

29. Applet Using Multiple Brokers Spatial Risk Applet MetBroker DEMBroker ChizuBroker Background maps Elevation data Meteorological data Resource Server Application text

30. Without Multiple Threads - Sequential MetBroker DEMBroker ChizuBroker Background maps Elevation data Meteorological data Resource Server Application text 12 seconds Spatial Risk Applet

31. With Multiple Threads – Parallel MetBroker DEMBroker ChizuBroker Background maps Elevation data Meteorological data Resource Server Application text 5 seconds Spatial Risk Applet

32. JavaBeans for Java GUI programs For rapid GUI development in Java can use existing MetBeans Transparently handle firewalls, latency... Making a simple data retrieval application takes about 30 minutes Used in many MetBroker demo applications

33. Easy Development of New Applications

34. Databases Linked Using “Drivers” Databases Applications Broker

35. Linking a Database to a Broker Need to write one Java class (a “driver”) with two or three main methods Typically use Java Database Connectivity (JDBC) drivers to query relational databases Can reuse existing classes for Oracle, Postgres, and MySQL JDBC. Drivers can read data from files via HTTP, or parse HTML tables. “Test harness” available for driver development

36. How much work? Add a new database to MetBroker –roughly working - 3 days –tested more thoroughly – 1 week –(assuming that database is readily accessible) Make a MetBroker applet –working prototype in one day –(assuming basic science is “done”)

37. Developer Resources Demos, documentation, downloads, pictures –http://www.agmodel.net Mailing lists – subscribe at web site Brokers are open source under the GNU Lesser General Public License. –Can be used in commercial applications –Any improvements must be released under same license

38. Linking your data to the grid lets you use grid tools developed by others. It doesn’t mean you must share your data with everyone (password protection). The key to interoperability is the interface Reminders

39. Contents Concepts –Remote Procedure Calls (RPC) –Brokers Using a broker –Metadata requests –Data requests –Issues –Linking your own data SOAP New brokers and future enhancements

40. SOAP An XML-based RPC mechanism Independent of programming language SOAP toolkits for C++, Visual Basic, Delphi, Java, ASP, PHP, Perl, Python… Firewall-friendly Basis of Microsoft.NET architecture A SOAP Web service is described in standard format using a.WSDL file (Web Services Description Language)

41. Visual Basic client application Application code MS SOAP Toolkit VB function calls VB objects MetSOAP SOAP over HTTP MetBroker Delphi client application Application code Borland SOAP Toolkit Delphi function calls Delphi objects SOAP over HTTP RMI Example - MetSOAP

42. SOAP Summary Positives: –Opens brokers to a range of programming languages –Already the standard for Web-based RPC –Avoids most firewall issues Negatives –Can’t reuse Java classes - must “mirror” in each language –Interoperability is still imperfect (but improving fast) –Inefficient for large data sets

43. Typical Array in SOAP... 23.43 21.79 23.3 21.11 19.49 18.48 16.52 14.73...

44. Model Deployment Options - Summary Java applet –flexible, responsive once loaded but requires plug-in Java servlet –simple browsers, rapid load, but clunky interface SOAP –direct access for Delphi, VB… Existing models (FORTAN,C,BASIC) –Use simple Java app. to retrieve data and write to a file

45. Contents Concepts –Brokers –Remote Procedure Calls (RPC) Using a broker –Metadata requests –Data requests –Issues –Linking your own data SOAP New brokers and future enhancements

46. New Brokers DEM Broker - Digital Elevation Model data –Japan 50m resolution –World 1km resolution ResourceServer –Internet-based service for software localization ChizuBroker – Web map services –www.mapfan.com (Japan) –www.mapzone.co.nz (New Zealand) SoilBroker – soils data

47. Future enhancements Broker-side interpolation (using DEMBroker) –temporal –spatial

48. Server Temperature Rainfall Radiation Client “Virtual station” Broker-side interpolation at single site

49. Future enhancements Broker-side interpolation (using DEMBroker) –temporal –spatial Other brokers –Place names –Crops and crop models (prototyped) –Agrichemicals

50. Final Points The Grid provides a stable, powerful foundation for Ag and NR software You can now develop grid applications in your favourite programming language. NARO is keen to help you to both develop new applications and link new databases. Database owners around the world will be keen to evaluate your applications.

51. Thank you