Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National Agricultural Research Center Tsukuba Japan

Contents Concepts –Remote Procedure Calls (RPC) –Brokers Using a broker –Metadata requests –Data requests –Issues –Linking your own data SOAP New brokers and future enhancements

Overall Goal: To help you make “portable” software applications (DSS).

Current Situation: Web-based but… Applications closely coupled to databases Limited range of applications at each site Duplicated development at each site

Soils Ag and NR Grid – a “Hotel Room” for Agricultural and Natural Resource DSS Room Weather Elevation Maps Connectors for local data DSS User interface in national language

New Zealand Soils Weather Elevation Map s

地 気象 地面の高 さ 地図 日本

中国 土壤 气象 海拔 交通图

Goal of Ag and NR Grid: To provide this “hotel room” in your country, so you can use models developed by others (and they can use your models).

Distributed Computing Concepts Remote Procedure Calls Brokers to handle heterogeneous databases

Remote Procedure Calls (RPC) RPCs let a program execute a procedure on a remote computer as if it were local.

Remote Procedure Call My Computer remote=findRemote() A=5 B=6 C=remote.add(A,B) Print C... Remote Computer procedure add(x,y) return x+y }...

Remote Procedure Calls (RPC) RPCs let a program execute a procedure on a remote computer as if it were local. Newer RPC mechanisms let programs send and receive data objects. RPC mechanisms: CORBA,.NET, Java RMI, SOAP... RMI – only Java programs can use SOAP – used by other languages as well

RPC vs Local Calls Latency (delay) –RPCs much slower than local calls –difficult to predict how long call will take May fail –network problems –problems with other computer (“server”) Security issues –firewalls...

But... Can distribute processing and data over multiple machines Great when multiple users must share the same dynamic data item or communicate with each other.

Mediated Architecture – “Brokers” Broker Domain Databases (Heterogeneous) Applications Consistent data access (using RPC mechanism)

MetBroker Korea Japan New Zealand Weather Databases Data Retrieval Applications South Africa USA Data Retrieval Servlet Weather Risk Crop Growth Disease Risk Web Browsers

Contents Concepts –Remote Procedure Calls (RPC) –Brokers Using a broker –Metadata requests –Data requests –Issues –Linking your own data SOAP New brokers and future enhancements

Data and Metadata Requests Metadata request Metadata result Broker Database (various) Application Standard request DB-specific query DB-specific result Standard result Metadata Data Interaction between applications and brokers uses objects – data packages

Example - MetBroker Services Weather Data –Single station –All stations in a region Metadata (data about data) –Database details (coverage of globe, login req.) –Regions within databases –Station details (latitude, longitude, start date, end date, elements recorded, resolution...)

Elements of a Station Data Request Weather station Required weather elements Time interval Username/password Resolution Allow summarizing

Send Request to MetBroker and Receive Result Data Result Tsuchiura Operated 1976-present 1999/1/3 06: /2/1 00: , -0.2, , 3.8, , -1.5, To MetBroker  Data Request amedas rain, air temperature 1999/1/1-1999/2/1 daily OK to summarize

Query Results - Single Station Station & query details rainair temp.solar rad. Station details and data sequences are encapsulated in a single result object

Broker Temperature Rainfall Radiation Client Spatial Request

Wakayama Rainfall DB AMeDAS Daily max/min temperatures & rainfall for all stations in region Spatial Request Background map dynamically retrieved from

Query results - multiple stations

Using Brokers – Implementation Issues

RMI Blocked by Firewalls Client Application Java Broker Server Machine Client Application Java Wrapper Servlet HTTP RMI Client-side firewalls

Applet Using Multiple Brokers Spatial Risk Applet MetBroker DEMBroker ChizuBroker Background maps Elevation data Meteorological data Resource Server Application text

Without Multiple Threads - Sequential MetBroker DEMBroker ChizuBroker Background maps Elevation data Meteorological data Resource Server Application text 12 seconds Spatial Risk Applet

With Multiple Threads – Parallel MetBroker DEMBroker ChizuBroker Background maps Elevation data Meteorological data Resource Server Application text 5 seconds Spatial Risk Applet

JavaBeans for Java GUI programs For rapid GUI development in Java can use existing MetBeans Transparently handle firewalls, latency... Making a simple data retrieval application takes about 30 minutes Used in many MetBroker demo applications

Easy Development of New Applications

Databases Linked Using “Drivers” Databases Applications Broker

Linking a Database to a Broker Need to write one Java class (a “driver”) with two or three main methods Typically use Java Database Connectivity (JDBC) drivers to query relational databases Can reuse existing classes for Oracle, Postgres, and MySQL JDBC. Drivers can read data from files via HTTP, or parse HTML tables. “Test harness” available for driver development

How much work? Add a new database to MetBroker –roughly working - 3 days –tested more thoroughly – 1 week –(assuming that database is readily accessible) Make a MetBroker applet –working prototype in one day –(assuming basic science is “done”)

Developer Resources Demos, documentation, downloads, pictures – Mailing lists – subscribe at web site Brokers are open source under the GNU Lesser General Public License. –Can be used in commercial applications –Any improvements must be released under same license

Linking your data to the grid lets you use grid tools developed by others. It doesn’t mean you must share your data with everyone (password protection). The key to interoperability is the interface Reminders

Contents Concepts –Remote Procedure Calls (RPC) –Brokers Using a broker –Metadata requests –Data requests –Issues –Linking your own data SOAP New brokers and future enhancements

SOAP An XML-based RPC mechanism Independent of programming language SOAP toolkits for C++, Visual Basic, Delphi, Java, ASP, PHP, Perl, Python… Firewall-friendly Basis of Microsoft.NET architecture A SOAP Web service is described in standard format using a.WSDL file (Web Services Description Language)

Visual Basic client application Application code MS SOAP Toolkit VB function calls VB objects MetSOAP SOAP over HTTP MetBroker Delphi client application Application code Borland SOAP Toolkit Delphi function calls Delphi objects SOAP over HTTP RMI Example - MetSOAP

SOAP Summary Positives: –Opens brokers to a range of programming languages –Already the standard for Web-based RPC –Avoids most firewall issues Negatives –Can’t reuse Java classes - must “mirror” in each language –Interoperability is still imperfect (but improving fast) –Inefficient for large data sets

Typical Array in SOAP

Model Deployment Options - Summary Java applet –flexible, responsive once loaded but requires plug-in Java servlet –simple browsers, rapid load, but clunky interface SOAP –direct access for Delphi, VB… Existing models (FORTAN,C,BASIC) –Use simple Java app. to retrieve data and write to a file

Contents Concepts –Brokers –Remote Procedure Calls (RPC) Using a broker –Metadata requests –Data requests –Issues –Linking your own data SOAP New brokers and future enhancements

New Brokers DEM Broker - Digital Elevation Model data –Japan 50m resolution –World 1km resolution ResourceServer –Internet-based service for software localization ChizuBroker – Web map services – (Japan) – (New Zealand) SoilBroker – soils data

Future enhancements Broker-side interpolation (using DEMBroker) –temporal –spatial

Server Temperature Rainfall Radiation Client “Virtual station” Broker-side interpolation at single site

Future enhancements Broker-side interpolation (using DEMBroker) –temporal –spatial Other brokers –Place names –Crops and crop models (prototyped) –Agrichemicals

Final Points The Grid provides a stable, powerful foundation for Ag and NR software You can now develop grid applications in your favourite programming language. NARO is keen to help you to both develop new applications and link new databases. Database owners around the world will be keen to evaluate your applications.

Thank you