DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan Potter and Patrick West

Outline Different DAP Servers Hyrax Architecture and customization Example Configuration of Hyrax

There are several different DAP servers Hyrax, developed by OPeNDAP TDS, developed by Unidata GDS, developed by COLA PyDAP, developed by Roberto De Almeida Others we won’t cover: Dapper; FDS; CODAR; OPeNDAP Server3; et cetera Terminology: We often say ‘server’ when describing what the technical press calls a ‘web application.’

Hyrax Supports multiple protocols –Data: DAP using HTTP/GET and HTTP/SOAP; Direct access (via HTTP); WCS/WFS funded, in development –Catalog: THREDDS; HTML directories Data formats: In binary distribution: NetCDF; HDF4; HDF5; FreeForm; many more available as source code. Includes ASCII data dump, HTML data access form, Info metadata page

Hyrax Architecture Two (or more) cooperating processes: –Front-end provides DAP interface –Back-end(s) read(s) data Both parts can be customized –Front-end: different network protocols –Back-end: different data formats/systems N-Tier design is flexible, secure OLFSBES Data

TDS Java Servlet network interface Supports multiple protocols –Data: DAP; WCS; NetCDF Subset; Direct access (via HTTP) –Catalog: THREDDS Data formats: NetCDF; HDF5; GRIB-1,2; NEXRAD; DORADE; BUFR; DMSP; GINI; more in development Can also read from any other DAP server Can serve aggregations

THREDDS Data Server HTTP Tomcat Server Datasets catalog.xml motherlode.ucar.edu THREDDS Server NetCDF-Java library Application IDD Data HTTPServer NetcdfSubset WCS OPeNDAP configCatalog.xml

PyDAP Python native implementation from the DAP2 specification; WSGI application Server uses run-time loaded modules Protocols: –Data: DAP (HTTP/GET); WMS/KML; EditGrid/Google Spreadsheets; JSON –Catalog: THREDDS using a script/batch system; full support planned Formats: CSV; netCDF; SQL Matlab; GRIB; HDF5

PyDAP Server Architecture: Plugins and Responses

GDS Java Servlet network interface and C/C++ back-end Supports multiple protocols: –Data: DAP (HTTP/GET) –Catalogs: THREDDS Formats: GRIB; HDF4; netCDF; BUFR; Binary; GrADS Station Can also read from many DAP servers GDS can perform sophisticated server-side operations and build interim data sets as a result. Server-side operations use GrADS command syntax.

GrADS Data Server

DAP Servers Summary Four servers, all developed by different groups, were described (Hyrax, TDS, PyDAP and GDS) Each supports DAP and THREDDS Each has unique features: –PyDAP: WMS, KML and python scripting –GDS: Extensive server-side operations –TDS: Aggregation, WCS –Hyrax: Customizable, SOAP interface, WCS/WFS support funded These are more properly called ‘web applications.’

Hyrax Architecture Front-end (OLFS) customization Authentication & Authorization Back-end (BES) customization Single/Multiple Machine Installations Security

OLFS Java Servlet Engine BES Unix Daemon BES Commands XML- encapsulated object File system with data files, SQL Database, … DAP2 THREDDS HTML Optional THREDDS catalogs Hyrax Architecture

The OLFS uses an ordered list of Dispatch Handlers. Each handler on the list is asked if it can handle the incoming request. The first handler on the list that claims the request gets to handle it. List order is important, as some handlers (for example THREDDS) may claim to handle requests that should be handled by a different handler. Handlers are free to do anything they need to handle a request: Contact a remote system/process, read files, spawn threads, et cetera. OLFS Dispatch Handlers

Extending the OLFS Extension ‘modules’ written in Java Added to a directory within Tomcat The new modules have complete access to the request information Both HTTP GET and POST requests There is some significant processing done before the handler is called: –Conditional GET Requests –Authorization & Authentication

OLFS and Authorization/Authentication Authorization & Authentication (Az/Ac) actually provided by Apache or Tomcat Apache/Tomcat provides a range of Az/Ac features: –Realms: Role-based Az/Ac Usernames & passwords -> Authorization Roles -> Authentication Realm info in RDBs, XML docs, LDAP, … –TLS/SSL: Server and client Az/Ac Secure data transmission Tomcat provides single sign-on capabilities –Clients must support cookies –Suitable for portals

BES Network Protocol and Process start/stop activities Data Store Interfaces BES Framework PPT* Initialization/ Termination DAP2 Access NetCDF3HDF4FreeForm … Data Catalogs Commands** BES Commands/ XML Documents *PPT: point to point protocol **Some commands are built in, most load at run-time

BES Extensibility In what ways can you extend the BES? New request handlers (data handlers like netcdf, hdf4) New response handlers (Cedar's Flat, Tab, Info responses) New commands (like our hello world example) Initialization/Termination Exception Handlers Reporters Container and Definition storage Aggregation engines Methods of returning your data (return as netcdf)

Possible Installations Both OLFS and BES run on one machine OLFS on one machine and BES on another One OLFS and several instances of BES on different machines OLFS communicating with one or more BESs and other backend processors

Installation Security A separate issue from Ac/Az The BES must be protected: – With a firewall or –TLS & Client certificates Running the OLFS and BES on separate machines limits the scope of a compromise of the OLFS Ensure that the BES, Tomcat and Apache all run with limited access to the server host

Hyrax Architecture Summary Hyrax can be installed on one machine or several Installation security merits serious consideration Authentication & Authorization are handled by the web servers (Tomcat and Apache)

Hands on: Hyrax Configuration Choices: –Single or Multiple machine –Single or Multiple back-end servers –Tomcat or Apache web server –Data formats –Catalog customization –Security Testing –Command line tools for system administrators –Web browser

APAC Workshop Configuration Run a single BES and the OLFS on one host (the virtual machine running SLAX Linux) Use Tomcat running on port 8080 Data formats: NetCDF and FreeForm No custom THREDDS catalog No firewall

APAC Hardware SLAX Linux VMware Virtual Appliance All the software needed has been built, installed and configured Start the virtual machine now…

Starting the Virtual Machine Insert the CD-ROM Drag the opendap_vm and Wintools folders to the Windows desktop Copy the ISO image from the CD ROM to the harddisk (in a command window, mkdir C:\SLAX and then use copy or dd to copy the iso to the new directory). –The dd command is in the Wintools folder –The C:\SLAX directory is coded into the virtual machine; use Vmware Workstation/Fusion to edit Open the opendap_vm folder and double click on the opendap_vm.vmx virtual machine configuration file or start VMware Player/Workstation/Fusion and open opendap_vm.vmx from there. The virtual host will boot and the SLAX (slackware Linux with KDE) desktop will appear.

Installing the Software Already present on the virtual machine But, if it wasn’t, you’d go to the OPeNDAP web site* and download: –Binaries or source for the BES and the data handlers needed. –Get the OLFS web archive file (which is a compiled java servlet). –You also need the Tomcat servlet engine *

Location of Server Files on the Virtual Machine On your virtual machine: –The Tomcat servlet engine is in /usr/local/javadev/apache/tomcat (this is the value of $CATALINA_HOME ) –The BES, data handlers and related source files are in /usr/local/src. The BES has been built and installed in /usr/local/ ($prefix) –The OLFS web archive file is $CATALINA_HOME/webapps/opendap.war –BES: bes.conf, found at $prefix/etc/bes/bes.conf –OLFS: olfs.xml and catalog.xml, found at $CATALINA_HOME/content/opendap

Background: Starting the Server Start the BES (back-end data processing component) Use bescmdln to verify it’s working Start Tomcat: This automatically starts all installed servlets –Servlets are installed by copying the.war file to the servlet’s webapps directory Verify it’s working using a web browser

Start the BES

Verify the BES is running

Start Tomcat & the OLFS Typical steps: –Unpack the olfs jar-file –Copy the opendap.war file to Tomcat’s webapps directory –Start Tomcat Since all but the last step has been done already, start Tomcat: –/usr/local/javadev/apache-tomcat /bin/startup.sh

…terminal view

Verify Tomcat is running

…and Hyrax

Complete the Configuration Steps you would typically perform: –Security: Set up a firewall! Limit access to port to this host only –Custom catalogs: Edit the catalog.xml configuration file. By default automatic catalogs are generated –Logging: Edit the log4j.xml file. By default all accesses are logged

Stopping Hyrax First, stop Tomcat using ‘shutdown.sh’ …then stop the BES using ‘besctl’

Hyrax Configuration Summary For our chosen configuration, we used only Tomcat, plus the Hyrax web application which consists of the BES and the OLFS The tools bescmdln, getdap and a web browser were used to test the installation There are a lot of options, but the default settings produce a working server Security is a must for a web application; use a firewall to isolate the BES so only the local host can connect to it.

Summary There are a number of DAP-compliant servers (Hyrax, TDS, PyDAP and GDS) were described All of the servers (read ‘web applications’) actually support several other protocols While DAP does nothing to support cataloging data, THREDDS supports just that Hyrax is customizable and can be installed in several different configurations