1. Don Murray Unidata Program Center
The Abstract Data Distribution Environment (ADDE) – A technical overview
Don Murray
Unidata Program Center

2. Outline What is the ADDE? Who uses ADDE? Technical details
Datasets and supported data types
Protocol details
Overview of ADDE requests and returned data
Java and ADDE
ADDE use in the IDV
Summary analysis

3. What is the ADDE?
Client/server data access model developed for McIDAS, but not limited to serving McIDAS data
In use for nearly 10 years
4 primary types of data objects
GRID, IMAGE, POINT, TEXT
Primary servers handle McIDAS format files (AREA, GRID, MD, LW)
Secondary servers read non-McIDAS formats (GINI, NEXRAD Level III, MODIS, netCDF,…)
Allows browsing and subsetting of datasets
ADDE was originally developed as part of the McIDAS display and analysis package. It is based on a client/server data access model and allows McIDAS users to access data on local and remote servers.
There are 4 basic types of data that ADDE defines – grid (gridded data), image (image data), point (observational data) and text (text bulletins). The design of ADDE is such that you could serve up a grid as an image, or an image as a grid.
While the core servers distributed with McIDAS understand McIDAS-formatted files, the API allows the development of subservers to handle non-McIDAS data formats. These subservers adapt the data to the appropriate ADDE data structure which is then passed on to the client.
Another important feature of ADDE is its browse and subset capabilities. You can probe a dataset to determine what its temporal and spatial characteristics as well as what parameters are available. And, you can subset the data before returning it to the client so if you had a 1km global Vis image, and you were only interested in a view over Albuquerque, you can pull out just that piece you are interested in.

4. Who uses ADDE? Unidata community McIDAS community
Cooperating set of servers providing data to McIDAS and IDV users
Meteoforum
McIDAS community
International government agencies (e.g., Spain, ESA, Australia)
US agencies (e.g., NESDIS, NTSB)
Satellite researchers

5. ADDE Dataset definition
A dataset is collection of one or more files with a common format. Each dataset has a name that consists of a group and a descriptor (usually written as group/descriptor).
Each dataset is in a group:
A group can have one or more data types in it. Examples:
RTIMAGES – real time image data only
BLIZZARD – data (images, grids, point data) for a case study or tutorial
Each group has descriptors which define a set of data of the same type (i.e., image, grid, point or text).
Examples:
RTIMAGES/GE-IR – set of GOES-East IR images in the RTIMAGES group
RTIMAGES/MOLL-IR – Mollweide Composite IR images in RTIMAGES
BLIZZARD/GRIDS – set of NGM grids for the BLIZZARD data group
Groups can be interrogated for a list of descriptors and data types it contains.

6. Currently Supported Data Formats
Image
McIDAS AREA, GINI, NEXRAD Level III radar, netCDF (output only), MODIS, AIRS, GVAR, POES, Level 1B, Meteosat (including MSG), NOWrad®, GMS and FY-n
Grid
McIDAS GRID, netCDF, GRIB (in development)
Point
McIDAS MD, netCDF, HDF4
Text
Plain text, McIDAS-XCD observation text (e.g., raw METAR, RAOB) and bulletins (e.g., watches and warnings)

7. Protocol Details
Client connects on a particular port (500, 503 or 112). With next version, only Port 112 will be used. Supports compression using compress or gzip.
Handshake from client involves sending:
Version info (make sure server supports this)
ADDE version (1 for now)
Pre-request information (for validation)
Server IP and compression type
Request Type (AGET, ADIR, etc)
Request Block
Server IP and compression type (again)
Client address
User, project, password (for authentication)
Request Type (again)
Actual request for data
Server sends back data or error message

8. Validation/Security
ADDE supports 3 types of validation for a request to a server:
IP filtering
Username
Project number
Security through obscurity – no mechanism for querying an ADDE server to find out what datasets are available.
TCP wrappers can be used to limit access to ports
Subservers can implement their own forms of validation/security (e.g., NEXRAD Level III server)

9. Request types Request Type Java equivalent Data Type Description ADIR
imagedata
image
image header information
AGET
imagedir
image header, navigation, calibration and data; data is returned line by line; comments
GDIR
griddirectory
grid
grid header information
GGET
griddata
grid header and entire grid
LWPR
datasetinfo
info
dataset information
MDFH
not implemented
point
point file header information
MDHD
point header information
MDKS
pointdata
point header and data
OBTG
obtext
text
observational weather text
TXTG
ASCII text file
WTXG
wxtext
textual weather information

10. RTIMAGES MOLL-IR 0 BAND=ALL X TRACE=0 AUX=YES VERSION=1
Anatomy of a Request
An ADDE request is a text string containing positional parameters and some key=value pairs (just like a McIDAS command).
A sample request for the latest Mollweide IR image:
RTIMAGES MOLL-IR BAND=ALL X TRACE=0 AUX=YES VERSION=1
Dataset Descriptor
Key=value1 value2 … valuen
(in this case, number of bands to request from image (ALL))
Server debug flag
ADDE Version
Dataset Group

11. Image data ADDE image data model supports multi-banded images
2 main types of requests
Directory (ADIR) and data (AGET)
Returned image object models McIDAS AREA format:
Directory block
Navigation block
Calibration block
Supplemental (AUX) block
Data block
Comment block
Image data is quantitatively useless unless it is transformed into physical units (calibration) and oriented relative to time and physical space (navigation). In addition, it is often necessary to know when and how an image was collected and processed. The actual image, along with these ancillary data , is collectively called an image object . Each image object in McIDAS-X is composed of the following blocks of information:
The directory block contains a list of ancillary information about the image, such as the number of lines and data points, the satellite ID, and the number of spectral bands.
The data block contains the matrix of image data values.
The line prefix block contains information about an image that may vary on a line-by-line basis, such as calibration or documentation information.
The navigation block contains information for determining the location of data points in physical space. More information about navigation is presented in the section titled McIDAS navigation later in this chapter.
The calibration block contains the information for converting image data from its internal (stored) units to more meaningful physical units, such as radiance or albedo. More information about calibration is presented in the section titled McIDAS calibration later in this chapter.
The supplemental (or auxiliary) block contains additional information that is specific to a data type. For example, information specific to radar data is stored in this block. Also, the latitude/longitude grid for the LALO navigation is stored in this block.
The comment block contains a variety of textual information, such as a list of commands run on the image object to-date.

12. Image Object Details Image Directory Data Block Navigation block
contains a list of ancillary information about the image, such as the day and time, number of lines and data points, the satellite ID, and the number of spectral bands.
Data Block
contains the matrix of image data values. Multibanded image have values interleaved:
Navigation block
contains information for determining the location of data points in physical space. Client must have navigation module which uses this block to convert (line,element)<->(lat,lon) for geolocation
Calibration block
contains the information for converting image data from its internal (stored) units to more meaningful physical units, such as radiance or albedo
AUX block
contains additional information that is specific to a data type. For example, information specific to radar data is stored in this block. Also, the latitude/longitude grid for the LALO navigation is stored in this block.

13. AREA=FILE coordinates, ignore TV coordinates for now
Image Coordinates
AREA=FILE coordinates, ignore TV coordinates for now

14. Image Data (con’t)
An image directory (ADIR) request returns all images matching the request
An image data (AGET) request returns only one image at a time.
Request refinements
Location can be specified by image, file or lat/lon coordinates.
Single or multi-banded images
Day and time
Calibration units
Size (number of lines/elements)
Resolution (line/element magnification)
Relative position number (i.e. last N images)

15. Grid Data
Grid data is two-dimensional data representing a parameter along an regularly spaced matrix (e.g., model output, objective analysis).
2 main types of requests
Directory (GDIR) and data (GGET)
Grid object consists of:
Grid Header
Data block
Gridded data is two-dimensional data representing an atmospheric or oceanic parameter along an evenly spaced matrix. For the matrix to be useful, ancillary information about the grid must also be known. This ancillary information, along with the gridded data, is collectively called a grid object . Grid objects in McIDAS-X contain two blocks of information.
The grid header contains a list of ancillary information about the grid, such as the parameters and units of the data in the grid, the level in the atmosphere or ocean the data represents, the grid navigation information, and the time.
The data block contains the matrix of gridded data values.

16. Grid Object Details Grid Directory Data block
contains a list of ancillary information about the grid, such as the parameters and units of the data in the grid, the level in the atmosphere or ocean the data represents, the grid navigation information, and the time.
Client must have navigation module which uses the navigation info to convert (row, column)<->(lat,lon) for geolocation
Data block
contains the matrix of gridded data values.

17. Grid Data (con’t)
A grid directory (GDIR) request returns stream of grid directories matching request
A grid data (GGET) request returns stream of grid data objects matching request
Request refinements:
Parameters or Derived quantities
Levels
Model run and/or valid day/time
Originating center
Maximum number of grids to return

18. Point Data
Point data typically represents data occurring at irregularly spaced locations around the Earth (ex. Surface observations, upper air reports, lightning flashes)
Main type of request is for data (MDKS)
Point data object consists of 5 blocks:
Parameter block
Unit block
Scale block
Form block
Data block
McIDAS-X point data typically represents data occurring at irregularly spaced locations on the Earth. For this data to be useful, ancillary information about the data must also be known. This ancillary information, combined with the actual point data values, is collectively called the point object .
Each point object in McIDAS-X contains five blocks of information.
The parameter block contains a list of the parameter names in the point object returned by the server.
The unit block contains a list of units for the parameters returned by the server.
The scale block contains a list of scaling factors for the floating- point values returned by the server.
The form block contains a list of the return forms for each of the parameters.
The data block contains the actual data values returned by the server.

19. Point data object details
The parameter block contains a list of the parameter names in the point object returned by the server.
The unit block contains a list of units for the parameters returned by the server.
The scale block contains a list of scaling factors for the floating- point values returned by the server.
The form block contains a list of the return forms for each of the parameters.
The data block contains the actual data values returned by the server.

20. MD File structure
Meteorological Data (MD) file schema determines data layout
An MD file is like a spreadsheet with each cell containing a predefined number of data values. Each cell contains data for a specific location at a given instant in time.

21. Point data (con’t) Request refinements Examples: List of parameters
Maximum number of obs to return (default 1)
The SELECT clause gives the user the ability to subset on any parameter in the dataset.
Examples:
SELECT='T[F] 40 50; ST WI, MI; TIME 12 13’ MAX=ALL
Selects all parameters for all observations between 40 and 50 degrees Fahrenheit for stations in Wisconsin and Michigan between 12 and 13 UTC
SELECT=‘ID KDEN’ PARM=T TD PRE MAX=ALL
Selects all temperature, dewpoint and pressure values for all times in the dataset for Denver

22. Text Data There are 3 types of text data that are served up by ADDE:
Flat files, ancillary data files
Weather bulletins such as forecasts, warnings, watches
observational data, such as METAR or RAOB reports
3 types of requests:
TXTG ASCII text file
WTXG textual weather information
OBTG observational weather text
Returned data has a header and the text

23. Java and ADDE
Client interface developed and refined collaboratively by Unidata, SSEC and Australian Bureau of Meteorology (BoM)
Provides ADDE data access for Java-based data analysis and display tools (e.g., IDV, Matlab)
Uses specialized URL
adde://server/request?keyword_1=value_1&keyword_2=value_2…keyword_n=value_n
“adde” specifies protocol
“request” specifies data type/action
keyword/value pairs refine request
Bundled with VisAD component library
Package edu.wisc.ssec.mcidas.adde contains core package
Package visad.data.mcidas has classes for converting ADDE data objects into VisAD objects.
The interface we’ve developed allows Java applications to access data from existing (and future) ADDE servers. Each of the groups has invested a lot of resources in setting up ADDE servers. But as we develop new tools, we still want access to the same datasets that our old tools use.
We accomplished this by creating a specialized Java URLConnection class that defines the ADDE request.
In this URL, adde defines the protocol
request specifies the particular data type/action. This might define whether the user wants to browse and image data set and return some metadata, or return a particular portion of the dataset.
Keyword/value pairs are used to refine the request. These might specify the geographic extent and size of the returned data, or the particular band of an image.

24. Java ADDE Use in Applications
Unidata
Integrated Data Viewer (IDV)
Access to satellite, Level III radar, METAR, synoptic, upper air and profiler data
SSEC
McIDAS AREA to netCDF converter for AWIPS use
Java client for browsing and copying real-time and archive imagery
MODIS data exploration
BoM
Development of subservers (Oracle/NEONS, radar)
Java-based clients
Australian Marine Forecast System
Tropical Cyclone Forecast

25. ADDE Use in IDV The IDV uses ADDE to access:
Satellite and Level III radar imagery
Surface (METAR, synop) data
Upper Air (RAOB) data
Profiler data
Text data
ADDE data objects are converted to VisAD data objects
Navigation of images done through AreaCoordinateSystem

26. IDV/ADDE Demo Satellite image (GINI East – 1km VIS)
Level III Radar selection (Denver)
RAOB
Sounding
Plan view
Profiler (Platteville)
Time/Height
3D view
Text
File (PUBLIC.SRV)
Denver METAR
Denver RAOB

27. Summary Analysis Strengths Weaknesses
Supports many data formats, especially for imagery
Subsetting capabilities
Datasets can be queried for metadata
Server freely available to research and education institutions through Unidata
Java/C/FORTRAN client APIs
Java client freely available
Point data limitations
MD file limitations (4 character names/unit names, 400 parameter limit, scaled integers)
Grid data limitations
2D grids only, 4 character param/unit names
Server configuration (best done from McIDAS)
Currently not separate from McIDAS
Limitations
When accessing point data, the MD (Meteorological Data) file structure has four limitations, which are explained below. These limitations pertain only to the MD file structure and are not limitations of the ADDE point object subsystem. Point servers for other formats, netCDF for example, use configuration files to map the parameter names in the file to McIDAS-compatible names.
Cells
Each cell is limited to 400 elements. For most observational data, this isn't problematic. The exception occurs with observations containing data at several levels of the atmosphere. For example, you can't store all the information for an upper air observation reporting values for eight different parameters at 50 levels of the atmosphere in one cell. Although the cell can accommodate the 400 values (8 x 50), it won't have enough space for the time and geographic location of the observation, which are also provided.
Element names and units
Element names and units are limited to four characters, which can be restricting when designating parameter units, especially derived parameters.
Character string elements
Character string elements are limited to four characters, which can be limiting for any type of alphanumeric parameter, such as station ID or country. You can bypass this restriction by using several parameters strung together to represent strings. McIDAS-XCD uses this method when representing five-character IDs associated with ship reports.
Numeric values
Numeric values can be stored only as scaled integers.

28. Future Development Ideas
Middleware to support metadata queries for available times and parameters (not just catalogs)
Enhanced data choosers which take into account the semantics of the datasets
Support more navigation modules in Java
Enhance netCDF point server
Serve up GEMPAK grids
Java server

29. ADDE Resources McIDAS Programmer’s Reference
McIDAS User’s Guide (ADDE section)
Javadoc for AddeURLConnection class: