2. ITSC/University of Alabama in Huntsville ADaM System Architecture Rahul Ramachandran, Sara Graves and Ken Keiser Mathematical Challenges in Scientific Data Mining IPAM January 14-18, 2002 Information Technology and Systems Center University of Alabama in Huntsville University of Alabama in Huntsville rramachandran@itsc.uah.edu

3. ITSC/University of Alabama in Huntsville Talk Overview Mining System Requirements ADaM System Architecture ADaM Plan Builder Research directions

4. ITSC/University of Alabama in Huntsville Mining System Requirements: When,Where and Who WHEN Real Time On-Ingest On-Demand Repeatedly WHERE User Workstation Data Archive Center Data Mining Center WHO Casual Users Domain Experts Mining Experts Data Mining

5. ITSC/University of Alabama in Huntsville Algorithm Development and Mining (ADaM) System ADaM system developed under NASA research grant The system provides knowledge discovery, feature detection and content-based searching for data values, as well as for metadata. It contains over 120 different operations to be performed on the input data stream. Operations vary from specialized atmospheric science data- set specific algorithms to different digital image processing techniques, processing modules for automatic pattern recognition, machine perception, neural networks and genetic algorithms.

6. ITSC/University of Alabama in Huntsville ADaM Features Handles science data set variability Multiple resolution/multiple scales Variability of formats Granularity of data Includes spatial/temporal dimensions Allows addition of new algorithms Allow scientists to select and sequence different operations

7. ITSC/University of Alabama in Huntsville ADaM Engine Architecture Preprocessed Data Preprocessed Data Translated Data Patterns/ Models Patterns/ Models Results Output GIF Images HDF-EOS HDF Raster Images HDF SDS Polygons (ASCII, DXF) SSM/I MSFC Brightness Temp TIFF Images Others... PreprocessingAnalysis Clustering K Means Isodata Maximum Pattern Recognition Bayes Classifier Min. Dist. Classifier Image Analysis Boundary Detection Cooccurrence Matrix Dilation and Erosion Histogram Operations Polygon Circumscript Spatial Filtering Texture Operations Genetic Algorithms Neural Networks Others... Selection and Sampling Subsetting Subsampling Select by Value Coincidence Search Grid Manipulation Grid Creation Bin Aggregate Bin Select Grid Aggregate Grid Select Find Holes Image Processing Cropping Inversion Thresholding Others... Processing Input HDF HDF-EOS GIF PIP-2 SSM/I Pathfinder SSM/I TDR SSM/I NESDIS Lvl 1B SSM/I MSFC Brightness Temp US Rain Landsat ASCII Grass Vectors (ASCII Text) Intergraph Raster Others...

8. ITSC/University of Alabama in Huntsville ADaM Mining Environment ADaM Mining Environment Mining Results Mining Engine (ADaM) Analysis Modules Input Modules Output Modules Analysis/Vis Tools Knowledge Base Distributed Clients Web-based Workstation based Other Systems Common Client API Data Stores Data Mining Server Event/ Relationship Search System

9. ITSC/University of Alabama in Huntsville ADaM Architecture

10. ITSC/University of Alabama in Huntsville ADaM Miner Engine Manages the processing of data through a series of specified operations Loads input, processing and output modules dynamically as needed at execution time Allows for the addition of newly developed modules without the need to rebuild the engine Interprets a mining plan script that provides the details about specified operations and the order that they should be executed

11. ITSC/University of Alabama in Huntsville ADaM Miner Database Used to store information that includes the names, locations and related metadata for input data sets available on the server Includes information about users, jobs, mining results, and other related information Simple relational database

12. ITSC/University of Alabama in Huntsville ADaM Daemon and Scheduler Scheduler Examines the list of jobs to be executed on the server and determines which job or jobs to execute at any given time Queues the requests and executes them sequentially. Daemon Handles all network communications with the mining system Is configured to listen on a specific port for any socket communications

13. ITSC/University of Alabama in Huntsville ADaM Input/Operation Filters Input/Output Filters are data readers and writers Operations are the algorithms Each of the operations and (input/output) filters is implemented as a shared library New modules may be added to the system without recompiling or relinking. All operations/filters either produce or operate on a data collection, which provides a common format for representing scientific data.

14. ITSC/University of Alabama in Huntsville General Mining Steps Select data files to be mined “Check-In” the data files into the Miner Database Write a “Mining Plan” consisting of sequence of input filter and operations Execute the Mining Plan using the engine Check and save results Iterate

15. ITSC/University of Alabama in Huntsville What is Check-In? Process of encoding information such as the names, locations and related metadata for input data sets available on the server Create complex data hierarchy in the database

16. ITSC/University of Alabama in Huntsville ADaM Plan Builder: Check-In Two Modes of Operation -General: which only requires minimal information -Advanced: requires more detailed information and Allows user to set up structured database Path to the data file Data file name Input Filter associated with the Data file Load an XML file containing existing Check-In specifications

17. ITSC/University of Alabama in Huntsville ADaM Plan Builder – Layout Plan Menu allows one to: Select a new plan Load existing plan Check-In data Input Menu contains the list of Input Filters one can select Operation Menu contains the list of operations one can select

18. ITSC/University of Alabama in Huntsville ADaM Plan Builder – Layout Panel where Mining Plan can be viewed either as text or a tree

19. ITSC/University of Alabama in Huntsville ADaM Plan Builder – Layout Description about the Operation/Input Filter can be viewed in this panel

20. ITSC/University of Alabama in Huntsville ADaM Plan Builder – Layout All the parameters needed for the Operation are described here

21. ITSC/University of Alabama in Huntsville ADaM Plan Builder – Layout Sample values for Operation’s parameters are shown in this panel

22. ITSC/University of Alabama in Huntsville ADaM Plan Builder – Layout Go Mine the data using the Mining Plan Allows user to select the operation and add it to the Mining Plan

23. ITSC/University of Alabama in Huntsville Research Directions Generic Data Reader for ADaM ESML – Earth Science Markup Language Programmers Guide for ADaM Distributed Mining Grid Mining Successful implementation and testing of the ADaM system on the NASA Information Power Grid Mining Onboard the Space Craft The EnVironmEnt for On-Board Processing (EVE) system

24. ITSC/University of Alabama in Huntsville ADaM Information Web site: datamining.itsc.uah.edu ADaM Lite beta version download Contact: rramachandran@itsc.uah.edu