JAS – Distributed Data Analysis Grid Enabled Analysis Workshop Caltech - June 23-25, 2003

June 2003 JAS – Distributed Data Analysis Contents JAS2 History client-server mode JAS2 and the Grid JAS3 What’s new JAS3 and AIDA Plans for Gridification

June 2003 JAS – Distributed Data Analysis JAS History First version of JAS2 released in Incremental improvements released over time.

June 2003 JAS – Distributed Data Analysis JAS2 History – Use Cases With WIRED event display Online Monitoring

June 2003 JAS – Distributed Data Analysis JAS2 History – Use Cases Custom Applications Web Servlets

June 2003 JAS – Distributed Data Analysis JAS Client-Server Mode Java Compiler + Debugger GUI Experiment Extensions (Event Display) User’s Java Code Padded Cell Data Analysis Engine DATA

June 2003 JAS – Distributed Data Analysis Distributed Analysis System: Goals Prototype for GRID enabled JAS analysis Run analysis on a farm of machines  Use multiple CPU’s in parallel for CPU-intensive analysis  Access multiple I/O channels for data-intensive analysis Use standard JAS (Client) as if we are running a local Job Get interactive feedback  Create analysis modules (code)  Control job execution  View results (Plots/Histograms) Access distributed datasets as if they were local datasets

June 2003 JAS – Distributed Data Analysis Distributed Analysis System: Architecture JAS Client JAS Data Server JAS Client JAS Client … Network Users Catalog Server Control Server Network JAS Data Server …

June 2003 JAS – Distributed Data Analysis JAS 2 – GRID interface (Tech-X)

June 2003 JAS – Distributed Data Analysis JAS3 Overview A completely new version of JAS Design based on Application Shell, into which many (optional) modules can be plugged  Highly customizable for different application domains –HEP/Astrophysics/Other –DST analysis/Online Monitoring/GRID analysis –Experiment/User specific modules  Modules can be updated independently of shell –Possible to release bug fixes fast Includes support for programming in many languages  Scripting: Python, Pnuts, Dynamic Java, …. –Command prompt  Java (compiled) Analysis (histograms, tuples, fitting) based on AIDA standard Not technically backwards compatible with JAS2 But migration is straightforward.

June 2003 JAS – Distributed Data Analysis AIDA Overview AIDA = Abstract Interfaces for Data Analysis Covers key areas for data analysis  Histograms, Tuples, Fitting, Data Points, Plotting, Management Developed collaboratively at series of workshops by groups at CERN, LAL, SLAC.  Next workshop June 30-July 4 -- CERN Interfaces developed for C++ and Java ( and maybe Python?) Several implementations/tools available  Anaphe/Lizard/LCG PI – CERN  Open Scientist – LAL  JAIDA/JAS/AIDAJNI – SLAC

June 2003 JAS – Distributed Data Analysis JAS3 and AIDA JAS3 has adopted AIDA for analysis AIDA allows us to leverage experience and skill of other developers AIDA is functionally more complete than JAS2 analysis package AIDA allows JAS to exchange data with other AIDA tools AIDA provides bridge to C++ programs (e.g. Geant4) AIDA encourages creativity and innovation JAS3 HEP Analysis tools based on JAIDA JAIDA = Java implementation of AIDA  JAIDA is part of FreeHEP library  Usable as standalone library for any Java Application AIDAJNI = Interface between C++ and Java AIDA  Allows C++ programs to use JAIDA, JAS3

June 2003 JAS – Distributed Data Analysis JAS3, AIDA and C++ C++ program AIDA C++ AIDA Implementation AIDA- JNI AIDA JAIDA JAS3.aida file (XML) Java program AIDA

June 2003 JAS – Distributed Data Analysis JAS3 and AIDA JAS3 supports all AIDA functionality, including Histograms (includes arithmetic, projections, etc.) Clouds (unbinned histograms, scatterplots) Plotter Tuples Fitting – AIDA interfaces allow for multiple fitters  Uncmin -- pure java minimizer  Minuit -- Fortran called by Java Native Interface (JNI) IO  AIDA XML, PAW, Root JAS3 supports user interaction with AIDA in three ways Scripting (Pnuts, Python etc) Compiled (Java) code GUI – Plotting, Fitting, Cuts etc.

June 2003 JAS – Distributed Data Analysis JAS3 Scripting JAS3 has multi-language OO scripting support Command line, Console, Editor Major components (e.g. AIDA) have scripting interfaces Currently have plugins to support Pnuts Pnuts – syntax almost identical to Java, fast, well documented and feature complete Python Python (using Jython) More scripting languages can be added  not restricted to Java implementations (e.g. could use C-Python, JPE)

June 2003 JAS – Distributed Data Analysis JAS3 Lightning Tour Tour designed to give you an overview of the capabilities of JAS3, you can try them out for yourself this afternoon. Welcome Page, gives initial info and links to example scripts and programs Memory monitor

June 2003 JAS – Distributed Data Analysis Opening Files Use file menu Drag from explorer

June 2003 JAS – Distributed Data Analysis Graphical Interface to AIDA Histograms, Clouds, Tuples all presented in AIDA tree.aida files,.hbook files,.root files all presented as AIDA objects Drag items onto page, or use (popup) menus

June 2003 JAS – Distributed Data Analysis Printing Can send individual plots or full page direct to printer Or save as PS, EPS, PDF, SWF, SVG, PNG, GIF… Or copy/paste into Word, PowerPoint etc.

June 2003 JAS – Distributed Data Analysis Java Editor, Compiler and Loader Built-in Java compiler Tree shows loaded programs Built-in editor for writing analysis code Unlike JAS2 which only supported “event analyzers” JAS3 allows any Java program to be loaded. This example “main routine” is taken directly from the AIDA manual

June 2003 JAS – Distributed Data Analysis Scripting Can also write and run scripts Console allows direct interaction with scripting language

June 2003 JAS – Distributed Data Analysis Pnuts Language Currently support Pnuts scripting language Complete and well documented  Fast (although not as fast as compiled Java) Syntax very similar to Java Can easily call compiled Java classes from scripts – best of both worlds Plan to support other languages in future In particular Python

June 2003 JAS – Distributed Data Analysis Record Sources Opening record (or event) based files causes the run control toolbar to appear Works similarly to JAS2 Job control, but now also supports random access and “tagged” data sets (mainly for event displays)

June 2003 JAS – Distributed Data Analysis HistogramProfile ScatterPlot XY Data (More appropriate for smaller data sets) Tuple Explorer - Plots Works with any tuple, read from file or dynamically created

June 2003 JAS – Distributed Data Analysis Tuple Explorer – Define Columns

June 2003 JAS – Distributed Data Analysis Tuple Explorer - Cuts

June 2003 JAS – Distributed Data Analysis Tuple Explorer - Tabulate

June 2003 JAS – Distributed Data Analysis To be used with record loop Tuple Explorer – Record Source

June 2003 JAS – Distributed Data Analysis JAS3 Spreadsheet Simple spreadsheet plugin for  Displaying results  Calculations  Simple Plots Supports reading/writing .csv files  Excel files Cut/Paste with Excel etc Coming Soon…  Scripting interface  GUI for building plots  User defined functions –Java, scripting

June 2003 JAS – Distributed Data Analysis Miscellaneous Features Save/Restore configuration User Preferences Plugin Manager

June 2003 JAS – Distributed Data Analysis Status Currently released JAS3 version AIDA functionality is quite solid Compiler, Loader, Record Loop all quite recently added,  Certainly still some rough edges Documentation limited but available Built-in example scripts and programs Tutorial on web If you are used to JAS2 you will find some functionality not yet ported to JAS3 Remote (client/server) access to data. 3D Lego/Surface plots

June 2003 JAS – Distributed Data Analysis JAS3 and the GRID We plan to add client-server/distributed capabilities to JAS3 similar to those in JAS2 Will be based on (distributed) AIDA  Next AIDA workshop (at CERN next week) will discuss this Want to use Grid standards where they exist  Work with others (PPDG-CS11,???) to define standards where they do not exist Want to be compatible with C++ servers Tech-X have submitted phase II SBIR and if approved will work closely if approved

June 2003 JAS – Distributed Data Analysis JAS3 Links, More Info JAS – Java Analysis Studio - JAS3 – JAIDA – AIDA – FreeHEP - FreeHEP Java Libraries - WIRED –