1. Simulation Production System
Science Advisory Committee Meeting
UW-Madison
March 1st-2nd 2007
Juan Carlos Díaz Vélez

2. outline introduction production tools job management & error handling
production sites & computing
production challenges
data storage and transfer

3. introduction SimProd written in Python
interfaces easily with IceTray through Boost
IceTray configurations written in well formatted XML are easy to store in database
daemons manage cluster job submission
SOAP interface for GUI client and for job monitoring

4. production tools GUI Production Client (& TUI)
designed for IceTray configuration and job submission (simulation & offline processing)

5. production tools GUI Production Client (& TUI) Production Database
designed for IceTray configuration and job submission (simulation & offline processing)
Production Database
store production history including all configured module parameters
provide information on configurable parameters for client

6. production tools GUI Production Client (& TUI) Production Database
store production history including all configured module parameters
provide information on configurable parameters for client
Production Server
accepts dataset requests from client
Provides job management including error handling
Separate daemons handle dataset submission, queue/job management & monitoring

7. production tools GUI Production Client (& TUI) Production Database
Production Server
accepts dataset requests from client
Provides job management including error handling
Separate daemons handle dataset submission, queue/job management & monitoring
Queuing Plugins(s)
adaptable to different sites and batch systems

8. production tools GUI Production Client (& TUI) Production Database
Production Server
Queuing Plugins(s)
adaptable to different sites and batch systems
Logging/monitoring Database
production status & troubleshooting
remote job management
unified monitoring for multiple clusters

9. production tools GUI Production Client (& TUI) Production Database
Production Server
Queuing Plugins(s)
Logging/monitoring Database
production status & troubleshooting
remote job management
unified monitoring for multiple clusters
Web Interface (Ian Rae)
cluster/dataset/job monitoring
search engine for production db
dataset statistics

10. job management job goes through series of states
Web Interface (Ian Rae)
job/server communication
job eviction
file transfer error

11. production sites
current simulation production grid

12. computing different architectures and OS and batch systems
different policies
each site provides a local contact person
work with local sys admin
maintain production
monitor runtime & completion
troubleshoot system
check data integrity

13. photonics
photon interaction probability tables are produces with detailed module of ice properties
full set of tables is ~14 GB (too large for memory (32-bit)
we sort events in zenith bins and process process each bin separately.
current production clusters have tables pre-installed on nodes
This limits our ability to add new clusters or large grids for simulation production.

14. data storage archive documented through DIF metadata
DIF (Directory Interchange Format) adapted to Astrophysics
SimProd automatically generates DIF from simulation parmeters.
pending: interface simprod with datawarehouse Ingest system
files collected from sites and stored at UW
local responsibles manually transfer data to UW
pending: automatic data movement from sites (testing gridftp)

15. work in progress some features planned for SimProd
automatic file transfer (GridFTP)
dynamic collection and reporting of detailed simulation statistics.
better user interface.
Search engine will allow user to display datasets based on different criteria (e.g. geometry, primary spectrum, grid site, etc.)

16. links Simulation Production Web Page
sim-prod documentation and wiki