1. eSDO Consortium Meeting
Elizabeth Auden
24 October 2006
2nd year internal review
RAL
24 October eSDO Consortium, RAL Elizabeth Auden

2. Agenda <12:00 Room booked all day all
12:00 Welcome and review of recent meetings EA, LC
12:30 Review of interactions with JSOC and AstroGrid EA
13:00 Progress report: data centre and visualization tools EA
13:30 Lunch all
14:00 Progress reports / demos: algorithms DB, TT, MS, SZ, VG
15:30 Technical update: unit tests, science tests, documentation EA
16:15 Discussion: increasing community awareness all
16:30 AOB all
17:00 finish    

3. Deliverables By end of December 2006
Completed unit tests
3 – 5 total science tests per algorithm defined
Streaming tool movie maker functionality
Remote DRMS at MSSL
So far (October 2004 – October 2006)
9 solar algorithms
7 deployed through AstroGrid
1 submitted to JSOC pipeline
3 Visualization tool prototypes
Streaming tool plus feedback report
Data centre
Network latency test report
ADS / AstroGrid DSA prototype being revamped to work with DRMS
Proposed UK data centre architecture evolving

4. Meetings AIA / HMI workshop, Monterey; Feb 2006
UK Solar Missions Forum, UCL; Feb 2006
MIST / UKSP, Aberystwyth; April 2006
Coronal Wave Meeting, UCL; July 2006
Solar Community Meeting, Manchester; July 2006
UK e-Science All Hands, Nottingham; Sept 2006
HELAS Roadmap workshop, Nice; Sept 2006

5. Update: Data Centre and Visualization Tools

6. Data Centre JSOC DRMS now available for Postgres
Postgres at MSSL; awaiting tables
AstroGrid DSA 3.0.4
DRMS currently trialled at Stanford, remote installation being configured at MSSL
Experimentation with remote DRMS / SUMS and AstroGrid underway now between MSSL and Stanford; next round of experimentation will include Lindau  

7. Data Centres Possible European Data Centre
TLAs:
DRMS: Data Resource Management System (JSOC)
SUMS: Storage Unit Management System (JSOC)
DSA: DataSet Access (AstroGrid)
EU Framework 6 support for European helioseismology data and algorithms
Possible full mirror of SDO data in Europe?
DSA / DRMS interface for AstroGrid access to full mirror of SDO data in Europe?
DRMS
DRMS
DRMS
SUMS
SUMS US
Europe
DSA UK
UK datasets

8. Visualization Tools Working Prototypes SDO Streaming Tool prototype
Image Gallery
Thumbnail Maker
(demos)
Future Work 
Movie Maker
Serverside Streaming Tool, Image Gallery and Movie Maker combined

9. Visualization Tools SDO streaming tool
User Facilities
Pan in space P
Zoom in space P
Pan in time
Zoom in time
Data Products
LoS Magnetogram
HMI continuum map
AIA images
Future Work
FFT / wavelet analysis for coronal wave investigations

10. Visualization Tools On-the-fly Image Galleries and Movies
Thumbnail Extraction P
Extract thumbnails in US from HMI and AIA science products
Image Gallery Generation P
User specifies start time, stop time, cadence, data product
Image gallery generated on the fly in web browser
Movie Generation
MPEG movie generated on the fly
Find data with AstroGrid HelioScope.
Use PLASTIC to load data into visualization tools.

11. SDO Streaming Tool Feedback
Feedback from formal and informal demonstrations at SOHO 18, HELAS workshop and All Hands Mtg.
Change navigation from arrows to “google grab”
Ensure that webstart tool works on Macs (tested on Linux and Windows; works on some Macs)
Move to serverside hosting
Add instrument specific processing
Enable movie functionality
Add coronal wave power analysis; add relevant eSDO algorithms

12. Update: Algorithms

13. Algorithm Talks and Demonstrations
RAL: Small Event Detection, Coronal Dimming
MSSL: Coronal Loop Detection, Magnetic Extrapolation, Helicity Computation
Sheffield: Local Helioseismology Inversion, Perturbation Map Generation, Subsurface Flow Analysis
 Birmingham: Mode Parameters

14. Technical Updates

15. Science Integration Tests
Developers and scientists:
Algorithm Deployment pages fulfil first half of science integration tests: compilation, execution, 1 test case.
Choose 3 – 5 total test cases for each algorithm.
Human-run end-to-end tests that showcase limits of algorithms.
Document and attach test input files on wiki deployment pages. 

16. CuTests
By 20 September 2006, nearly all algorithms had at least 1 CuTest per .C program.
By end of December 2006, all algorithms should have thorough CuTest unit testing.
Where possible, test:
Varied numerical input: all zeros, negative numbers, numbers outside valid high and low ranges, expected input
Empty files, invalid files, files with incorrect dimensions, valid files
Varied string inputs: valid flags, incorrect flags, empty flags

17. Path to Files in Algorithms
Any text or FITS file distributed with an algorithm must be called with a full path, not a relative path.
Use makefiles to configure full paths; see Coronal Loop makefile in CVS for an example.
Files with relative paths:
Dataio.c optimal_constants.h
Initialization.c Inversion.c
Bfield.c inout.h
Relax1.c
Loop.c

18. Makefiles
Makefiles have been created for all algorithms at MSSL with configurable paths to external libraries.
Makefiles, test inputs, README files and individual *.h libraries will be bundled into tarballs with C code for distribution.
External libraries, such as cfitsio, FFT3W and GSL will be linked in deployment pages and makefiles, but not included in distribution.

19. Documentation Algorithm Deployment pages look great.
Please make sure that in-code documentation is good:
Follow documentation patterns in eSDO code standard
Variable and method names should be as self-describing as possible without becoming unwieldy
Flag beginning and end of long for / if / while loops  

20. Community Awareness Try Our Software Instructions and links:
Instructions and links:
Algorithms with AstroGrid workbench
Visualization tools with HelioScope
Data Centre prototype (under construction)