1. EVLA Software - Overall Architecture, Science & Online Domains Bryan Butler NRAO

2. 2006May08EVLA Advisory Committee Meeting2 Telescope Data Model Export Data Format Science Data Model Feedback to telescope Proposal Submission And Handling Observation Preparation EVL A VLBAALMAGBT EVLA Sched EVLA Control ALMA Sched ALMA Control GBT Sched GBT Control Data Capture Archive Telcal OfflineVO Observer Domain Mostly Telescope- Independent Common Software VLBA Sched VLBA Control Quick Look Pipeline GBT Postproc Telescope Domain Science Domain Mostly Telescope- Specific Project Software Mostly Telescope- Independent Common Software NRAO-wide Design

3. 2006May08EVLA Advisory Committee Meeting3 NRAO-wide Design The EVLA software system needs to conform to the NRAO-wide design to the extent that it has been developed. Unfortunately, the NRAO-wide effort has languished for the past two years (this is changing - a new “e2e Project Manager” [Nicole Radziwill] and “e2e Project Scientist” [Ed Fomalont] have been hired in CV).

4. 2006May08EVLA Advisory Committee Meeting4 Domains & Models The NRAO-wide design introduced the concept of “domains” - different (large) pieces of the system that can be grouped sensibly. It presented three such domains: Observer Telescope Science It also presented a number of “models” - descriptions of different (smaller) pieces of the system: Observatory Project Observing Archive Telescope Data Science Data

5. 2006May08EVLA Advisory Committee Meeting5 EVLA Overall Design In the spring of 2004, the EVLA software group undertook an effort to develop and document a high level design (led by Morgan, Ryan, Sowinski, and Waters). This culminated in a completed design, which was reviewed by the NRAO eOC in June 2004. The design presented in the next few slides is based mostly on that effort, with extension and modification in the past two years.

6. 2006May08EVLA Advisory Committee Meeting6 EVLA Software Requirements The software design and implementation is driven by a number of requirements documents: e2e Science Software Requirements Engineering Software Requirements Real-time System Software Requirements Operations Software Requirements These do not have everything in them (for instance Proposal Handling and User Database, which are covered in separate [less “requirementy”] documents), but are fairly complete, and notably, the e2e requirements document has extensive requirements for PST and OPT.

7. 2006May08EVLA Advisory Committee Meeting7 The main flow of information (and processes; the “workflow” or “dataflow”) is: Major Elements (“Models”) Proposal Project(s) Program(s)Schedule(s) Commands A Scheduling Block is an atomic unit of observing. It is made up of a sequence of scans; a scan is made up of source(s), resource(s) (hardware definition - both FE and BE), timing information, and a “mode”. The mode defines the subscan(s), which are comprised of a single source, resource, and timing information. Data

8. 2006May08EVLA Advisory Committee Meeting8 EVLA High Level Architecture DATAFLOW

9. 2006May08EVLA Advisory Committee Meeting9 EVLA High Level Architecture Note that most of the major subsystems have a direct counterpart in current VLA software - we have a significant amount of experience in what is needed there. What has been lacking is the electronic storage and passage of information between subsystems, and therefore the ability to do much of this automatically.

10. 2006May08EVLA Advisory Committee Meeting10 Observation Preparation Tool (OPT) Astronomer or Staff Project EVLA Observing Heuristics Program Block (Set of Scheduling Blocks for one Program) Proposal Submission Tool (PST) To Observation Scheduling Tool EVLA Design Detail, Pre-Observing Science Domain Portal Proposal Handling Tool (PHT) Proposal Authenticated Astronomer or Staff

11. 2006May08EVLA Advisory Committee Meeting11 EVLA Design Detail, Pre-Observing Online Domain Observation Scheduling Tool (OST) Executor Next SB Execution State Equipment State Metadata to DCAF Operator Environment From OPT Results from TelCal Sequence of Configurations Antenna Delays Archive Operator Heuristics Metadata to DCAF To AMCS & CMCS From AMCS & CMCS

12. 2006May08EVLA Advisory Committee Meeting12 EVLA Design Detail, Real-Time Domain Hardware M&C AMCS CMCS RF EVLA Antennas FOTS Receiver Station, Baseline Boards Lag Frames CBE State Counts Raw Vis Equipment State, Data Addressing Info, Messages, Alerts, etc. From Executor FF To Archive & TelCal To DCAF

13. 2006May08EVLA Advisory Committee Meeting13 EVLA Design Detail, Post-Observing Online Domain SDM Data Capture And Format (DCAF) From CMCS TelCal SDM From AMCS & CMCS To Executor And Archive To Archive Quick Look Pipeline (QLP) Astronomer or Operator Observation Status Monitor Tool (OSMT) M&C Archive Portal Authenticated Astronomer or Operator M&C Archive To Archive (?) TelCal Results

14. 2006May08EVLA Advisory Committee Meeting14 From DCAF Post-Processing Image Cubes VO Astronomer Default Image Pipeline (DIP) Cubes (?) From CMCS EVLA Design Detail, Post-Observing Science Domain Archive Archive Search and Retrieval Tool (ASRT) Astronomer Portal Authenticated Astronomer Reprocessed Proprietary Products Existing Proprietary Products Open Products Open Products Trigger

15. 2006May08EVLA Advisory Committee Meeting15 Detailed Subsystem Designs & Prototypes For most of the major subsystems, we either have a very advanced prototype (Portal, PST, Executor, OSMT), an early prototype (PHT, OPT, OST, ASRT, TelCal), or at least a much more detailed design (DCAF). The areas where we have done little work and in fact have only preliminary (high level) designs are for the pipelines (QLP, DIP).

16. 2006May08EVLA Advisory Committee Meeting16 Portal, User Database, Authentication We know that we need some way to restrict access to the various tools, and the information available within them. We do this with a “portal”, through which users access the various tools. It authenticates users, and generates a unique token which is then used to verify a user’s login status. We have our own implementation of this, but recognize that we may need to migrate to the VO method (or have a translation layer).

17. 2006May08EVLA Advisory Committee Meeting17 Portal, User Database, Authentication

18. 2006May08EVLA Advisory Committee Meeting18 Portal, User Database, Authentication Users enter their own User Database information, except: Institution information - they can only select an institution from a pre-set list (we want to use this to automatically generate reports to the NSF, which require precise information on institutions) - if the institute is not there, they indicate so, and we (well, I) add it. We allow so-called “3rd party” user registrations during proposal preparation and submission, adding significant complexity (a sore spot with us, but demanded by operations).

19. 2006May08EVLA Advisory Committee Meeting19 Proposal Submission Tool (PST) Mainly a tool to collect form data (web browser) Mostly telescope independent, with “resources” the exception Used to enforce telescope “policy” Coupled to other EVLA tools only loosely, via Portal and databases. Currently also supports GBT.

20. 2006May08EVLA Advisory Committee Meeting20 PST - Main Processes Main Processes: Model - retrieve and write data to database Controller - business logic to map user input (from browser) into objects which are then written to database View - the look-and-feel of the interface (done in browser) Validation of various fields - an important and significant part of the tool Help system

21. 2006May08EVLA Advisory Committee Meeting21 PST - Model The Model drives everything, so what’s in there? science information - title, category, “mode”, abstract, scientific justification, and some misc. info. Authors, including which is the PI and “contact author” Sources Resources (telescope hardware setup) “Sessions” (a guide to SB setup) Student Support Essentially, everything that is necessary to: Referee the proposal Assign telescope time (and money) Automatically generate SBs (mostly for novice users, but experienced users will use this too!)

22. 2006May08EVLA Advisory Committee Meeting22 PST - “The View” Although the logic is driven by the model, most of the programming complexity here is in the view. How do we do this? 6 main “tabs” in the browser, to represent the 6 main parts of the model. There is also some superstructure, to allow for higher level functions (edit/create, submit, print, choose telescope, etc.), but, again, most of the complexity is in the view for these 6 tabs.

23. 2006May08EVLA Advisory Committee Meeting23 PST - “The View” Let’s look at the actual tool…

24. 2006May08EVLA Advisory Committee Meeting24 PST - Submit We must support both normal “deadline” submissions, as well as “Rapid Response” submissions (this is fundamentally a policy issue). Upon submission, the proposal handling process is initiated. What is stored is a “proposal” (as represented by a Proposal Data Model). This is NOT the Project Data Model, but rather is a guide to the creation of the initial PDM. This is an important distinction, and something we came to a recent agreement on with ALMA (for them this is the Science View of the PDM).

25. 2006May08EVLA Advisory Committee Meeting25 Proposal Handling Tool (PHT) Presently, only very initial “wrangling” (view, print), and then splits into GB and VLA specific handling GB and VLA separately support: –Adding additional data (edit XML by hand or script) –Fixing ‘problems’ –Pretty-printing, for referees and scheduling committee Future: –Referee process –Scheduling committee details Points of interest: –Requirements for VLA are in hand, but not in the form of the detailed requirements for other areas, rather more like a “User Story”. We need to transform this to real requirements, including the GB process (which have not been written down to our knowledge). –Does this go in the PST (with maybe part in the OPT) or as a separate tool? The fundamental output from the PHT is Projects, as represented by a Project Data Model.

26. 2006May08EVLA Advisory Committee Meeting26 Observation Preparation Tool (OPT) This is the process that takes a more generic view of a Project, and turns it into something that can actually be used to command the hardware of the telescope (and run ancillary tasks). The fundamental output of the OPT is Program Blocks (PBs) (remember that a PB is a collection of SBs, with some extra information - mostly “contingencies”) It needs to support 3 “levels” of user: Novice (automatic generation of PBs for “standard modes”) Intermediate (this is the tough one!) Expert (allow for script level editing)

27. 2006May08EVLA Advisory Committee Meeting27 OPT - commonality with PST Objects in common with PST: Sources Resources (hardware configuration) Things not in common: Things not in OPT: –Front page stuff –Authors –Sessions (well, kind of - since they “represent” SBs) –Student Support Things not in PST –Scan builder and organizer –PB organizer –Detailed correlator setup tool –Calibrator tool

28. 2006May08EVLA Advisory Committee Meeting28 OPT - Detailed Design

29. 2006May08EVLA Advisory Committee Meeting29 OPT - Detailed Design Modify PB

30. 2006May08EVLA Advisory Committee Meeting30 OPT - Detailed Design Modify SB

31. 2006May08EVLA Advisory Committee Meeting31 OPT - Current Status We currently have an early prototype of the GUI (duplicating the look-and-feel of the PST) in place which will authenticate users and has the most minimal PB functions (create, delete, etc.). Much of our work here, however, has been on the specification of the details of the objects (the “data models”) for the various parts of the system. We are starting here, knowing that many of the elements in the system will be needed here and will be common. These include the definitions of Proposal, Project, Program Block, and Scheduling Block, and as parts of that, Sources, Resources, and Scans. We start out with an XML document provided by a domain expert, and then turn that into objects. We are currently having detailed discussions with ALMA to attempt to have as much as common in these objects. It is clear that early parts of the process (Proposal) can be common, and that general concepts (major elements of SBs, for instance) can be common; it is not yet clear the level to which true commonality will be achieved.

32. 2006May08EVLA Advisory Committee Meeting32 OPT - Plan Our plan is to have new major functionality available within the OPT roughly every 3 months. Major milestones are: 30Apr06: framework with minimal functionality 31Jul06: Add VLA calibrator database access, initial spectral setup 31Oct06: Full calibrator setup, more observation setup 31Jan07: VLA mostly supported. Some validation/PB creation 30Apr07: Beginning prototype WIDAR support 31Jul07: VLA fully supported; prototype WIDAR mostly supported 31Oct07: Prototype WIDAR fully supported

33. 2006May08EVLA Advisory Committee Meeting33 Observation Scheduling Tool (OST) We (well, Barry) have been conducting tests of dynamic scheduling with the VLA during the past 2 (3?) reconfigurations. Again, we consider these as prototypes for the final EVLA subsystem - giving us invaluable information on the practical aspects of dynamic scheduling of a many-element radio interferometer.

34. 2006May08EVLA Advisory Committee Meeting34 OST - Block Diagram

35. 2006May08EVLA Advisory Committee Meeting35 OST - Lessons Learned Here are the lessons learned - I need the full list from Barry. A few things I know: ALMA system didn’t work well (as expected, per Allen Farris’ email) System is inordinately fond of short SBs Currently effort-intensive (but getting better)

36. 2006May08EVLA Advisory Committee Meeting36 OST - Plan Here is the plan. I need to get with Barry on this, but my understanding is that he wants the VLA fully dynamically scheduled by the end of ‘06 (yes, he has indeed said that!). For the EVLA-specific part, we are assigning some effort beginning summer ‘06 to this.