The PEACE Database: Maintaining your SDDAS database access, data export and data visualisation tools R.J. Wilson and A.N. Fazakerley Mullard Space Science Laboratory Spring 2006 Version 1.1

Content Introduction Overview of the activities Recommended approach Graphic “how to” guide Explanation of each activity Note: This document tells you how to update your own local SDDAS installation, on your own machine. If you use msslAB to run SDDAS then all this is done for you.

Introduction (1) The PEACE Database uses IDFS format files, and a suite of software known as SDDAS is freely available which enables you to work with the database, export data and visualise the data The Database changes with time as new data is added and as calibrations change Processed data in the Database (SPINPAD, moments, spacecraft potential estimates etc.) may be reprocessed from time to time Tools discussed here allow you to check whether your local copy of the database is up to date, and to make updates when necessary Other tools allow you to update the SDDAS software We also provide suggestions on maintenance of your local copy of the database

Introduction (2) The Master Server for all PEACE data is at MSSL (UK) Its IP number is currently A mirror copy of the PEACE database is at SwRI (USA) Its IP number is currently It is intended to be regularly updated to track changes at MSSL

Overview of the activities Sync With Server Update your local copy of the Hierarchy from root server i.e. Tells your computer what products are available MegaMetaPromote Using your local copy of the Hierarchy, update your local copy of the data catalogue, from which you may promote data. Download Beta Apps Get the latest versions of SDDAS applications to plot/export data. Clearing out your promoted data area Stop your local Hard Disk filling up, and stops you using older data.

Recommended approach to maintenance Proposed Schedule and proposed order or execution: 1.Clearing out your promoted data area: At least monthly for Cluster, weekly for Double Star (Weekly) 2.Download Beta Apps: At least monthly (Monthly) 3.Sync With Server: Monthly, or if you hear a new virtual has been added. (Monthly) 4.MegaMetaPromote (always after Sync with Server): Weekly at least. If you’re working with very recent data consider daily. (Daily) For reference, the main MSSL group computer is updated at the frequency in blue.

Caring for SDDAS… in pictures… Select Admin Tools Sync with Server MegaMetaPromote Download Beta Apps

Select Admin Tools At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Mars Express Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Select Admin Tools At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Mars Express Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Mars Express Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Mars Express Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get a GUI like the one shown here: You can highlight just parts the you want to update, and press “OK” – or just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files) Material that is not up to date on your computer Material that is no longer on the master server, so can be deleted from your local system.

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get a GUI like the one shown here: You can highlight just parts the you want to update, and press “OK” – or just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files) When saving, it is normal to see a pop-up box asking you to confirm that you want to overwrite the current.cfg files

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII… At his point just click “ Ok ” as you want all the Cluster spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII… At his point just click “ Ok ” as you want all the Cluster spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…) At this point just click “ Ok ” as you want all the Cluster spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…) At this point just click “ Ok ” as you want all the Cluster II spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…) At this point just click “ Ok ” as you want all the Cluster II spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

Download Beta Apps At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Download BETA Apps ” Select “ Go ” Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download them.

Download Beta Apps At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Download BETA Apps ” Either select “ Go ”, or… Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download them.

Download Beta Apps At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Download BETA Apps ” Either select “ Go ”, or… Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download the ones you want.

Download Beta Apps At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Download BETA Apps ” Either select “ Go ”, or… Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download the ones you want. To finish now select “ Quit ”.

Clearing out your promoted data area When you promote data (or VIDFs, PIDFs) they go in to the hierarchy subdirectories such as those beneath: /home/sddas/data/CLUSTERII (for Project CLUSTERII) /home/sddas/data/DOUBLESTAR (for Project DOUBLESTAR) These files stay there and take up room on your hard disk. –i.e. a single sensor Cluster PEACE data file (*D) from PAD may be 10 Mb in size… If want both HEEA & LEEA for a short period (hours), over all 4 spacecraft that’s 80 Mb. –A single day Double Star PEACE 3DF data file (*D) can easily be 500 Mb, If you want both TC-1 and TC-2 then that’s 1000Mb, or 1 Gb. You local Hard Disk will quickly become full if you do not delete these files after use. This can then stop your entire computer working properly due to lack of disk space. Delete files you no longer need!!! Files to delete are…

Clearing out your promoted data area PIDFS: Only small files (~80k), relatively few of them. VIDFS: Only small files (~80k) but many. Header files: Small-ish files (up to a few Mb) but many. Data files: LARGE files (10s to100s of Mb) and many of them. Why delete the small (or large) files at all? Maybe they’ve been updated on the master server with new calibrations or re-processed. Deleting them ensures you promoted the latest version next time you use them. A brute force method of deleting all promoted files, for project CLUSTERII, assuming your local data directory is in /home/sddas/data/ : rm /home/sddas/data/CLUSTERII/*/*/*/*.pidf.v2 - delete PIDFs rm /home/sddas/data/CLUSTERII/*/*/*/*V.v3 - delete version 3 VIDFs rm /home/sddas/data/CLUSTERII/*/*/*/*[IV] - delete version 2 VIDFs, both file types rm /home/sddas/data/CLUSTERII/*/*/*/*[HD] - delete Header and Data files Similar for projects DOUBLESTAR or MARS. Do not delete anything from the Database directories, i.e. leave /home/sddas/data/CLUSTERII/*/*/Database alone. The use of wildcards illustrated above is safe as no files in the Database directory match the ending letters above. (…but if you do after all delete from the Database directories you can recover using MegaMetaPromote)

Explanation of the Activities

Anatomy of IDFS 1 In IDFS you have a hierarchy from mission down to instrument: Project > Mission > Experiment > Instrument i.e. CLUSTERII > CLUSTER-1 > PEACE > 3DF At the Instrument level you have “virtual instruments”, which are subsets of the instruments data. i.e. CP3DFH and CP3DFL “virtuals” have a set naming convention: (Project, experiment, description) CP3DFH = Cluster PEACE 3DF HEEA DP3DF = DoubleStar PEACE 3DF (single sensor) CFUNIT = Cluster FGM UNIT vector DSPRIME = DoubleStar Staff PRIME parameters Name must be unique and no more than 8 characters. DHIAPRIM = DoubleStar HIA PRIME parameters DHIDPRIM = DoubleStar HID PRIME parameters (Exception to convention: SPINAD with virtuals such as HSPAD & LSPAD)

Anatomy of IDFS 2 There are 4 types of file, VIDF, PIDF, Header and Data PIDF: Plot Interface Description File. –Just one per virtual, contains labels and unit choices. Filenames of: {virtual}.pidf.v2 - text file that you can read i.e. CPPADH.pidf.v2 VIDF: Virtual Instrument Description File. –Contains calibrations and conversions for units. –Can be many VIDFs per virtual, each file contains a start and stop time inside it. –Filename contains start time (to nearest minute). –VIDFS must not overlap in period. {virtual}{start-time}V - version2 VIDF text file (for human use) {virtual}{start-time}I - version2 VIDF binary file (for SDDAS use) {virtual}{start-time}V.v3 – version3 VIDF text file (for human & SDDAS use) {Start-time} in format: (year)(doy)hhmm, ie for 2005-Feb-07 14:00 i.e. CPPADH V.v3 HEADER and DATA file: they come as a pair, contain the actual data. –Header file is tiny, data file is BIG. –Can be up to 3 different file-pairs per day for Cluster, 1 file-pair per day for Double Star. {virtual}{start-time}H - binary file (for SDDAS use) {virtual}{start-time}D - binary file (for SDDAS use) i.e. CPPADH H & CPPADH D

Sync With Server Updates your machine with the root server that contains the latest version of the Hierarchy. i.e. including any new virtuals recently added. Modifies: /usr/local/sddas/config/DB.cfg Contains list of what servers you can promote from for each Project. (IP addresses listed rather than server names.) /usr/local/sddas/config/PROJECTS.cfg Contains list of full hierarchy for each Project. Doing a Sync with Server updates these files for you. It shows differences only between your current copy and the master server. Makes sure you have permissions to write/save these files, if not use the chmod command or run as superuser. You need to know the IP address of the server you wish to Sync to: CLUSTERII/DOUBLESTAR IP at MSSL

MegaMetaPromote In your local SDDAS Data directory (e.g. here we assume it is /home/sddas/data/ ? ) you have your Hierarchy. Under each experiment you will have a Database directory, i.e. /home/sddas/data/CLUSTERII/CLUSTER-1/PEACE/Database Within the Database directory, each virtual has 4 files in two pairs: {virtual}.HD.DBF{virtual}.HD.NDX - Header & Data list {virtual}.I.DBF{virtual}.I.NDX - VIDF list These contain what your computer thinks is available for promotion, as opposed to what is really available for promotion from the master server. Do a MegaMetaPromote to update this list from the master server. Makes sure you have permissions to write/save these files in *ALL* the Database directories, if not use the chmod command or run as superuser. This is essential as you can only promote the data that your computer thinks is available...

Download Beta Apps SDDAS is made up from a number of separate programs ( ~80!) To ensure you have the latest version of each you Download Beta Apps Download Beta Apps updates older versions of programs with newer ones from Southwest Research Institute (SwRI). Older versions are copied to /usr/local/sddas/backups rather than being over-written. Should the new version be worse (i.e. new bugs) than the last you can then restore the old one from the backups directory. Makes sure you have permissions to write/save these files in the /usr/local/sddas/backups directory, if not use the chmod command or run as superuser. Do not use “Download Applications” – this is rarely updated by SWRI, Always use the Beta Apps.