The PEACE Database: Maintaining your SDDAS database access, data export and data visualisation tools R.J. Wilson and A.N. Fazakerley Mullard Space Science.

Slides:



Advertisements
Similar presentations
Lecture 10 Sharing Resources. Basics of File Sharing The core component of any server is its ability to share files. In fact, the Server service in all.
Advertisements

DOS & Windows O/s Prof. Sujata Rao Less 5.
FILEMAKER SERVER SOFTWARE & REMOTE ADMINISTRATION
Endnote Tutorial The Version pictured is version 9.0 May 8, 2007.
 Use the Left and Right arrow keys or the Page Up and Page Down keys to move between the pages. You can also click on the pages to move forward.  To.
VMWare Workstation Installation. Starting Vmware Workstation Go to the start menu and start the VMware Workstation program. *Note: The following instructions.
Computing Fundamentals Module Lesson 5 — File Management with Windows Explorer Computer Literacy BASICS.
Using CD-R and CD-RW media to backup SIGNZ data. Step One … a bit of preparation.
CPIT 102 CPIT 102 CHAPTER 1 COLLABORATING on DOCUMENTS.
Scheduling Discoverer Reports Scheduling Standard Reports Printing & Re-printing Standard Reports Focus on Reports Session 2 To print: Right click Choose.
DNR-322L & DNR-326.
5-9/12/2005 CPE How to format your computer and re-install Windows XP.
Chapter 12 - Backup and Disaster Recovery1 Ch. 12 – Backups and Disaster Recovery MIS 431 – Created Spring 2006.
A+ Guide to Software, 4e Chapter 4 Supporting Windows 2000/XP Users and Their Data.
70-290: MCSE Guide to Managing a Microsoft Windows Server 2003 Environment Chapter 12: Managing and Implementing Backups and Disaster Recovery.
Command Console Tutorial BCIS 3680 Enterprise Programming.
Outlook 2007 Tips, Tricks, and Tools. Overview Main Screen Navigation Pane View Pane Reading Pane To–Do Bar Create a New Message Contacts Create a Signature.
Basic Computer Maintenance Basic Computer Maintenance Clean and Cool Deleting Temporary Files Scandisk Backup Your Data How to.
Programming with App Inventor Computing Institute for K-12 Teachers Summer 2012 Workshop.
70-290: MCSE Guide to Managing a Microsoft Windows Server 2003 Environment, Enhanced Chapter 12: Managing and Implementing Backups and Disaster Recovery.
70-293: MCSE Guide to Planning a Microsoft Windows Server 2003 Network, Enhanced Chapter 14: Problem Recovery.
COMPUTER FUNDAMENTALS David Samuel Bhatti
Designing a Classroom Web Site Using NVU Beginning Level.
Panorama High School E.G.P./ Training to Put Students’ Grades on the Website Wednesday, September 29,
Installing the SAFARIODBC.EXE For use with Excel May 3, 2002.
70-290: MCSE Guide to Managing a Microsoft Windows Server 2003 Environment, Enhanced Chapter 12: Managing and Implementing Backups and Disaster Recovery.
Chapter 18: Windows Server 2008 R2 and Active Directory Backup and Maintenance BAI617.
1 IE in the Classroom The Internet Explorer Web Browser EDW647 Internet for Educators Roger Webster, Ph.D. Millersville University Department of Computer.
Installing and Using Active Directory Written by Marc Zacharko.
Setting Up an on-line Store Tutorial Using SmartStore.biz This Tutorial assumes you have downloaded the software from This Tutorial.
FTP Server and FTP Commands By Nanda Ganesan, Ph.D. © Nanda Ganesan, All Rights Reserved.
Teacher’s Assessment Assistant Worksheet Builder Starting the Program
File Management Presented to The Glades Computer Club January 4, 2001.
Downloading and Installing Autodesk Revit 2016
11 DISASTER RECOVERY Chapter 13. Chapter 13: DISASTER RECOVERY2 OVERVIEW  Back up server data using the Backup utility and the Ntbackup command  Restore.
U3A Computing Beginners Class Leader – Brian Moore Week 2 of 10 weeks. Mondays 4:15 to 5:45 pm.
Computing Fundamentals Module Lesson 10 — File Management with Windows Explorer Computer Literacy BASICS.
Publishing Your Web Pages Ann Emmanuel SIUE Web Administrator
Using Windows Explorer Windows explorer is different from Internet Explorer It is a Windows program designed to let you explore your hard drive and manage.
XP New Perspectives on Windows 2000 Professional Windows 2000 Tutorial 2 1 Microsoft Windows 2000 Professional Tutorial 2 – Working With Files.
Installing of a VAULT server. Few recommendations: The Vault Server can be any dedicated server or normal PC operating under Windows 2003 server or Windows.
Downloading and Installing Autodesk Inventor Professional 2015 This is a 4 step process 1.Register with the Autodesk Student Community 2.Downloading the.
Chapter 10 Chapter 10: Managing the Distributed File System, Disk Quotas, and Software Installation.
GISMO/GEBndPlan Overview Geographic Information System Mapping Object.
Darek Sady - Respondus - 3/19/2003 Using Respondus Beginner to Basic By: Darek Sady.
Fall 2003Sylnovie Merchant, Ph.D. ACCESS Tutorial Note: The purpose of this tutorial is to provide an introduction to some of the functions of ACCESS in.
Using This PowerPoint This PowerPoint presentation assumes your Computer Science teacher has provided you with the InstallingJava folder, which contains.
Instructions for Uploading Points and Photos for Linking Uploading GPS points requires the following: At each site or transect note stop, “mark” a GPS.
Installing Parental Filter Version By NWSP.
Chapter 3 Automating Your Work. It is frustrating when you have to type the same passage of text repeatedly. For example your name and address. Word includes.
IT1001 – Personal Computer Hardware & system Operations Week7- Introduction to backup & restore tools Introduction to user account with access rights.
When the program is first started a wizard will start to setup your Lemming App. Enter your company name and owner in the fields designated “Company Name”
Computer Literacy BASICS: A Comprehensive Guide to IC 3, 5 th Edition Lesson 3 Windows File Management 1 Morrison / Wells / Ruffolo.
When the program is first started a wizard will start to setup your Lemming App. Enter your company name and owner in the fields designated “Company Name”
+ Publishing Your First Post USING WORDPRESS. + A CMS (content management system) is an application that allows you to publish, edit, modify, organize,
1 Week # 4 Introduction to PDM PDM is a workbench environment that lets programmers and system operators navigate the three levels of the AS/400’s object-based.
Software - Utilities Objectives Understand what is meant by utility software and application software Look at common utilities – Security – Disk organisation.
SurveyDIG 2.1 Tutorial. Tutorial Contents Introduction Introduction Item Groups Item Groups –Creating new Groups –Naming Convention –Searching/Editing.
CACI Proprietary Information | Date 1 PD² SR13 Client Upgrade Name: Semarria Rosemond Title: Systems Analyst, Lead Date: December 8, 2011.
Mail Merge for Lotus Notes and Excel User Guide
Computer Literacy BASICS
Using Mail Merge for Lotus Notes
Creating a Windows 10 Virtual machine
Basic Computer Maintenance
Microsoft Windows 2000 Professional
Macrosystems EDDIE: Getting Started + Troubleshooting Tips
What Is Google File Stream
Importing Data from Excel (XLS) into P6 Client
bitcurator-access-webtools Quick Start Guide
Review of Previous Lesson
Presentation transcript:

The PEACE Database: Maintaining your SDDAS database access, data export and data visualisation tools R.J. Wilson and A.N. Fazakerley Mullard Space Science Laboratory Spring 2006 Version 1.1

Content Introduction Overview of the activities Recommended approach Graphic “how to” guide Explanation of each activity Note: This document tells you how to update your own local SDDAS installation, on your own machine. If you use msslAB to run SDDAS then all this is done for you.

Introduction (1) The PEACE Database uses IDFS format files, and a suite of software known as SDDAS is freely available which enables you to work with the database, export data and visualise the data The Database changes with time as new data is added and as calibrations change Processed data in the Database (SPINPAD, moments, spacecraft potential estimates etc.) may be reprocessed from time to time Tools discussed here allow you to check whether your local copy of the database is up to date, and to make updates when necessary Other tools allow you to update the SDDAS software We also provide suggestions on maintenance of your local copy of the database

Introduction (2) The Master Server for all PEACE data is at MSSL (UK) Its IP number is currently A mirror copy of the PEACE database is at SwRI (USA) Its IP number is currently It is intended to be regularly updated to track changes at MSSL

Overview of the activities Sync With Server Update your local copy of the Hierarchy from root server i.e. Tells your computer what products are available MegaMetaPromote Using your local copy of the Hierarchy, update your local copy of the data catalogue, from which you may promote data. Download Beta Apps Get the latest versions of SDDAS applications to plot/export data. Clearing out your promoted data area Stop your local Hard Disk filling up, and stops you using older data.

Recommended approach to maintenance Proposed Schedule and proposed order or execution: 1.Clearing out your promoted data area: At least monthly for Cluster, weekly for Double Star (Weekly) 2.Download Beta Apps: At least monthly (Monthly) 3.Sync With Server: Monthly, or if you hear a new virtual has been added. (Monthly) 4.MegaMetaPromote (always after Sync with Server): Weekly at least. If you’re working with very recent data consider daily. (Daily) For reference, the main MSSL group computer is updated at the frequency in blue.

Caring for SDDAS… in pictures… Select Admin Tools Sync with Server MegaMetaPromote Download Beta Apps

Select Admin Tools At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Mars Express Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Select Admin Tools At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Mars Express Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Mars Express Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Mars Express Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get: You can select just parts the you want – but easier to just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files)

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get a GUI like the one shown here: You can highlight just parts the you want to update, and press “OK” – or just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files) Material that is not up to date on your computer Material that is no longer on the master server, so can be deleted from your local system.

Sync with Server At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Config Files Editor ” Select Pull Down Menu: “Action > Sync With Server…” From the list on the right, click the IP of the server you wish to sync with – and the number appears on the left. Cluster/Double Star Then select “ Ok ”. If files are already in sync it will tell you, otherwise you get a GUI like the one shown here: You can highlight just parts the you want to update, and press “OK” – or just select “ Select All ” and then “OK” You are returned to the earlier screen. Now you MUST use the pull down menu: “File > Save” DON’T FORGET TO SAVE! (and make sure you have permissions to save the files) When saving, it is normal to see a pop-up box asking you to confirm that you want to overwrite the current.cfg files

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII… At his point just click “ Ok ” as you want all the Cluster spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII… At his point just click “ Ok ” as you want all the Cluster spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…) At this point just click “ Ok ” as you want all the Cluster spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…) At this point just click “ Ok ” as you want all the Cluster II spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

MegaMetaPromote At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “Promote Meta Data” Select “ Source ” You get a list of Source Definitions to click through. Easiest to just click the top level once, i.e. click CLUSTERII (and/or DOUBLESTAR…) At this point just click “ Ok ” as you want all the Cluster II spacecraft. This returns you to earlier window, with CLUSTERII in the Source at the top. Now just click “ Go! ” and wait for it to update. [NB Do not Promote VIDFs as there are 100,000s of them.] Always do this after, not before, Sync with Server!

Download Beta Apps At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Download BETA Apps ” Select “ Go ” Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download them.

Download Beta Apps At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Download BETA Apps ” Either select “ Go ”, or… Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download them.

Download Beta Apps At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Download BETA Apps ” Either select “ Go ”, or… Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download the ones you want.

Download Beta Apps At console prompt, type: sdcontrol & Select “ Administration Tools ” Select “ Download BETA Apps ” Either select “ Go ”, or… Alternatively click the Updated Tab to see a list of files that have newer version at SWRI, then download the ones you want. To finish now select “ Quit ”.

Clearing out your promoted data area When you promote data (or VIDFs, PIDFs) they go in to the hierarchy subdirectories such as those beneath: /home/sddas/data/CLUSTERII (for Project CLUSTERII) /home/sddas/data/DOUBLESTAR (for Project DOUBLESTAR) These files stay there and take up room on your hard disk. –i.e. a single sensor Cluster PEACE data file (*D) from PAD may be 10 Mb in size… If want both HEEA & LEEA for a short period (hours), over all 4 spacecraft that’s 80 Mb. –A single day Double Star PEACE 3DF data file (*D) can easily be 500 Mb, If you want both TC-1 and TC-2 then that’s 1000Mb, or 1 Gb. You local Hard Disk will quickly become full if you do not delete these files after use. This can then stop your entire computer working properly due to lack of disk space. Delete files you no longer need!!! Files to delete are…

Clearing out your promoted data area PIDFS: Only small files (~80k), relatively few of them. VIDFS: Only small files (~80k) but many. Header files: Small-ish files (up to a few Mb) but many. Data files: LARGE files (10s to100s of Mb) and many of them. Why delete the small (or large) files at all? Maybe they’ve been updated on the master server with new calibrations or re-processed. Deleting them ensures you promoted the latest version next time you use them. A brute force method of deleting all promoted files, for project CLUSTERII, assuming your local data directory is in /home/sddas/data/ : rm /home/sddas/data/CLUSTERII/*/*/*/*.pidf.v2 - delete PIDFs rm /home/sddas/data/CLUSTERII/*/*/*/*V.v3 - delete version 3 VIDFs rm /home/sddas/data/CLUSTERII/*/*/*/*[IV] - delete version 2 VIDFs, both file types rm /home/sddas/data/CLUSTERII/*/*/*/*[HD] - delete Header and Data files Similar for projects DOUBLESTAR or MARS. Do not delete anything from the Database directories, i.e. leave /home/sddas/data/CLUSTERII/*/*/Database alone. The use of wildcards illustrated above is safe as no files in the Database directory match the ending letters above. (…but if you do after all delete from the Database directories you can recover using MegaMetaPromote)

Explanation of the Activities

Anatomy of IDFS 1 In IDFS you have a hierarchy from mission down to instrument: Project > Mission > Experiment > Instrument i.e. CLUSTERII > CLUSTER-1 > PEACE > 3DF At the Instrument level you have “virtual instruments”, which are subsets of the instruments data. i.e. CP3DFH and CP3DFL “virtuals” have a set naming convention: (Project, experiment, description) CP3DFH = Cluster PEACE 3DF HEEA DP3DF = DoubleStar PEACE 3DF (single sensor) CFUNIT = Cluster FGM UNIT vector DSPRIME = DoubleStar Staff PRIME parameters Name must be unique and no more than 8 characters. DHIAPRIM = DoubleStar HIA PRIME parameters DHIDPRIM = DoubleStar HID PRIME parameters (Exception to convention: SPINAD with virtuals such as HSPAD & LSPAD)

Anatomy of IDFS 2 There are 4 types of file, VIDF, PIDF, Header and Data PIDF: Plot Interface Description File. –Just one per virtual, contains labels and unit choices. Filenames of: {virtual}.pidf.v2 - text file that you can read i.e. CPPADH.pidf.v2 VIDF: Virtual Instrument Description File. –Contains calibrations and conversions for units. –Can be many VIDFs per virtual, each file contains a start and stop time inside it. –Filename contains start time (to nearest minute). –VIDFS must not overlap in period. {virtual}{start-time}V - version2 VIDF text file (for human use) {virtual}{start-time}I - version2 VIDF binary file (for SDDAS use) {virtual}{start-time}V.v3 – version3 VIDF text file (for human & SDDAS use) {Start-time} in format: (year)(doy)hhmm, ie for 2005-Feb-07 14:00 i.e. CPPADH V.v3 HEADER and DATA file: they come as a pair, contain the actual data. –Header file is tiny, data file is BIG. –Can be up to 3 different file-pairs per day for Cluster, 1 file-pair per day for Double Star. {virtual}{start-time}H - binary file (for SDDAS use) {virtual}{start-time}D - binary file (for SDDAS use) i.e. CPPADH H & CPPADH D

Sync With Server Updates your machine with the root server that contains the latest version of the Hierarchy. i.e. including any new virtuals recently added. Modifies: /usr/local/sddas/config/DB.cfg Contains list of what servers you can promote from for each Project. (IP addresses listed rather than server names.) /usr/local/sddas/config/PROJECTS.cfg Contains list of full hierarchy for each Project. Doing a Sync with Server updates these files for you. It shows differences only between your current copy and the master server. Makes sure you have permissions to write/save these files, if not use the chmod command or run as superuser. You need to know the IP address of the server you wish to Sync to: CLUSTERII/DOUBLESTAR IP at MSSL

MegaMetaPromote In your local SDDAS Data directory (e.g. here we assume it is /home/sddas/data/ ? ) you have your Hierarchy. Under each experiment you will have a Database directory, i.e. /home/sddas/data/CLUSTERII/CLUSTER-1/PEACE/Database Within the Database directory, each virtual has 4 files in two pairs: {virtual}.HD.DBF{virtual}.HD.NDX - Header & Data list {virtual}.I.DBF{virtual}.I.NDX - VIDF list These contain what your computer thinks is available for promotion, as opposed to what is really available for promotion from the master server. Do a MegaMetaPromote to update this list from the master server. Makes sure you have permissions to write/save these files in *ALL* the Database directories, if not use the chmod command or run as superuser. This is essential as you can only promote the data that your computer thinks is available...

Download Beta Apps SDDAS is made up from a number of separate programs ( ~80!) To ensure you have the latest version of each you Download Beta Apps Download Beta Apps updates older versions of programs with newer ones from Southwest Research Institute (SwRI). Older versions are copied to /usr/local/sddas/backups rather than being over-written. Should the new version be worse (i.e. new bugs) than the last you can then restore the old one from the backups directory. Makes sure you have permissions to write/save these files in the /usr/local/sddas/backups directory, if not use the chmod command or run as superuser. Do not use “Download Applications” – this is rarely updated by SWRI, Always use the Beta Apps.