GIST 23 DWD, 27-29th Apr 2005 GGSPS development and operations Andy Smith RAL

GIST 23 DWD, 27-29th Apr 2005 Outline GGSPS development New/ongoing development Maintenance/improvements to existing software Reprocessing Operations – s/w reliability, archive stats MSG-2 preparations: h/w & s/w installation and testing Change Request: FTP of L0 data from EUM to RAL

GIST 23 DWD, 27-29th Apr 2005 New development Product download and catalogue search Prototype -> production Introduced logging – useful stats Error handling and reliability improved Eng products and plots: improved instrument monitoring New (useful! working!) monthly trend plots Updated daily plots on the way Additions to eng products (to be fed through to plots) Ingest of L0 data re-ordered from UMARF – ongoing L1.5 Geo & stray light - ongoing

GIST 23 DWD, 27-29th Apr 2005 Maintenance Archive: cataloguing speed improved by work-around for database performance problem Processing: updates to aid re-processing L15N confidence flags re-defined, new summary information in catalogue Improved reliability of reception of L0 data from EUM failsafe against disk failure/network outages Fixes to known software faults: 6 fixes installed Nov ’04 – Apr ’05 5 new ones discovered

GIST 23 DWD, 27-29th Apr 2005 Other changes TSOL jitter Moving to SFTP “push” by EUMETSAT Improves reliability, removes reliance on extra PC Catalogue database: No repeat of database “hanging” problem: 1 hang since April 2004 Performance problem remains: slow since ~1 yr ago Web access to UMARF for re-order of missing raw data Available since Nov 2004 Missed science data retrieved, awaiting ingest

GIST 23 DWD, 27-29th Apr 2005 Reprocessing Work done: Software was developed in summer 2004 Some additional changes made to existing code Remaining New machine purchased, now being set up GGSPS software being installed Detailed procedures/tools being defined May 2005: should be able to reprocess routinely, and support development/validation of science products Addition of re-ordered L0 data: further improvement

GIST 23 DWD, 27-29th Apr 2005 Future development Monthly averages of L2 fluxes: archiving of the RMIB L2 “BARG” product generation of L2 monthly means addition of clear sky fluxes to the L2 means work started on BARG product definition framework can be set up before finalising clear-sky algorithm Plus non-L2 work: Eng Plots: ongoing improvements to monitoring Geolocation: ongoing Stray light/gains: ongoing

GIST 23 DWD, 27-29th Apr 2005 Ops: reliability Operator interventions Nov - April: Level of intervention remains low: ~weekly, up to 1 month TSOL transfers (10 cases) Power cuts (2 cases + 1 knock-on air-con failure) Other s/w errors (1) Software upgrades, machine moves (5) Systems and database manager errors(3) Conclusion: the software runs well, hardware sets the limitations

GIST 23 DWD, 27-29th Apr 2005 Archive stats Catalogued L0 packets vs expected (at / minute) Month% PacketsProblems Nov94.8 RAL system dumps (33.5 hrs) Dec94.6 RAL site power cut 28 th Dec. (37.5 hrs) Jan94.8 RAL air-con failure (33.5 hrs) Feb98.0 RAL site network (11.8 hrs) Mar99.8 Data corruption (0.5 hr), RAL power (0.6 hr) Apr98.5 (25d) EUM power lost (11.5 hr)

GIST 23 DWD, 27-29th Apr 2005 MSG-2 GGSPS-2: 2 new machines purchased and installed disk archive ordered – 9TB possibility of merging GERB-1/2 data archives installed GGSPS software GERB-1 tables needed revive tools for commissioning period – no SV data Install ops/monitoring tools, eng. plots New web pages required for GERB-1?

GIST 23 DWD, 27-29th Apr 2005 MSG-2 Network lines and testing GERB-1/2 dual ops: 2 data streams over 1 line interface tests with EUMETSAT over ISDN - successful leased-line upgrade soon, repeat interface test RMIB interface test GGSPS system shake-out/commissioning

GIST 23 DWD, 27-29th Apr 2005 Change Request Transfer of raw GERB data from EUM to RAL Current: leased line, permanent connection, 1 pkt/0.6s Proposed: Secure FTP, i.e. use standard internet connection transfer files ~equivalent to GGSPS GL0 files add TSOL jitter to the GERB packet GGSPS team have discussed this in the light of concerns raised by EUMETSAT – no technical objection We need GIST/PSG approval to proceed with this change

GIST 23 DWD, 27-29th Apr 2005 Change Request Cons/EUMETSAT concerns: coding effort is required to adapt the GGSPS and IMPF current method conforms to existing EUM standard issues of security/data corruption (use secure FTP) risk of slow transfer in periods of high internet loading single-point connection at EUM (no ISDN back-up) risk due to internet connection reliability Pros: save cost of leased line and ISDN back-up SFTP is used currently for TSOL jitter transfer – proven method (at lower data rate than L0) remove reliance on separate TSOL data stream may reduce gaps in data set/reliance on UMARF

GIST 23 DWD, 27-29th Apr 2005