1. IOCP Key Point 0 © Rafa Leon Guillermo Buenadicha OPS OEX SMOS LEOP & SODAP OPERATIONS © Rafa Leon

2. Guillermo Buenadicha OPS OEX SUMMARY LEOP OPERATIONS PRESODAP OPERATIONS SODAP OPERATIONS FUTURE OPERATIONS CONCLUSIONS FOS CONFIGURATION SOURCE FOS INPUTS and EADS DASA REPORTS

3. Guillermo Buenadicha OPS OEX MIRAS LEOP OPERATIONS LEOP phase comprised from Launch up to Payload Switch On, i.e. from the 02/11/09 to 17/11/09 Main events for the payload in the period were: Partial SW on, Arm deployment Thermal control through Arm Segment Heater Lines Nominal Pointing of Instrument + ASHL control Phase manned with teams (electrical, system, thermal) both at CNES and at ESAC. Main source of data, exception made of the one of Partial SW ON, was the Proteus E_HKTM. Files received at ESAC after each S_Band visibility.

4. Guillermo Buenadicha OPS OEX MIRAS LEOP: PARTIAL SW ON NIR TEMP Hinge Temperatures (both for the hot and the cold hinges) within the model predictions and thermal range  go ahead for deployment PLPC capable of data acquisition and display within few minutes OBT LO LOCK

5. Guillermo Buenadicha OPS OEX MIRAS LEOP: ARM DEPLOYMENT ARMON Pulse NOMON Pulse REDOFF Pulse NOMNOMRELEASEDOFF REDREDEOT SW 3SW 3 travelEOT SW 2SW 2EOT SW 1 = ARM DEPLOYEDRELEASED to DEPLOYED A18:00:51.69618:00:51.94618:00:52.4460.75018:00:52.44618:00:52.9461.0018:03:39.446167.018:03:55.196182.75018:04:12.446200.0 B18:00:53.19618:00:53.44618:00:53.9460.75018:00:54.07118:00:54.4461.0018:03:31.946157.87518:03:47.571173.518:04:06.946192.875 C18:00:54.69618:00:54.94618:00:55.4460.75018:00:55.44618:00:55.9461.0018:03:30.696155.25018:03:46.446171.018:04:04.571189.125

6. Guillermo Buenadicha OPS OEX MIRAS LEOP: THERMAL ASSESSMENT Partial SW ON Deployment Cycling ASHLSW ON 10 deg SW ON 22 deg NOM Attitude ARM TEMPERATURES

7. Guillermo Buenadicha OPS OEX MIRAS LEOP: NORMAL POINTING + ASHL C2 Thermostat problem. For that segment, the nominal ASHL line did not trigger but the redundant did instead ASHL Current Lines drained above foreseen limits (originally 4.2 amps for 4 segments). C2 analyzed and reported in ARTS (SMOS-21). Monitored further after manoeuvres and no change. Suspected explanation is different set points for Nom and Red thermostats than the ones in specs. Close monitoring in case of use of Backup-1 configuration SW ON.  no operational impact Current Lines limits changed to 4.4 amps, no further triggering.

8. Guillermo Buenadicha OPS OEX MIRAS PRESODAP This phase went from Instrument SW On the 17/11/09 up to formal start of SODAP on the 23/11/09 SW On implemented through 3 different visibilities, final 22 degrees setting done on 18/11/09 in the morning. Activities in the period: SW ON Payload monitoring FCP-SDP-100 First Data Acquisition  Auto downlink mode left enabled Commanding tests ITL management Instrument commanding test OBOP test

9. Guillermo Buenadicha OPS OEX MIRAS PRESODAP: PLM MONITORING SW ON fully nominal, some minor remarks The PLM TM SSC between 136 and 205 were not received inside S- Band packets. Packets were received in X_Band PUS TM. To be understood as an effect of the MIL BUS initialization. Received the event “652: Time_Correlator_Unexpected_UTC” with parameter "UTC_Received". Not seen in the partial SW ON, possibly due to the PLM reaction after first GPS received. During GPS absence it has been noticed that there are two values inside TM (3,25) that are not consistent. SPC10107 PPS_ERROR_FLAG shows "valid" (row 0) all the time DPC10107 UTC Cur Source shows "GPS" (raw 0) all the time The TM value First Scrub Finished DMASME38 has a behaviour not understood (looks like wrong SW initialization). It triggers, however, at the end of first scrubbing. “Stripe” effect of first scrubbing now fully understood and nominal.

10. Guillermo Buenadicha OPS OEX MIRAS PRESODAP: OBT CLOCK The OBT has a (foreseen) dependency on temperature. The OBT frequency is faster than the quoted 65536 hz (still meeting specs). This created problems in the first week to the PLPC time correlation algorithm (too frequent time correlation resets). Stability over time looks however good. PLPC implements currently a workaround that allows a maximum deviation up to 10 msec. A permanent fix to characterise the clock slope is being developed. Caveat is that re-ingestion of the same data causes duplicates in different packet times. Also, under non nominal conditions (i.e. re ingestion of CCU reset data) the time grows exponentially, forcing a cleanup.

11. Guillermo Buenadicha OPS OEX MIRAS PRESODAP: COMMANDING TESTS In this phase the commanding to the PLM was verified, in an incremental manner. Driven by the FCP procedures FCP-SDP-xxx Initial direct and time tagged ping connection TC’s Enabling of the ITL queue and Thermal Settings reporting. Complex ITL management (insertion, partial and complete deletion, clearing) Instrument commanding test FCP-SDP-500, forcing generation of reports and exercising different PUS services On Board, including command rejections. OBOP test, exercising all the calibration OBOPs and also the cyclic functions On Board. All tests successful. No unforeseen TC failures.

12. Guillermo Buenadicha OPS OEX MIRAS PRESODAP: ACQUISITION The PLM autodownlink function was left enabled on purpose after SW ON. Therefore, ESAC XBAS was in charge of data acquisition. Autodownlink disabled on 19/11/09 before the afternoon passes. The passes on 20/11/09 were commanded using SPGF, in order to ensure proper data acquisition of the OBOP test for PLM data calibration purposes. All acquisitions were successful. The instrument mode was changed to Full Pol Mode on 20/11/09 to prepare a saturated Mass Memory for SODAP tests.

13. Guillermo Buenadicha OPS OEX MIRAS PRESODAP: CAVA & CAKE MEETING

14. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS Already covered and defined by M. Zundo. FOS support to Mission Planning generation through SPGF FOS support to sequence of commanding, since most of the SODAP tests involved later manual modification of the SPGF planning outputs Support to analyze the data downlink statistics Support to analysis of data gaps and identification of problems Reorientation of some tests to re-dump given mass memory locations for MM P3 analysis. No unforeseen command failures in the period. The mission planning and the command generation in the first 2 weeks of the period were based on a daily cycle. Extra S_Band passes provided by CNES to ensure it.

15. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PLM ANOMALIES Even though some pertain to PRESODAP period, they were all characterised and analyzed during SODAP. RAM single bit errors (3) MM single bit errors (8) CMN Unlock (6) CMN A2 drift (3) Mass Memory Partition 6 (7 th) Latch Up (1) MM Partition 3 (4 th ) gap problems RAM and MM single bits will be monitored along mission for radiation statistics, no clear correlation with geographical areas or weaker MM zones.

16. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: SEU SUSPECT ANOMALIES MM Single Bit errors RAM Single Bit errors CMN Unlocks MM Partition LU Latitud e Longitu deNameDescription 63.99-95.66SB MM21/11/2009 -46.72-1.44SB MM22/11/2009 77.08-159.44SB MM22/11/2009 -9.36-57.58SB MM25/11/2009 -60.0462.43SB MM30/11/2009 -0.70133.00SB MM01/12/2009 54.00-77.00SB MM05/12/2009 67.5072.00 SB RAM17/11/2009 51.00-53.00 SB RAM25/11/2009 21.27-30.00 SB RAM01/12/2009 -68.6597.47CMNC3 - 19/11/2009 -7.47-27.20CMNA2 - 19/11/2009 -62.39102.39CMNH1 - 24/11/2009 -9.20-33.40CMNA1 - 30/11/2009 -37.00-43.00CMNA2 - 01/12/2009 -78.00-150.00CMNC2 - 03/12/2009 -25.00-11.00 LATCH UP02/12/2009

17. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN UNLOCK CMN Unlock: 6 cases observed The ones in arm C show only Unlock flag, whereas those in Arm A (A1, A2, H1) have both Unlock and LO Out Power flag Characterised by flag triggering, and sudden drop plus recovery of the PMS values. ARTS anomaly recorded SMOS-22 No events since 03/12/09: 150 hours Guinness record!!! CMNDateTimeLatLong C319/11/2009 10:01:27.193-68.6597.47 A219/11/2009 19:44:29.555-7.47332.80 H124/11/2009 10:05:00.333-62.39102.39 A130/11/2009 08:21:21.518-9.20326.60 A201/12/2009 20:25:45.437-37.00317.00 C203/12/200900:57:27.234-78.00210.00

18. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN UNLOCK FOR A2 The lock after unlock is regained in all 6 cases. However, later analysis showed that A2 did not lock on the proper frequency in both cases. Analysis of frequency locking can only be done by correlation data processing, not through HKTM telemetry. After consultation with Project and Industry, the A2 recovery was defined through a switch off and on of complete arm A  CNES intervention since switch is Proteus. Procedure defined PRO-CRP-300, validated in MIRASIM and EM. Switch off and on performed on the 01/12/09 at 13:50z. Later analysis showed that the Arm A started in default LICEF-U configuration, procedure updated to include a cycle calibration/dual post switch triggering. Open question about ARM switch qualified lifecycles Current operational system procedure not formalised  the A2 unlock requires later processing of data to ascertain proper locking (L1PP & SPQC changes in progress). Also procedure involves CNES operation, possibility to include it on current OIA as payload contingency.

19. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: A2 RECOVERY ARM A PMS ARM B PMS 01/12/09 13:50

20. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN A2 DRIFT The A2 LO Out Power flag has triggered 4 times in 3 distinct periods, always related to External Pointing Manoeuvres. Not seen in ECM on 08/12/09, after payload SW off and on 03/12/09 !!! In those cases, the LO Locking Flag has not triggered. This effect has not been observed in any of the other segment CMN’s. The PMS behaviour is not that of a sharp and sudden decrease, but rather a lowering of its value drifting away from the values of the adjacent segments in the arm The LO Out Flag duration is proportional to the manoeuvre duration. In the 3 cases recorded (1 st, 2 nd and 3 rd of December) the drift away took place over the SAA region

21. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN A2 DRIFT CASE ON 01/12/09

22. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN A2 DRIFT CASE ON 02/12/09

23. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN A2 DRIFT CASE ON 02/12/09

24. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN A2 DRIFT ARM A SW Off 2 nd A2 Unlock 2 nd ECM 3 rd ECM 1 st ECMPLM SW On

25. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN A2 DRIFT

26. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN A2 DRIFT Last OCM First A2 Unlock No evidence found, though, of thermal correlation for the other CMN unlocks

27. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: CMN A2 DRIFT 08/12/09

28. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PARTITION 7 LATCH UP A partition 7 latch-up took place on the 02/12/00 at 06:59z The effect is that the MM disables that partition. Reason for latch up is protection against over-current. The geo-location of the event shows it over the SAA. No associated CCU over-current detected Recovered through payload SW OFF/ON on 03/12/09 at 15:40. Performed in one single pass, including later reprogramming of ITL  CRP-SYS-100

29. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PARTITION 7 LATCH UP A known SW problem prevents the high level SW (ASW) from proper managment of a reduced MM size. The LLSW is capable, though, of proper handling. The impact is that X Band dumps that request a non valid new logical address are failed, and also that the write and read stop at (MM max size – 1 partition) not further progressing. Ongoing tests in EM + analysis of possible SW change would be required. ARTS SMOS-24

30. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PARTITION 3 GAPS MM scrubbing function has been reporting Double Bit errors since beginning of mission In parallel, the SODAP data analysis has found data gaps in X Band data. Once analyzed, the gaps and the DB errors seem to be both related to a given MM partition, P3 (4 th out of 12), and inside that partition, to an specific memory chip (7 th ), and even inside that chip, to an specific area 20 Gbits 1.6 Gbits 256 Mbits ~ 32 Mbits ~ 38xxxx to 4Bxxxx

31. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PARTITION 3 GAPS Findings so far: Sometimes (not always), data is not written in some locations of the affected area, but the write pointer is updated. In those cases, the scrubbing function flags double bit errors when passing the first time after the write on the area, but not the following ones. The data in the non written areas is that of the last writing. The writing gap respects the science block, i.e. always 24 science packets are written before the gap. The gap duration is variable but less than one epoch. The proper data after gap starts with a reduced data block.

32. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PARTITION 3 GAPS Scrubbing cycle is around 70 minutes. The scrubbing resolution is around 1520 MM packets Writing over the region depends on instrument mode (around 35 hours in Full mode, and 60 in Dual) The gaps found in TM and the addresses reported in the scrubbing are not the same. The gaps are not consistent across different write passes on the area. Under investigation, one case of Partition 5 DB happened on the same inner chip and area, but no data gap found. P5 and P3 share the same slave FPGA controller Some cases of DB EDAC have been triggered upon read operations over the area, but again not always. Problem reported in ARTS (SMOS-23)

33. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PARTITION 3 GAPS DOYDateTimePartitionDBSBDB AdressHEXDeviceAddressMM Packet 32218/11/200910:00P3 affected region crossing1723794 32218/11/200910:11:0431010462362063C6E0463C6E041718444 32218/11/200910:11:08310105171971644CC03644CC031722130 32420/11/200909:57:233401049958406421C006421C001720946 32420/11/200909:57:253176197105265451646392B646392B1722759 32420/11/200911:07:123992310526540164638F964638F91722759 32420/11/200919:58P3 affected region crossing1716478 32420/11/200920:02P3 affected region crossing1726730 32420/11/200920:25:4531010559867964B4ED764B4ED71724999 32622/11/200907:12P3 affected region crossing1721070 32622/11/200907:19:29310104989734642042664204261720905 32622/11/200907:19:303111105172050644CC52644CC521722131 32622/11/200907:19:3238610526383964632DF64632DF1722748 32723/11/200918:25P3 affected region crossing1722999 32723/11/200919:35:5131583710465561663CEB0063CEB001718659 32925/11/200905:30:00P3 affected region crossingNO GAP 33026/11/200913:17:2251210430075163780CF63780CF2618577 33026/11/200917:00:00P3 affected region crossing1728192 33026/11/200917:46:0234561052661236463BCB6463BCB1722763 33228/11/200904:15P3 affected region crossingNO GAP 33329/11/200915:18P3 affected region crossing1722744 33329/11/200916:03:5532420104380760638B958638B9581716811 33329/11/200916:03:5835410474593063E4BCA63E4BCA1719266 33501/12/200902:32P3 affected region crossing 33602/12/200913:42P3 affected region crossing 33602/12/200914:12:3537314105235711645C4FF645C4FF1722559 33602/12/200914:43:393268105174066644D432644D4321722145 33602/12/200914:43:413476105235711645C4FF645C4FF1722559 33804/12/200912:50P3 affected region crossing 33804/12/200913:01:363155105080861643681D643681D1721518 33804/12/200913:01:36333221052051956454DCB6454DCB1722354 34107/12/200907:00P3 affected region crossing 34107/12/200908:33:053339105235659645C4CB645C4CB1722559

34. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PARTITION 3 GAPS

35. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PARTITION 3 GAPS

36. Guillermo Buenadicha OPS OEX MIRAS SODAP OPERATIONS: PARTITION 3 GAPS

37. Guillermo Buenadicha OPS OEX FUTURE OPERATIONS & ANOMALY INVESTIGATION MIRASIM + EM operational temperature reduction procedure validation in case it is needed. Definition of a system (i.e. DPGS + FOS + CNES) procedure to handle A2 recovery from wrong unlock. Further data gathering and characterization of P3 anomaly. However, detailed data analysis outside the scope of SODAP will be difficult. EM and MIRASIM validation of Latch up recovery without Power Off and On. Implementation of further limit monitoring and trend analysis. Required a better geo location capability for the different events, current AAV-PVT data not suitable. Workaround through SPGF too cumbersome.

38. Guillermo Buenadicha OPS OEX ANOMALY SUMMARY RAM single bit errors No action, monitoring MM single bit errors No action, monitoring C2 Nominal Thermostat (ARTS 21) No action. Comment to be added in PRO-FCP-101 CMN Unlock (ARTS 22) Further investigations required. A2 unlock requires system procedure, ground tools and possibly other actions (i.e. cool down) CMN A2 drift Further investigations. Assessment of behaviour in slew locked in proper frequency. Mass Memory Partition 6 (7 th) Latch Up (ARTS 24) Milder recovery recommended, further tests in EM. ASW modification required. MM Partition 3 (4 th ) gap problems (ARTS 23) Further investigations (industry is required). ASW modification and/or DPGS modification required.

39. Guillermo Buenadicha OPS OEX CONCLUSIONS No immediate stopper for ICOP blue phase start However, A2 CMN lock into wrong frequency would need characterization and operational fix if persistent. To be considered that current recovery procedure involves event detection, locking frequency definition out of correlation data, and recovery via Arm Power Cycle through Proteus, making use of the power switch  required to assess its lifetime. Also, other possible SEU driven events (like MM P6 Latch Up) could reduce the data throughput and would require long engineering time for its fix (days of data loss). Partition failure under evaluation in the EM. SW change would alleviate the impact. Minor though systematic data loss in the P3 of Redundant MM. Under analysis for better characterization, still implies random data loss. The two previous events speak in favour of assessment of SW modification in order to handle gracefully MM partition disabling.

40. Guillermo Buenadicha OPS OEX FOS CONFIGURATION

41. Guillermo Buenadicha OPS OEX FOS CONFIGURATION: PLPC Version 2.8 Added one patch for better load of stacks in Manual Stack Time Correlation settings changed, to cope with clock on board too fast, thus causing resets of function. Current workaround allows for maximum 10 msec deviation Problems  Re-ingestions store packets in different point in time Not possible to cover a gap involving the switch off and on. Patch under production  foreseen end of year. Modification and cleanup in the Report generation, too many reports produced back to Gantt Explorer and SPGF Modification of the verification window up to 3 weeks to cater for 2 weeks planning. Still pending verification with CNES of the TC REJECT files. To be exercised with validation chain. Agreement reached with CNES to receive during mission lifetime E_HKTM synchronized with the pass Some data loss in S_Band data to be analyzed with CNES (punctual data gaps + G/S cases like HBK)

42. Guillermo Buenadicha OPS OEX FOS CONFIGURATION: SPGF 3.3.1 Some changes in the configuration SVAL Auto commanding settings defined Definition of nominal mode in order to expand the conflict analysis of external manoeuvres and zones In use since 17/11/12 for the planning The LO cyclic calibration introduces a long delay in the generation of the different products (~hours to run conflict analysis) The planning of external manoeuvres becomes more and more complex taking into consideration all the scientific constraints + S_Band visibilities + exclusion of zones. Could be even more difficult in case of avoidance of the P3 zone for calibrations. Under analysis a change to directly send requests to SVAL avoiding the DPGS Core.

43. Guillermo Buenadicha OPS OEX FOS CONFIGURATION: PXMF & VALIDATION CHAIN PXMF 2.8 Used during SODAP by SODAP team Data ingestion based on X_Band, non linear nor complete during SODAP. Kept up to date in DB and SW version System proves useful in case of gaps in S_Band, however no consolidation of TM exercised yet. VALIDATION CHAIN MIRASIM PLPCDEV SPGF-2 ESTEC Machines Allow validation of DB’s, patches and procedures SPGF-2 used for geolocation

44. Guillermo Buenadicha OPS OEX FOS CONFIGURATION: DATABASE & FOP DataBase 4.3 currently installed Launch version was 4.0 Main modifications are implementation of new limit monitoring. Implementation of common set of TM checks on Proteus TM together with CNES. FOP Improvements to FCP-SDP-100 Creation of new PRO-CRP-300 for ARM OFF/ON

45. Guillermo Buenadicha OPS OEX FOS CONFIGURATION: MUST Used for trend analysis and offline data analysis Currently populated up to the best possible availability of data Manual population of Server DB  tedious task, automatic population under analysis. The current MUST DB size grows too much for a complete mission (30 Gb for 1 month of effective mission) Reduction of parameters Ingestion of a selection of periods Under analysis a replica of MUST external to the firewall for other users access (Industry + PLSO) Generation of Synthetic parameters in MUST ongoing

46. Guillermo Buenadicha OPS OEX FOS CONFIGURATION: Web Server FTP Public access of FOS related items available in: 193.146.123.163 /smos/FOS Mission Planning Txt tab separated + ESOV compatible formats Available information for weekly scheduling including calibrations, manoeuvres and x_band visibilities. Anomaly Plots and textual data of given events (i.e. CMN unlocks) Operational Data Interchange Reports In progress (this presentation available) Others

47. Guillermo Buenadicha OPS OEX FOS CONFIGURATION: STAFFING IN THE PERIOD BS and GB fully on the job, extended shifts including some non working days. Amparo Garrigues still leading Mission Planning Stephen Cooke support in LEOP and SODAP Massimiliano Mazza System support EADS Casa support teams for system and thermal Rocio Fernandez in learning phase  however heavily taken by DPGS activities ESOC support (JP) + Elena Checa (ESTEC) for thermal MANY THANKS TO ALL !!! On Call service active Under definition the triggers for it No triggering from CNES

48. Guillermo Buenadicha OPS OEX FOS CONFIGURATION: Conclusion FOS Ground Segment in really good shape to support rest of IOCP. Minimal anomalies detected, all of them corrected or ongoing. FOS configuration optimized after 5 weeks of operations, more alarms, better trend analysis support, refined mission planning Adequate manpower to support operations in this phase Operational Interface Agreement definition ongoing with DPGS On Call triggers being defined, service already available. Pending: It is required a translation capability from Proteus AAV+PVT to Geo localization (BD contacted). Ideally to be implemented as Synthetic in PLPC, alternative would be in MUST. SPGF speed very much affected by high LO calibration frequency. PLPC Time Correlation problem to be fixed. Understanding of S_Band packet loss in certain passes.