1. FCF PAR Splinter, POIWG #34, July 2013 – 1 Glenn Research Center ZIN Technologies FCF PAR Splinter POIWG #34 July 25, 2013

2. FCF PAR Splinter, POIWG #34, July 2013 – 2 Glenn Research Center ZIN Technologies Splinter Agenda Splinter Objectives FCF Anomaly Resolution Process Flight Note F052537B Summary/Highlights PAR/IFI Simple Guidelines - FCF Perspective FCF Open/Closed PARs Moving Forward BACKUP CHARTS FROM PREVIOUS PRESENTATION FCF Console Operations Example FCF Off-Nominal Conditions and Recovery Steps

3. FCF PAR Splinter, POIWG #34, July 2013 – 3 Glenn Research Center ZIN Technologies Splinter Objectives Quickly Review FCF Anomaly Resolution Process Quickly Review Flight Note F052537B Summary/Highlights Understand OB Management Concerns Regarding F052537B Review PAR/IFI Simple Guidelines - FCF Perspective Objectives Moving Forward

4. FCF PAR Splinter, POIWG #34, July 2013 – 4 Glenn Research Center ZIN Technologies FCF Anomaly Resolution Process Introduction Real Time Anomaly Process Sustaining Engineering Anomaly Process Engineering Review Boards

5. FCF PAR Splinter, POIWG #34, July 2013 – 5 Glenn Research Center ZIN Technologies FCF Anomaly Resolution Process Scope of the anomaly resolution process includes the FCF Project and its sub-rack payloads  Process has been formalized in the controlling FCF plans in conjunction with the ZIN Quality System (See next slide)  Intent is to extend to all ZIN supported GRC payloads  Defined division of authority between contractor and GRC management Two distinct phases of anomaly resolution  Real-time Operations ­Tightly controlled to previously validated processes ­Limited by allocated ISS resources, timelines and operational constraints.  Sustaining Engineering ­Open to detailed engineering analysis and development and validation of new processes, software or hardware

6. FCF PAR Splinter, POIWG #34, July 2013 – 6 Glenn Research Center ZIN Technologies FCF Anomaly Resolution Process Real-time Operations Anomaly Process Process controlled by the FCF Payload Operation Handbook (FCF- DOC-2169) The FCF Operations Team is responsible for the real-time execution of payload operations and anomaly resolution  Responsible for logging of all issues and anomalies associated with flight operations  Responsible for identification and execution of approved off-nominal operational procedures  Coordinates directly with Project Science staff supporting console operations  Coordinates with POIC staff  Coordinates with FCF Increment Manager and GRC manager for actions outside authorized activities

7. FCF PAR Splinter, POIWG #34, July 2013 – 7 Glenn Research Center ZIN Technologies FCF Anomaly Resolution Process Real-time Operations Anomaly Process ZIN Technologies

8. FCF PAR Splinter, POIWG #34, July 2013 – 8 Glenn Research Center ZIN Technologies  Console team identifies operational anomalies and logs the event in the operations log  Console team identifies applicable alternate procedures as corrective action and implements under the authority of the Rack Officer (RO) FCF Anomaly Resolution Process Real-time Operations Anomaly Process Initial Anomaly Response

9. FCF PAR Splinter, POIWG #34, July 2013 – 9 Glenn Research Center ZIN Technologies  Console team performs an assessment as to whether daily science objectives are impacted  Coordinates with Project Science for decision to continue or contact GRC PM  Project Scientist has authority to continue/modify planned science operations. Authority is specifically defined and limited as documented in the Console Handbook  Items outside the scope of PS authority or decision to cease daily operation is coordinated through GRC project manager FCF Anomaly Resolution Process Real-time Operations Anomaly Process Project Science Authority

10. FCF PAR Splinter, POIWG #34, July 2013 – 10 Glenn Research Center ZIN Technologies  Anomalies that do not impact science are coordinated through the ZIN Increment Manager (IM)  IM coordinates with console and sustaining engineering staff and can recommend alternative operational process for anomaly resolution. RO coordinates proposed procedure with POIC.  If approved, alternative operations are captured as red- lines to the Daily Operations Plan and implemented FCF Anomaly Resolution Process Real-time Operations Anomaly Process ZIN Increment Manager Coordination

11. FCF PAR Splinter, POIWG #34, July 2013 – 11 Glenn Research Center ZIN Technologies  Anomalies that can not be resolved through the PS authority or Increment manager authority are coordinated with GRC Project Manager  GRC manager in coordination with PS can authorize operations with reduced/alternate science return (modified operations) or issue order to shutdown rack operations FCF Anomaly Resolution Process Real-time Operations Anomaly Process GRC PM Coordination

12. FCF PAR Splinter, POIWG #34, July 2013 – 12 Glenn Research Center ZIN Technologies Process controlled by the FCF Utilization Plan (FCF-PLN-0875) The FCF Sustaining Engineering (SE) team is responsible for the on-going maintenance and support for all FCF flight hardware and software  Responsible for overall tracking of all issues and anomalies associated with flight operations ­Issues reported as part of real-time flight operations, ­Anomalies with the performance of flight systems including software functions, hardware functions, hardware failures, and data anomalies The SE team is responsible for establishing a system for initiating, assignment, status tracking and closure of the identified issues and anomalies FCF Anomaly Resolution Process Sustaining Engineering Anomaly Process

13. FCF PAR Splinter, POIWG #34, July 2013 – 13 Glenn Research Center ZIN Technologies FCF Anomaly Resolution Process Sustaining Engineering Anomaly Process ZIN Technologies

14. FCF PAR Splinter, POIWG #34, July 2013 – 14 Glenn Research Center ZIN Technologies  The NCR Review Board (NRB) screens incoming issues and categorizes as anomaly or NCR  The SE manager prioritizes issues and assigns tasks to engineering team formulate resolution plans FCF Anomaly Resolution Process Sustaining Engineering Anomaly Process Issue Screening

15. FCF PAR Splinter, POIWG #34, July 2013 – 15 Glenn Research Center ZIN Technologies  The SE staff works in conjunction with the operations team and performs engineering analysis to establish the cause of an anomaly and prepares an appropriate resolution plan. SE manager is responsible for approval of resolution plans Anomaly resolution is conducted in coordination with customer representatives  ZIN coordinates with GRC manager to determine if either the ZIN or GRC ERB criteria are met FCF Anomaly Resolution Process Sustaining Engineering Anomaly Process Issue Analysis

16. FCF PAR Splinter, POIWG #34, July 2013 – 16 Glenn Research Center ZIN Technologies ZIN ERB Criteria a)Hardware Failure b)Software modification required c)Payload Anomaly Request (PAR) issued d)Design requirements not met ZIN FCF ERB - Members Chairperson – ZIN Project Manager ZIN participant(s) – Responsible engineer Related Engineering Disciplines Operations Safety Product Assurance GRC participants - GRC Project Manager Quality Assurance GRC Project Scientist GRC FCF Chief Engineer Other GRC engineering support personnel as identified by GRC Manager Formalized review process conducted in coordination with GRC GRC participation defined by GRC manager FCF Anomaly Resolution Process Sustaining Engineering Anomaly Process ZIN Engineering Review Board

17. FCF PAR Splinter, POIWG #34, July 2013 – 17 Glenn Research Center ZIN Technologies  Conducted under authority of GLPR 7123.36  Primary objective to formally review and approve significant technical actions and to provide an in-depth systems design engineering approach  Provides access to resources and expertise of GRC code DT  At request of GRC Project Manger when independence from contractor process is desired GRC OPS ERB GLPR 7123.36 GRC OPS ERB - Criteria a)GRC Project Manager Request b)GRC Project Science Request c)ZIN ERB Request GRC ERB - Members a)Per GRC GLPR 7123.36 FCF Anomaly Resolution Process Sustaining Engineering Anomaly Process GRC Engineering Review Board

18. FCF PAR Splinter, POIWG #34, July 2013 – 18 Glenn Research Center ZIN Technologies  SE manager schedules and coordinates resources to implement approved resolution plans  After implementation is completed and shown successful the anomaly is marked as closed in the tracking system  NCR’s closure is formally reviewed by the NCR board including GRC participation Ensure that all elements directly and indirectly impacted by the root cause issue are sufficiently addressed FCF Anomaly Resolution Process Sustaining Engineering Anomaly Process Issue Resolution and Closure

19. FCF PAR Splinter, POIWG #34, July 2013 – 19 Glenn Research Center ZIN Technologies ZIN ERB ZIN QS ZIN ERB Criteria a)FCF Project Manager Request b)Science Requirement Not Met c)S/W Modification Required d)Unplanned Maintenance e)Recurring Anomaly ZIN FCF ERB - Members Chairperson – ZIN Project Manager ZIN participant(s) – Responsible engineer Related Engineering Disciplines Operations Safety Product Assurance GRC Participants - GRC Project Manager Quality Assurance GRC Project Scientist GRC FCF Chief Engineer Other GRC engineering support personnel as identified by GRC Manager 1 ZIN NCR Board ZIN QS GRC OPS ERB GLPR 7123.36 GRC OPS ERB – Members Per GLPR 7123.36 GRC OPS ERB - Criteria a)GRC Project Manager Request b)GRC Project Science Request c)ZIN ERB Request ZIN NCR Board Chairperson – ZIN Project Manager ZIN Program Manager ZIN Product Assurance Manager ZIN Assigned Engineer NASA Project Manager Other NASA Participants NASA SR&QA NASA Project Chief Engineer NASA Software QA Other GRC engineering support personnel as identified by GRC Project Manager ZIN NCR - Criteria a)Hardware Failure b)Software modification required c)Payload Anomaly Request (PAR) issued d)Design requirements not met FCF Anomaly Resolution Process Sustaining Engineering Anomaly Process ERB Membership and Criteria Summary

20. FCF PAR Splinter, POIWG #34, July 2013 – 20 Glenn Research Center ZIN Technologies Flight Note F052537B Summary/Highlights POIC and Payload Facility teams will use non-invasive recovery techniques to attempt to recover an anomalous situation, even for new anomaly signatures, using approved documentation (with Flight Director “Go”) POIC and Payload Facility team will not use invasive recovery techniques to attempt to recover without MER approval Definitions were provided for Standard and Urgent notification criteria and Standard and Urgent Support Requests MER initiates and leads anomaly investigations for OB-managed facilities that require MER support to resolve  As necessary, FITs, ARTs, and IFIs to document the problem and manage its resolution

21. FCF PAR Splinter, POIWG #34, July 2013 – 21 Glenn Research Center ZIN Technologies Flight Note F052537B Summary/Highlights The POIC PAR is the primary system for documenting Payload Facility anomalies, and their resolutions  PARs are opened during real-time operations by the POIC flight control team, and assigned to the Payload Facility team for description, investigation, and resolution In parallel, the MER may also open an IFI to document engineering investigations, root cause and recurrence control.  PIRATe will maintain Payload Facility IFIs with inputs from the associated PAR and Payload Facility team POIC will open a PAR for every new payload anomaly, however not every PAR will require an IFI PARs are used by POIC in real-time, because they define approved actions in response to known anomalies.  In the event of a discrepancy, PARs will take precedence over IFIs PAR and IFI closure are not linked. Either one may stay open while the other is closed per current POIC and MER processes. MER Mgr concurrence for closure will be added to the PAR form

22. FCF PAR Splinter, POIWG #34, July 2013 – 22 Glenn Research Center ZIN Technologies Flight Note F052537B Summary/Highlights The agreements from the previous two previous slides have worked well for FCF over the past nine months  FITs have been held to document problems and have been a big help in obtaining necessary resources, particularly crew time, to resolve problems more quickly –IOP hard drive change out –MDCA needle and reservoir issues –Non-FCF problem with SAMS JEM sensor and LEHX power cycle  Decisions on whether a FIT is necessary have been worked on a problem by problem basis FCF team has concerns about possible changes to the previous agreements based on OB management concerns with the F052537B guidelines, particularly opening IFIs per the existing OB criteria  Guarantees having to maintain paper in two different systems  Would migrating payload facility problems from PAR-centric to IFI-centric assist in problem resolution?

23. FCF PAR Splinter, POIWG #34, July 2013 – 23 Glenn Research Center ZIN Technologies PAR/IFI Simple Guidelines FCF Perspective PARs and IFI decision based on the rack to ISS systems interface  IOP and EPCU are the only electronic boxes with an ISS interface –Commands, all telemetry, water, power  PARs assigned for problems that occur at the rack to ISS interface looking inward –Internal rack expertise lies with the FCF team for any rack internal problems  PARs with an IFI should be considered for problems at the rack to ISS interface looking outward –Pending an initial problem investigation by FCF team, an IFI would make sense to control additional troubleshooting of hardware external to the rack, i.e., ISS systems  This approach is largely how problems have been managed over the past several months

24. FCF PAR Splinter, POIWG #34, July 2013 – 24 Glenn Research Center ZIN Technologies FCF PAR Summary CIR Hardware PARs CIR Software PARs FIR Hardware PARs FIR Software PARs

25. FCF PAR Splinter, POIWG #34, July 2013 – 25 Glenn Research Center ZIN Technologies CIR PARs  Closed Hardware PARs –CIR-HW-0001, CIR Possible Suspect J47 Cable, FCF-NCR-1044  Status: Closed, will not recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: Possible IFI Since OSO Hardware –CIR-HW-0002, CIR Unable to Fill Chamber, FCF-NCR-1043  Status: Closed, will not recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI required –CIR-HW-0003, CIR Unable to Dispense Fuel, MDC-NCR-1013  Status: Closed, will not recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI required, procedure updated –CIR-HW-0005, CIR FOMA Off-Nominal Condition After Power-up, FCF-NCR-1040  Status: Closed, Has Recurred Numerous Times  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI required, known possible condition FCF PAR Summary

26. FCF PAR Splinter, POIWG #34, July 2013 – 26 Glenn Research Center ZIN Technologies CIR PARs  Closed Hardware PARs –CIR-HW-0006, CIR Fiber issue on GMT 313, No NCR  Status: Closed, will not recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required, Launch configuration issue –CIR-HW-0008, Burned Out Ignitor, No NCR  Status: Closed, Has Just Recently Recurred, Limited Life Item  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required, Consumable Item –CIR-HW-0009, CIR Rack Movement with Alignment Guides Installed, No NCR  Status: Closed, see CIR-HW-0012  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: Possible IFI Since OSO Hardware FCF PAR Summary

27. FCF PAR Splinter, POIWG #34, July 2013 – 27 Glenn Research Center ZIN Technologies CIR PARs  Open Hardware PARs Ready for Closure –CIR-HW-0004, CIR CIA MDCA Water Supply Leak, MDC-NCR-1001  Status: Open Pending, Will Not Recur, READY FOR CLOSURE  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: No IFI Required –CIR-HW-0007, Snubber Pin Handle Broken, No NCR  Status: Open Pending, Will not Recur, READY FOR CLOSURE  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: Possible IFI Since OSO Hardware –CIR-HW-0010, Intermittent Communication Problems Within Camera Packets, FCF-NCR-1051, READY FOR CLOSURE  Status: Open Pending, Will Not Recur, Failed LCTF DCM SN2001 will be re-flown  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: No IFI Required –CIR-HW-0011, Failure of MDCA FLEX2 Test Points, MDC-NCR-1019  Status: Open, Will Not Recur, Needles Re-designed, READY FOR CLOSURE  MER IFI Recommendation: No IFI  FCF IFI Recommendation: No IFI Required FCF PAR Summary

28. FCF PAR Splinter, POIWG #34, July 2013 – 28 Glenn Research Center ZIN Technologies CIR PARs  Open Hardware PARs Ready for Closure –CIR-HW-0012, Excessive Lateral Play with 4 Alignment Guides Installed, No NCR  Status: Open, May Recur, READY FOR CLOSURE  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: Possible IFI Since OSO Hardware –CIR-HW-0013, Fuel Dispense Problem during MDCA FLEX2 TEST POINT OPS, MDC-NCR-1025  Status: Open, Will Not Recur, READY FOR CLOSURE  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: No IFI Required –CIR-HW-0014, Tear in Bayonet O-Ring, MDC-NCR-1030  Status: Open Pending, Could Recur, READY FOR CLOSURE  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: No IFI Required –CIR-HW-0015, IPSU1 Could Not Obtain Images From Its Camera Package DCM1, FCF-NCR-1119  Status: Open, Will Not Recur, Cable Swapped, READY FOR CLOSURE  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: No IFI Required FCF PAR Summary

29. FCF PAR Splinter, POIWG #34, July 2013 – 29 Glenn Research Center ZIN Technologies CIR PARs  Open Hardware PARs Ready for Closure –CIR-HW-0016, Water Flow Control Assembly Showing Low Flow, PRACA #8232  Status: Open, Will Recur – Flow sensor issue, READY FOR CLOSURE  MER IFI Recommendation: IFI Recommended  FCF IFI Recommendation: No IFI Required –CIR-HW-0017, CIR Reduced Fuel Dispensing Observed, MDC-NCR-1054  Status: Open, Will Not Recur, READY FOR CLOSURE  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: No IFI Required –CIR-HW-0018, CIR IOP Hard Disk Drive right fastener failure, FCF-NCR-1122  Status: Open, May Recur, READY FOR CLOSURE  MER IFI Recommendation: IFI Recommended  FCF IFI Recommendation: No IFI Required –CIR-HW-0019, Bent MDCA Needle, MDC-NCR-1057  Status: Open, Will Not Recur, READY FOR CLOSURE  MER IFI Recommendation: IFI Recommended  FCF IFI Recommendation: No IFI Required FCF PAR Summary

30. FCF PAR Splinter, POIWG #34, July 2013 – 30 Glenn Research Center ZIN Technologies CIR PARs  Open Hardware PARs Ready for Closure –CIR-HW-0023, Broken Snubber Pin Tab on Lower Left Alignment Guide, No NCR  Status: Open, Will Not Recur, READY FOR CLOSURE  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: Possible IFI Since OSO Hardware –CIR-HW-0024, Broken Fuel Line on Reservoir 1 at MDCA Needle, CIR-NCR-0002  Status: Open, Will Not Recur, READY FOR CLOSURE  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required FCF PAR Summary

31. FCF PAR Splinter, POIWG #34, July 2013 – 31 Glenn Research Center ZIN Technologies CIR PARs  Open Hardware PARs –CIR-HW-0020, CIR Camera's failed to communicate with IPSU Upon Bootup, Needs NCR  Status: Open, May Recur  MER IFI Recommendation: No IFI Recommended  FCF IFI Recommendation: No IFI Required –CIR-HW-0021, CIR Taxi Violations Requiring CIR HRDL Power Cycle to Clear, Needs NCR  Status: Open, May Recur  MER IFI Recommendation: No IFI Recommended  FCF IFI Recommendation: No IFI Required –CIR-HW-0022, Duplicate of CIR-HW-0021  Status: Withdrawn –CIR-HW-0025, CIR rack smoke detector failed to power up during rack activation, Needs NCR  Status: Open, May Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: IFI Required Due to Smoke Detection Issue FCF PAR Summary

32. FCF PAR Splinter, POIWG #34, July 2013 – 32 Glenn Research Center ZIN Technologies CIR PARs  Closed Software PARs –CIR-SW-0001, Unexpected Packet Burst, FCF-NCR-1000  Status: Closed, Will Not Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: IFI would have been recommended –CIR-SW-0002, CIR loss of Health and Status, FCF-NCR-1042  Status: Closed, Will Recur, Loss of low rate telemetry  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –CIR-SW-0003, Unexpected Operations of CIR Image Processing Units (IPSU), FCF- NCR-1041  Status: Closed, Will Not Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –CIR-SW-0004, IOP SCSI Bus Lockup at HRDL/Hard Disk Activation, PRACA #6108  Status: Closed, Will Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required FCF PAR Summary

33. FCF PAR Splinter, POIWG #34, July 2013 – 33 Glenn Research Center ZIN Technologies CIR PARs  Closed Software PARs –CIR-SW-0005, GMT 182 Unstable Low-Rate Data, FCF-NCR-1042  Status: Closed, Will Recur, See CIR-SW-0002  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –CIR-SW-0007, MDCA proxy failed to connect during startup, MDC-NCR-1012  Status: Closed, Will Not Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –CIR-SW-0008, Failure of CIR Chamber to Vent, No NCR  Status: Closed, Will Not Recur, Daily Operations Plan error  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –CIR-SW-0010, Intermittent Communication betweem Camera Packets, No NCR  Status: Withdrawn FCF PAR Summary

34. FCF PAR Splinter, POIWG #34, July 2013 – 34 Glenn Research Center ZIN Technologies CIR PARs  Closed Software PARs –CIR-SW-0011, Unexpected Powerdown of HIBMS Camera, FCF-NCR-1068  Status: Closed, Will Not Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –CIR-SW-0012, CIR IOP Commanding Thread Lock-up, FCF-NCR-1089  Status: Closed, Will Recur, IOP not responding to commands  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –CIR-SW-00015, IOP Experiencing Difficulty Communicating with CIR Packages  Status: Withdrawn FCF PAR Summary

35. FCF PAR Splinter, POIWG #34, July 2013 – 35 Glenn Research Center ZIN Technologies CIR PARs  Open Software PARs Ready for Closure –CIR-SW-0006, Ineffective File Transfer from PL MDM to CIR rack, PRACA #6208  Status: Re-Opened, Will Recur, READY FOR CLOSURE  MER IFI Recommendation: No IFI  FCF IFI Recommendation: No IFI Required –CIR-SW-0009, Lost Comm with IPSU 1 & MDCA Which Placed CIR Rack in a Bad State, FCF-NCR-1035  Status: Open, Will Not Recur, but rack reboots will be required as recovery, READY FOR CLOSURE  MER IFI Recommendation: IFI Recommended  FCF IFI Recommendation: No IFI Required –CIR-SW-0013, CIR Lockup during Testing of New Script, FCF-NCR-1088  Status: Open, Will Not Recur, READY FOR CLOSURE  MER IFI Recommendation: No IFI  FCF IFI Recommendation: No IFI Required –CIR-SW-0014, CIR IPSU lock up during Image Transfer Operations, FCF-NCR- 1091  Status: Open, Fixed in IPSU software release 12-02, READY FOR CLOSURE  MER IFI Recommendation: No IFI  FCF IFI Recommendation: No IFI Required FCF PAR Summary

36. FCF PAR Splinter, POIWG #34, July 2013 – 36 Glenn Research Center ZIN Technologies CIR PARs  Open Software PARs Ready for Closure –CIR-SW-0016, CIR IPSU Lock-up Requiring IOP Reboot, FCF-NCR-1107  Status: Open, All IPSUs locked up, Fixed in IPSU software release 12-02, READY FOR CLOSURE  MER IFI Recommendation: No IFI  FCF IFI Recommendation: No IFI Required –CIR-SW-0017, Inconsistent Behavior from CIR IPSU1, FCF-NCR-1109  Status: Open, Fixed in IPSU software release 12-02, READY FOR CLOSURE  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: No IFI Required –CIR-SW-0018, CIR IOP Loss of Telemetry from IPSU and FCU, FCF-NCR-1118  Status: Assigned, May Recur, Similar to CIR-SW-0016, Use As Is, READY FOR CLOSURE  MER IFI Recommendation: No IFI  FCF IFI Recommendation: No IFI Required –CIR-SW-0019, CIR IPSU stale data/shutdown, FCF-NCR-1137  Status: Assigned, Will Recur, IPSU2 loss of communication with IOP, Use As Is, READY FOR CLOSURE  MER IFI Recommendation: No IFI  FCF IFI Recommendation: No IFI Required FCF PAR Summary

37. FCF PAR Splinter, POIWG #34, July 2013 – 37 Glenn Research Center ZIN Technologies CIR PARs  Open Software PARs Ready for Closure –CIR-SW-0020, Loss of H&S and Medium Rate Telemetry, FCF-NCR-1123  Status: Failure of IOP Hard Drive SN2001, READY FOR CLOSURE  MER IFI Recommendation: IFI Recommended  FCF IFI Recommendation: No IFI Recommended –CIR-SW-0021, IPSU #3 comm issues & HRDL lockup, FCF-NCR-1131  Status: Problem fixed in HP release 13-01, READY FOR CLOSURE  MER IFI Recommendation: IFI Recommended  FCF IFI Recommendation: No IFI Recommended –CIR-SW-0022, Unexpected Loss of LRT, FCF-NCR-1042  Status: PAR should be withdrawn, Duplicate of CIR-SW-0002  MER IFI Recommendation: IFI Recommended  FCF IFI Recommendation: No IFI Recommended FCF PAR Summary

38. FCF PAR Splinter, POIWG #34, July 2013 – 38 Glenn Research Center ZIN Technologies FIR PARs  Closed Hardware PARs –FIR-HW-0002, Broken Snubber Pin, No NCR  Status: Closed, Will Not Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: Possible IFI Since OSO Hardware –FIR-HW-0003, Damaged Threads on Microgravity Rack Barrier Post, No NCR  Status: Closed, Will Not Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: Possible IFI Since OSO Hardware –FIR-HW-0004, IOP Lock-Up During Power on of Hard Drive 2  Status: Withdrawn –FIR-HW-0005, ARIS broken pushrod, No NCR  Status: Closed, Will Recur, although no FIR ARIS operations are planned in the near term  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: Possible IFI Since OSO Hardware –FIR-HW-0008, No H&S upon rack activation  Status: Withdrawn FCF PAR Summary

39. FCF PAR Splinter, POIWG #34, July 2013 – 39 Glenn Research Center ZIN Technologies FIR PARs  Open Hardware PARs Ready for Closure –FIR-HW-0006, Water Flow Control Assembly, PRACA #8232  Status: Open, Will Recur, READY FOR CLOSURE  MER IFI Recommendation: IFI Recommended  FCF IFI Recommendation: No IFI Required –FIR-HW-0007, White Light Bulb Burnt-out, No NCR  Status: Bulb is a consumable item, READY FOR CLOSURE  MER IFI Recommendation: No IFI Recommended  FCF IFI Recommendation: No IFI Required –FIR-HW-0010, FIR Microscope Platform unable to perform Z axis translation, LMM-NCR-1042  Status: Re-work has been completed and returned to nominal operations, READY FOR CLOSURE  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required FCF PAR Summary

40. FCF PAR Splinter, POIWG #34, July 2013 – 40 Glenn Research Center ZIN Technologies FIR PARs  Open Hardware PARs –FIR-HW-0001, Dry CVB Module Cooler Failure, CVB-NCR-1018, LMM-NCR- 1012  Status: Open, Will Not Recur, Temperature stability of dry module  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –FIR-HW-0009, FIR rack smoke detector failed to power up during rack activation, Needs an NCR  Status: Open, May recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: IFI Required Due to Smoke Detection Issue FCF PAR Summary

41. FCF PAR Splinter, POIWG #34, July 2013 – 41 Glenn Research Center ZIN Technologies FIR PARs  Closed Software PARs –FIR-SW-0001, FIR Command Drop Out of the IPSU, LMM-NCR-1013  Status: Closed, Will Recur, See FIR-SW-0008  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –FIR-SW-0002, Fluid Science Avionics Processor: White Light Off Nominal State, FCF-NCR-1008  Status: Closed, Will Recur, Loss of communication with the White Light  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –FIR-SW-0003, FSAP reboot due to loss of comm with lamp  Status: Withdrawn, Duplicate of FIR-SW-0002 –FIR-SW-0004, FIR Health and Status Stale Indication, FCF-NCR-1042  Status: Closed, Will Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required FCF PAR Summary

42. FCF PAR Splinter, POIWG #34, July 2013 – 42 Glenn Research Center ZIN Technologies FIR PARs  Closed Software PARs –FIR-SW-0005, FSAP Downlink Manager Process Lockup, LMM-NCR-1014  Status: Closed, Will Recur, See FIR-SW-0006  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –FIR-SW-0006, Script Parser Process Pause, LMM-NCR-1014  Status: Closed, Will Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –FIR-SW-0007, Data Management Task Lock up, LMM-NCR-1014  Status: Closed, Will Recur, See FIR-SW-0006  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –FIR-SW-0008, IPSU Lost Comm with Cameras and Rebooted, LMM-NCR-1016  Status: Closed, Will Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required FCF PAR Summary

43. FCF PAR Splinter, POIWG #34, July 2013 – 43 Glenn Research Center ZIN Technologies FIR PARs  Closed Software PARs –FIR-SW-0009, FIR Loss of All Telemetry and Most H&S, FCF-NCR-1087  Status: Closed, Will Not Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required –FIR-SW-0010, IOP lock up during power on of hard drive 2 or HRDL, PRACA #6108  Status: Closed, Will Recur  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required FCF PAR Summary

44. FCF PAR Splinter, POIWG #34, July 2013 – 44 Glenn Research Center ZIN Technologies FIR PARs  Open Software PARs Ready for Closure –FIR-SW-0011, FIR I/O Processor Stopped Accepting Commands After CVB Test Script, PRACA #7746  Status: Fixed in IOP release 12-01, Will Not Recur, READY FOR CLOSURE  MER IFI Recommendation: No IFI Recommended  FCF IFI Recommendation: No IFI Required –FIR-SW-0012, Intermittent loss of telemetry from FIR rack, FCF-NCR-1093  Status: Open, Will Not Recur, Use As Is, READY FOR CLOSURE  MER IFI Recommendation: No IFI Recommended  FCF IFI Recommendation: No IFI Required –FIR-SW-0013, FIR HRDL Lost Proxy, FCF-NCR-1094  Status: Open, Will Recur, Use As Is, READY FOR CLOSURE  MER IFI Recommendation: No IFI Recommended  FCF IFI Recommendation: No IFI Required –FIR-SW-0015, Ineffective File Transfer from PL MDM to FIR rack, PRACA #6208  Status: Re-Opened, Will Recur, READY FOR CLOSURE  MER IFI Recommendation: No IFI  FCF IFI Recommendation: No IFI Required FCF PAR Summary

45. FCF PAR Splinter, POIWG #34, July 2013 – 45 Glenn Research Center ZIN Technologies FIR PARs  Open Software PARs Ready for Closure –FIR-SW-0016, Loss of Both Low and Medium Rate Data after Issue of Check Disk Command, Needs an NCR  Status: Open, Will Recur, Running chk_dsk function locked out telemetry processor, Use As Is, READY FOR CLOSURE  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: No IFI Required –FIR-SW-0018, FIR loss of medium rate and low rate telemetry, FCF-NCR-1138  Status: Open, Could Recur, Operator Error, READY FOR CLOSURE  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required FCF PAR Summary

46. FCF PAR Splinter, POIWG #34, July 2013 – 46 Glenn Research Center ZIN Technologies FIR PARs  Open Software PARs –FIR-SW-0014, FSAP Communication Not Established During Power-up, FCF- NCR-1106  Status: Open, Will Recur  MER IFI Recommendation: Insufficient Data  FCF IFI Recommendation: No IFI Required –FIR-SW-0017, IOP rebooted unexpectedly, FCF-NCR-1134  Status: Open, Will Recur, IOP spontaneously rebooted  MER IFI Recommendation: IFI Recommended  FCF IFI Recommendation: IFI could be opened –FIR-SW-0019, H&S Not Received Following FIR Rack Activation, Needs NCR  Status: Open, May Recur, Possibly related to FIR-SW-0004  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: IFI possibly if PLMDM to FIR communication needs checkout –FIR-SW-0020, IPSU Failure to boot from IOP, Needs NCR  Status: Open, May Recur, Duplicate of FIR-SW-0012  MER IFI Recommendation: Not Reviewed  FCF IFI Recommendation: No IFI Required FCF PAR Summary

47. FCF PAR Splinter, POIWG #34, July 2013 – 47 Glenn Research Center ZIN Technologies Open work from Payload Facility Process Roll Out (October 2012)  Define and document details of expectations on the PAR/IFI processes  Determine how to synchronize the two processes  Determine impact to CoFR processes FCF specific concerns for current operations  FCF team responds to multiple queries from multiple places for a given problem –Need to identify communication paths for rack status and problem status –Need to understand requirements on timeliness of reporting for problem status, resolution progress, etc.  Managing inputs in two systems over and above our own problem tracking system  FCF needs to get numerous PARs before the PSRT for closeout Moving Forward

48. FCF PAR Splinter, POIWG #34, July 2013 – 48 Glenn Research Center ZIN Technologies BACKUP CHARTS Backup Charts

49. FCF PAR Splinter, POIWG #34, July 2013 – 49 Glenn Research Center ZIN Technologies FCF Console Operations Basic FCF operations  FCF Console Positions  Daily Operations Plan  Auxiliary procedures  On orbit H&S monitoring  Ground exception monitoring  Command verifications –CS telemetry and telemetry end items Basic CIR operations scenarios Basic FIR operations scenarios

50. FCF PAR Splinter, POIWG #34, July 2013 – 50 Glenn Research Center ZIN Technologies CIR Console Personnel (RO and DMO required for any powered ops)  Rack Officer – RO –Responsible for overall rack management –Updates console logs for operational status  Data Management Officer – DMO –Executes commands at RO direction, authorizes PD commanding –Manages downlink resources  Payload Developer – PD –Typically on console for test point days only –Controls test point execution through scripted sequences –Not required for transfer operations, downlink operations, most crew activity FIR Console Personnel (RO and DMO required for any powered ops)  Rack Officer – RO –Responsible for overall rack management –Updates console logs for operational status  Data Management Officer – DMO –Executes commands at RO direction, authorizes PD commanding –Manages downlink resources  Payload Developer – PD –On console for the majority of operations days –Controls majority of FIR operations FCF Console Operations

51. FCF PAR Splinter, POIWG #34, July 2013 – 51 Glenn Research Center ZIN Technologies FCF Operational Controls  All nominal FCF operations are controlled by a procedure called the Daily Operations Plan –Contains procedural steps to be executed by the RO and/or DMO during operations –Includes commands to send, detailed command parameters, and telemetry verifications per command –Comprised of a series of building block templates called Ops Templates  Each Ops Template has been verified in test on the appropriate Ground Integration Unit  Auxiliary procedures have been developed for known potential unplanned and off-nominal conditions –Follow the Daily Operations Plan format –Identify controlled recovery steps to return the rack to the day’s Daily Operations Plan sequence or to obtain data for future troubleshooting and/or analysis if recovery is not immediately possible FCF Console Operations

52. FCF PAR Splinter, POIWG #34, July 2013 – 52 Glenn Research Center ZIN Technologies CIR/MDCA Auxiliary Operations Procedures  Appendix B.1 - Rack Off Nominal with Package Bad Off  Appendix B.2 - Rack Off Nominal with FCU Off-Nominal Condition  Appendix B.3 - Rack Off Nominal with IPSUx Off-Nominal Condition  Appendix C.1 - FCU Off Nominal with High Pressure in Manifold #1  Appendix C.2 - FCU Off Nominal with High Pressure in Manifold #2  Appendix C.3 - FCU Off Nominal with High Pressure in Manifold #1 and Manifold #2  Appendix D - Abnormal End of CIR FLEX-2 Operations Script  Appendix E - Low Rate Health and Status Interruption - Data Collection  Appendix F - Command Sequence Loss of Synchronization  Appendix G - IPSU Image Acquisition Continues After MDCA Test Point  Appendix H - File Uplink (Forward Process) Packet Dropout FCF Console Operations

53. FCF PAR Splinter, POIWG #34, July 2013 – 53 Glenn Research Center ZIN Technologies CIR/MDCA Auxiliary Operations Procedures (continued)  Appendix I.1 - Recover from MDCA Bad-Off Condition  Appendix I.2 - MDCA Proxy Fails to Connect at Startup  Appendix I.3 - MDCA Components Not at Expected Positions  Appendix I.4 - Recover from MDCA Automated Sequence Abort on Imaging Failure  Appendix J - VSP Reconfiguration  Appendix K - FDSS Fails to Power On at Rack Startup  Appendix L.1 - Power CIR When Only Aux Power Is Available  Appendix L.2 - Power CIR on Aux Power When Main Power Fails During Operations  Appendix M - PL MDM File Transfer Fails  Appendix N.1 - IPSU1 Anomaly During MDCA Test Point Operations  Appendix N.2 - IPSU2 Anomaly During MDCA Test Point Operations  Appendix N.3 - IPSU4 Anomaly During MDCA Test Point Operations  Appendix N.4 - IPSU3 Anomaly During MDCA Test Point Operations FCF Console Operations

54. FCF PAR Splinter, POIWG #34, July 2013 – 54 Glenn Research Center ZIN Technologies CIR/MDCA Auxiliary Operations Procedures (continued)  Appendix O.1 - Fully Retract Dispenser #1 When Fuel Reservoir Is Empty  Appendix O.2 - Fully Retract Dispenser #2 When Fuel Reservoir Is Empty  Appendix P.1 - IPSU File Transfers  Appendix P.2 - Missing/Incorrect Image File Transfers from IPSUs to IOP  Appendix R - Start SAMS Data Collection  Appendix S - Rack Lock Up  Appendix T - FTP Anomaly During Image File Transfers from IPSUs to IOP FCF Console Operations

55. FCF PAR Splinter, POIWG #34, July 2013 – 55 Glenn Research Center ZIN Technologies FIR/LMM Auxiliary Operations Procedures  Appendix B.1 - Rack Off Nominal with Bad-Off Condition  Appendix B.2 - Rack Off Nominal with FSAP Off-Nominal Condition  Appendix B.3 - Rack Off Nominal with IPSU Off-Nominal Condition  Appendix B.4 - Rack Off Nominal with FSAP CVB Off Nominal  Appendix B.5 - Rack Off-Nominal at Power Up  Appendix C - Flight Ops Script Abnormal End  Appendix D - Low Rate H and S Interruption - Data Collection  Appendix E - Command Sequence Loss of Synchronization FCF Console Operations

56. FCF PAR Splinter, POIWG #34, July 2013 – 56 Glenn Research Center ZIN Technologies FIR/LMM Auxiliary Operations Procedures  Appendix F.1 - IOP Uplink Packet Dropout  Appendix F.2 - LMM/FSAP Uplink Packet Dropout  Appendix F.3 - LMM/IPSU Uplink Packet Dropout  Appendix G - VSP Reconfiguration  Appendix H - FDSS Fails to Power On at Rack Startup  Appendix I - Power FDSS When Only Aux Power to FIR Available  Appendix J - IPSU Anomaly  Appendix L - FSAP Anomaly  Appendix N - PL MDM File Transfer Fails  Appendix O - White Light Lamp Fails to Turn On  Appendix P - Obtain IPSU Directory Listing  Appendix Q - Obtain FSAP Directory Listing FCF Console Operations

57. FCF PAR Splinter, POIWG #34, July 2013 – 57 Glenn Research Center ZIN Technologies FIR/LMM Auxiliary Operations Procedures  Appendix Q - Obtain FSAP Directory Listing  Appendix R - FIR/LMM File Update  Appendix S - Start SAMS Data Collection  Appendix T - FIR/ARIS File Update  Appendix U.1 - Recover from ARIS Transition to NOGO  Appendix U.2 - Quick Response - Uncontrolled Rack Motion  Appendix U.4 - ARIS-IOP Communications Lockup  Appendix U.5 - ARIS Script Load Failure  Appendix V - Rack Lock Up  Appendix W - Loss of HRDL Proxy FCF Console Operations

58. FCF PAR Splinter, POIWG #34, July 2013 – 58 Glenn Research Center ZIN Technologies FCF Operational Controls  Telemetry Verifications –Each FCF signal is defined in an FCF document called the Signals List (FCF-LST-0874) which defines the signal, calibration data, ground and flight cautionary limits, etc. –FCF command and telemetry database contains the telemetry limits and calibration data as well  Any limit violation of ground limits is “popped” onto the RO display for action –FCF flight software has Health and Status monitoring on over 140 signals for the CIR and over 75 signals for the FIR  Corrective action is executed by the flight software based on the criticality of the signal  Corrective actions vary from power off of the offending package or subpackage to an orderly shutdown of the entire rack »Loss of water or air cooling systems are two examples of signals that result in an orderly shutdown of the rack  Recovery from a H&S violation requires a series of commands and all active rack packages are placed in a command restrictive state (Idle State) FCF Console Operations

59. FCF PAR Splinter, POIWG #34, July 2013 – 59 Glenn Research Center ZIN Technologies FCF Operational Controls  Command Verifications –Each command is echoed back in Command Status (CS) telemetry  CS telemetry shows command status states like Received, Queued, Executing, Complete, Abnormal End –Daily Operations Plan calls out specific telemetry end items to verify for each command –Subsequent commands are not transmitted without the completion of the Daily Operations Plan command verification steps with a few exceptions…  For example: File Uplink command breaks small files into pieces and re- assembles them on orbit. These commands are queued to send 1 per second automatically. CS telemetry verifications still exist, but are not a gate to allowing the next File Uplink command to be released FCF Console Operations

60. FCF PAR Splinter, POIWG #34, July 2013 – 60 Glenn Research Center ZIN Technologies CIR/MDCA/FLEX2 Operational Activities  Dominated by a three phase test point cycle –Phase 1 = Test point operations  Requires alignment guide removal/installation (can be done before test point day)  Requires FOMA/nitrogen/venting operations, real time video during burn events. SAMS data also highly desired –Phase 2 = Transfer operations  Requires rack power and minimal commanding »Commands are issued to start scripts that execute ftp file transfers internal to the rack –Phase 3 = Image downlink operations (planned on same day as transfer operations)  Requires HRDL resource and minimal commanding  Miscellaneous powered activities –Includes flight software updates  Depending on update, software updates can involve intensive commanding when command path file uplink is employed  Crew activities (unpowered activities) –Bottle change outs, MDCA fuel reservoir change outs, etc. FCF Console Operations

61. FCF PAR Splinter, POIWG #34, July 2013 – 61 Glenn Research Center ZIN Technologies FIR/LMM/ACE Operational Activities  Longer duration test runs than CIR operations –Each ACE Sample is tested for 9 continuous operational days, with a 7 day break between samples –Image downlink is “continuous” through the MRDL link  Downlink rate is adjustable up to 2 Mbps  Miscellaneous powered activities –Includes flight software updates  Depending on update, software updates can involve intensive commanding when command path file uplink is employed and PLMDM transfers  Crew activities (unpowered activities) –Configure LMM for ACE operations –ACE Sample Inspections and Changeouts FCF Console Operations

62. FCF PAR Splinter, POIWG #34, July 2013 – 62 Glenn Research Center ZIN Technologies FIR/LMM/CVB Operational Activities  Longer duration test runs than CIR operations –The CVB module is tested for 10 operational sessions of 16 hours each and 3 operational sessions of 40 hours each –Image downlink is “continuous” through the MRDL link  Downlink rate is adjustable up to 2 Mbps  Miscellaneous powered activities –Includes flight software updates  Depending on update, software updates can involve intensive commanding when command path file uplink is employed and PLMDM transfers  Crew activities (unpowered activities) –LMM Configuration for CVB operations –CVB Module Installation and Removal FCF Console Operations

63. FCF PAR Splinter, POIWG #34, July 2013 – 63 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions Introduction CIR IPSU Operations FIR IPSU Operations FIR FSAP Operations IOP Operations

64. FCF PAR Splinter, POIWG #34, July 2013 – 64 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions CIR IPSU Power Cycles CIR IPSU Reboot on GMT 2012/135 (May 14, 2012)  Referenced to PAR CIR-SW-0014, Title: CIR IPSU lock up during Image Transfer Operations  For this instance, the MDCA Auto Sequence (AS) aborted when the AS did not receive a good indication for communication with IPSU1  IPSU1 and its camera packages needed to be power cycled quickly to make use of the fuel droplet still present on the needles following the AS abort  CIR is seeking authority to proceed with a quick recovery of a failed IPSU without having to seek authorization from the POD and/or Flight Director

65. FCF PAR Splinter, POIWG #34, July 2013 – 65 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions CIR IPSU Power Cycles Detailed explanation of IPSU operations on GMT 2012/135 (May 14, 2012)  All rack communications, whether external (looking out to ISS) or internal (looking in to CIR rack), are managed by the Input/Output Processor (IOP)  All test point operations are initiated by the Multi-Droplet Combustion Apparatus (MDCA) Automated Sequence (AS)  Each command to an IPSU originated by the AS experiences multiple “hops”, specifically MDCA to IOP, IOP to IPSU, IPSU back to IOP, IOP back to MDCA. If the command response does not return to the MDCA in the prescribed amount of time, the AS aborts –Recovery step for the problematic IPSU is a power cycle –Auxiliary Procedure I.4, Recover from MDCA Automated Sequence Abort on Imaging Failure contains the detailed recovery steps and anomaly logging instructions –The root cause of the loss of command response has been investigated extensively, but is unknown. No additional effort is planned to determine the root cause –This problem existed prior to CIR launch

66. FCF PAR Splinter, POIWG #34, July 2013 – 66 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions CIR IPSU Power Cycles Detailed explanation of IPSU operations on GMT 2012/135 (May 14, 2012)  It is important to note that all other IPSU indications are nominal, including nominal telemetry communication and nominal command response to ground commands  Experience has demonstrated an IPSU power cycle provides the greatest chance for proper AS behavior when the next AS is initiated  The AS abort scenario has another important consideration –Fuel is still deployed and the IPSU recovery needs to be completed as quickly as possible to improve the likelihood of executing a successful test point –A timely power cycle is required because:  If the fuel evaporates too quickly, the droplet can be lost completely from the needle tips  If the fuel evaporates enough, a vapor cloud can form around the droplet making ignition difficult or impossible –Fuel is a precious commodity, with each fuel reservoir holding only 2 ml of fuel  Quick action is extremely important to not waste fuel!

67. FCF PAR Splinter, POIWG #34, July 2013 – 67 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions CIR IPSU Power Cycles Summary of CIR IPSU power cycle causes  MDCA AS Abort –Cause: See previous discussion –FCF based Non-Conformance Report (NCR): MDC-NCR-1020 –Auxiliary Procedure: Appendix I.4  Communication failure between IOP and IPSU –Cause: IPSU stops responding to the IOP’s request for telemetry. IOP eventually marks the IPSU Bad Off and the rack transitions to an off-nominal state –FCF based Non-Conformance Report (NCR): LMM-NCR-1016 –Auxiliary Procedure: Appendix N.1 through N.4, P.2, T  IPSU stops updating telemetry it sends to the IOP –Cause: IPSU continues to respond to IOP requests for telemetry, but the provided IPSU telemetry frame is stale. IOP marks the IPSU off-nominal –FCF based Non-Conformance Report (NCR): LMM-NCR-1016 –Auxiliary Procedure: Appendix N.1 through N.4, P.2, T  Communication failure between IOP and IPSU –Cause: IPSU stops responding to commands –FCF based Non-Conformance Report (NCR): FCF-NCR-1091, FCF-NCR-1107 –Auxiliary Procedure: Appendix N.1 through N.4, P.2, T

68. FCF PAR Splinter, POIWG #34, July 2013 – 68 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions FIR IPSU Reboot Summary of FIR IPSU reboot causes (from PAR FIR-SW-0008)  Loss of IEEE-1394 nodes/Camera Manager Task Suspended –Cause: IEEE-1394 nodes are lost or Camera Manager task suspends and an FSAP watchdog task automatically reboots the IPSU –FCF based Non-Conformance Report (NCR): LMM-NCR-1016 –Auxiliary Procedure: None, Automatic reboot  IPSU Goes Bad Off –Cause: IPSU fails to reboot properly and the IOP detects this condition and powers off the IPSU –FCF based Non-Conformance Report (NCR): LMM-NCR-1016 –Auxiliary Procedure: Appendix B.3  IPSU reboots –Cause: IPSU will spontaneously reboot itself which requires a restart of the lmm_flight_ops script to restore lost camera settings –FCF based Non-Conformance Report (NCR): LMM-NCR-1016 –Auxiliary Procedure: Appendix J

69. FCF PAR Splinter, POIWG #34, July 2013 – 69 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions FIR IPSU Reboot Summary of FIR IPSU reboot causes (from PAR FIR-SW-0008)  LMM Camera Task Hung –Cause: LMM camera chain becomes blocked after move experiments command –FCF based Non-Conformance Report (NCR): LMM-NCR-1016 –Auxiliary Procedure: Appendix B.3  Invalid Camera Manager State –Cause: LMM camera manager indicates an invalid state, preventing experiments from being managed properly. Restarting LMM IPSU software and restoring the IPSU camera settings can often fix the problem –FCF based Non-Conformance Report (NCR): LMM-NCR-1016 –Auxiliary Procedure: Appendix B.3

70. FCF PAR Splinter, POIWG #34, July 2013 – 70 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions FIR FSAP Reboot Summary of FIR FSAP reboot causes (from PAR FIR-SW-0006)  Loss of LMM/FSAP Issued File Downlink –Cause: Problems with file downlinks originated by LMM FSAP software –FCF based Non-Conformance Report (NCR): LMM-NCR-1014 –Auxiliary Procedure: Appendix L  CVB in “Hazard Mode” –Cause: CVB module marked in the “hazard” state due to CVB module pressure limit exceedance. Can only be cleared by loading a new CVB modlule state table and restarting the LMM CVB software –FCF based Non-Conformance Report (NCR): None –Auxiliary Procedure: Appendix B.4  FSAP stops sending telemetry to the IOP –Cause: FSAP degraded operations lead to communication problems –FCF based Non-Conformance Report (NCR): LMM-NCR-1014, LMM-NCR- 1027 –Auxiliary Procedure: Appendix B.2

71. FCF PAR Splinter, POIWG #34, July 2013 – 71 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions FIR FSAP Reboot Summary of FIR FSAP reboot causes (from PAR FIR-SW-0006)  Script Parser Invalid State –Cause: LMM script parser entered an illegal parsing state –FCF based Non-Conformance Report (NCR): LMM-NCR-1015 –Auxiliary Procedure: Appendix L  Script Parser Hung –Cause: LMM script parser stops processing scripts –FCF based Non-Conformance Report (NCR): LMM-NCR-1015 –Auxiliary Procedure: Appendix L

72. FCF PAR Splinter, POIWG #34, July 2013 – 72 Glenn Research Center ZIN Technologies FCF Off-Nominal Conditions IOP Soft Reboot Summary of Situations Requiring an IOP Soft Reboot  PLMDM File Transfer Issues (CIR-SW-0006, FIR-SW-0016) –Cause: PLMDM file transfers failing. Nominal Appendix N or M steps unsuccessful in establishes reliable transfer link  HRDL Activation Issues (CIR-SW-0004, FIR-SW-0010, FIR-SW-0013) –Cause: SCSI bus lock up on HRDL activation or SCSI bus lock up on IOP HD2 activation  Each of these scenarios needs coordination with CADRE for a soft reboot