PACS IIDR 01/02 Mar 2001 On-Board Data Compression1 On-Board Data Compression Concept A. N. Belbachir Vienna University of Technology

PACS IIDR 01/02 Mar 2001 On-Board Data Compression2 Signal Description Photometry mode (Bolometers): –16 bits detector signal –40 Hz readout rate –SNR –data rate of 1600 Kbits/s –SWL detectors with 16x16x8 data values (1280 Kbits/s) –LWL detectors with 16x16x2 data values (320 Kbits/s)

PACS IIDR 01/02 Mar 2001 On-Board Data Compression3 Signal Description Spectroscopy mode (Photoconductors): –16 bits detector signal –256 Hz maximum readout rate –Reset interval from1/32s to 16s –SNR –data rate of 3600 Kbits/s –SWL detectors with 18x25 data values (1800 Kbits/s) –LWL detectors with 18x25 data values (1800 Kbits/s) –18x26 data is the science input to the SPU. –Data rate is 3744 Kbits/s

PACS IIDR 01/02 Mar 2001 On-Board Data Compression4 Maximum Data Rate The Maximum data rate = (science data + DEC/MEC header)/s Photometry: [ (16  16  10) channels  16bits + (2  60  8) header]  40 Hz = Kbits/s Spectroscopy: [ (18  26  2) channels  16bits + (2  72  8) header]  256 Hz = 4032 Kbits/s

PACS IIDR 01/02 Mar 2001 On-Board Data Compression5 Downlink Rate and Compression Ratio The downlink rate is 100 Kbits/s Compression ratio for raw data is at least –16.38 in photometry –40.32 in spectroscopy The DEC/MEC header contains –The observation configuration Timing parameters Readouts specifications Observation identification... –The compression parameters

PACS IIDR 01/02 Mar 2001 On-Board Data Compression6 Typical Signals Simulation Photometry Voltage Time

PACS IIDR 01/02 Mar 2001 On-Board Data Compression7 Typical Signals Simulation Photometry Voltage Time

PACS IIDR 01/02 Mar 2001 On-Board Data Compression8 Typical Signals Simulation Spectroscopy Voltage Time

PACS IIDR 01/02 Mar 2001 On-Board Data Compression9 Typical Signals Simulation Spectroscopy Voltage Time

PACS IIDR 01/02 Mar 2001 On-Board Data Compression10 SPU Data Flow DPU SPU SWL DEC/ MEC SPU PSU SPU LWL S/C Power via DPU 28V Primary Voltage Data command/response / HK compressed data Data command/response / HK compressed data

PACS IIDR 01/02 Mar 2001 On-Board Data Compression11 SPU Transmission Modes Compressed data mode: –Transmission of compressed data + header and raw data of few channels. –Transmission of SPU HK Raw data mode: –Transmission of lossless compressed data from selected detectors –Transmission of compressed header and SPU HK

PACS IIDR 01/02 Mar 2001 On-Board Data Compression12 SPU Transmission Modes Compressed data plus raw data mode –Switch between compressed data and raw data transmission mode for the processing of the same data –Transmission of compressed header and SPU HK Transparent mode –Transmission of data from selected detectors without compression –Transmission of compressed header and SPU HK

PACS IIDR 01/02 Mar 2001 On-Board Data Compression13 Compression Parameters Default compression ratio Double the compression ratio Triple the compression ratio Quadruple the compression ratio Do not compress

PACS IIDR 01/02 Mar 2001 On-Board Data Compression14 DPU Commands Stop compression Start compression Perform SPU test in photometry Perform SPU test in spectroscopy Perform warm Reset Start the program for the bolometer test mode Load to SPU RAM (5 specific load commands) Dump from SPU RAM Start Peak-up Software

PACS IIDR 01/02 Mar 2001 On-Board Data Compression15 SPU HLSW Switching Modes 28 software modes in total –18 compression modes according to DEC/MEC raw data –9 software modes according to DPU commands –SPU standby

PACS IIDR 01/02 Mar 2001 On-Board Data Compression16 SPU HLSW Flowchart Yes TOP Data Input from DEC/MEC and/or DPU No Communication I/F Peak- up I/F Warm Reset Dump Yes No Stop OBS Compression Mode Identification Data Output from SPU to DPU Communication I/F Watch Process Application SW Command Indentification Data Compression (+ OBS HK) Send HK DPU Command Test Spec. Load Start OBS Test Phot. Bol. Test

PACS IIDR 01/02 Mar 2001 On-Board Data Compression17 Reduction / Compression Scheme 100Kbit/s Spectroscopy 1800 Kbit/s Photometry 320 Kbit/s 1280 Kbit/s Integ- ration Robust Avera- ging Glitch Detec- tion Sample Rejec- tion Temporal Redun. Reduc. Spatial Redun. Reduc. Raw Data Selection Lossl. Coding Ramp Fit- ting Glitch Detec -tion Integ- ration Ramp Rejec- Temp. Red. Reduc. Spat. Red. Reduc. Raw Data Selection Lossl. Coding Pre- pro- ces- sing

PACS IIDR 01/02 Mar 2001 On-Board Data Compression18 Preprocessing This module is still TBD Ramp linearization necessary or not

PACS IIDR 01/02 Mar 2001 On-Board Data Compression19 Robust Averaging and Ramp Fitting Robust averaging in photometry –Preprocessing (TBD) –Median fitting –Mean value calculation Ramp fitting in spectroscopy –Fitting with RANSAC –Least square fitting

PACS IIDR 01/02 Mar 2001 On-Board Data Compression20 Glitch Detection To ensure not to integrate over invalid sensor readings (i.e. glitches) Multilevel glitch detection –Intrinsic deglitching at individual sample level –Extrinsic deglitching at ramp/averages level and considering subsequent slopes/averages

PACS IIDR 01/02 Mar 2001 On-Board Data Compression21 Integration in Spectroscopy On-board integration of sensor readings to achieve the desired compression ratio Special emphasis: –Guarantee integration over the right readings (i.e. synchronized with positions of chopper) –Not to integrate over ramps affected by glitches 2 step process: 1. Discard all data of CRE integration blocks with glitches 2. Add slope data of successive ramps within the same chopper position

PACS IIDR 01/02 Mar 2001 On-Board Data Compression22 Redundancy Reduction Temporal redundancy reduction –Calculation of references frames and differences Spatial redundancy reduction –Calculation of references pixels and differences

PACS IIDR 01/02 Mar 2001 On-Board Data Compression23 Lossless Coding Redundancy reduction reduces magnitude of pixel’s values Implement the run-length encoding algorithm to achieve additional compression

PACS IIDR 01/02 Mar 2001 On-Board Data Compression24 Compression Ratio from Redundancy Reduction (Worst Case) Photometry: –SNR: for 16 bits signal –4 bits for the noise and 12 bits for the offset signal –for 12 averages: the compression ratio is 3.2 Spectroscopy: –SNR: 450 for 16 bits signal –10 bits for the noise and 6 bits for the offset signal –for 2 slopes: the compression ratio is 1.23

PACS IIDR 01/02 Mar 2001 On-Board Data Compression25 Compression Ratio from Lossy Compression (Worst Case) Photometry: –5 samples to average in order to fulfill the telemetry requirements of 100 Kbits/s –Total compression ratio: 3.2  5 = 16 Spectroscopy in the most challenging case with a reset interval of 1/32s: –Compression ratio of 8 from the ramp fitting –Integration of 4 ramps –Total compression ratio: 8  4  1.23 = The remaining ratio is achieved by the spatial redundancy reduction and the lossless coding

PACS IIDR 01/02 Mar 2001 On-Board Data Compression26 Compressed data rate Photometry: þCompressed science data + compressed header + no data from additional channels þ Kbits/s without the spatial redundancy reduction and lossless coding algorithms Spectroscopy: þCompressed science data + compressed header + raw data of 2 channels per board þ Kbits/s for the most challenging case

PACS IIDR 01/02 Mar 2001 On-Board Data Compression27 Data Stream from DEC/MEC in Photometry (SWL SPU) array 6array 7array array 2array 3 array array 4 array row column header array array array 3 header array array array 7 header array array 8

PACS IIDR 01/02 Mar 2001 On-Board Data Compression28 Data Stream from DEC/MEC in Photometry Data Stream: Multiple of 4.2K for the SWL SPU Block1 Frame 1 Chopper Move (max. 10 Hz) ~4.2K Next Frames Block2 ~1.56K Block3 ~1.06K Multiple of 1.06K for the LWL SPU

PACS IIDR 01/02 Mar 2001 On-Board Data Compression29 Data Stream from DEC/MEC in Spectroscopy Data Stream: Multiple of 1K per each SPU board Frame1 Block 1 Chopper Move (max. 10 Hz) ~2K Next Frames Frame2 ~1K Frame3 ~1K Block2 ~2K Frame4 ~1K

PACS IIDR 01/02 Mar 2001 On-Board Data Compression30 Memory Constraints Buffer (64KB) Processing Memory (1.44MB) To/From DPU  100 Kbits/s From DEC/MEC  4032Kbits/s Buffer (0.5MB) Buffer switch of Data Input Program Memory (1.5 MB) Link Buffer (32KB) To/From SPU Board SPU RAM Organization

PACS IIDR 01/02 Mar 2001 On-Board Data Compression31 Memory Organization Requirements Start Processing with 0.5 MB buffer –2s of observation in Spectroscopy (36KB for the header) –3s of observation with blue detectors in photometry (21KB for the header) –12s of observation with red detectors in photometry (28KB for the header) 32 bits address in the data memory 16 bits signal per detector data 2 readouts data per 32 bit address

PACS IIDR 01/02 Mar 2001 On-Board Data Compression32 Memory Requirements 1.5 Mbytes of RAM for the program memory 2.5 Mbytes of RAM for the data memory –2 buffers of 0.5 Mbytes to store DEC/MEC raw data –1 buffer of 64 Kbytes to store the compressed data + HK –About 1.4 Mbytes for data processing

PACS IIDR 01/02 Mar 2001 On-Board Data Compression33 Processing Requirements in Spectroscopy (Worst Case) n = 8 samples at 256Hz (Nb. of samples per ramp) R = 64 ramps for the most challenging case 1/32s reset Processing Power: –Ramp fitting computationally most demanding 20 FLOPs for 1 Fit 10 FLOPs for 1 Residual 20 FLOPs per sample for glitch detection 20 FLOPs per ramp for glitch detection 20 FLOPs for the integration of 4 ramps 20 FLOPs for the redundancy reduction

PACS IIDR 01/02 Mar 2001 On-Board Data Compression34 Processing Requirements in Spectroscopy For 450 detectors we have  R(n 2 + 3n + 3) For 2 s observation  26MFLOPs/2s=13MFLOPs/s

PACS IIDR 01/02 Mar 2001 On-Board Data Compression35 Processing Requirements in Photometry (Worst Case) n = 5 samples at 40Hz (Nb. of samples to average) A = 20 averages in a 3 s buffer for the SWL SPU Processing Power: –Robust averaging computationally most demanding 40 FLOPs for 1 robust averaging 10 FLOPs for 1 Residual 20 FLOPs per sample for Glitch detection 20 FLOPs per average for glitch detection 20 FLOPs for the redundancy reduction

PACS IIDR 01/02 Mar 2001 On-Board Data Compression36 Processing Requirements in Photometry For 2048 detectors we have  2048  30  A(n + 4) For 3 s observation  11MFLOPs/3s = 4MFLOPs/s

PACS IIDR 01/02 Mar 2001 On-Board Data Compression37 Compressed Entity Blocks:Nb. of blocks the compressed entity packet has been split into (the block unit is 1004B) Type: Type of the observing Mode (spectroscopy or photometry) PIXB/PIXR: relationship between the HK and the science packets ‘the same index’ CDMHS: Compressed DEC/MEC Header Size CDMH: Compressed DEC/MEC Header field DECID: Decompression Code ID CSD:Compressed Science Data Compressed Entity Packet Structure CSD (TBD Bytes) PIXB/PIXR (2 Bytes) Blocks (1Byte) CDMH (CDMHS Bytes) DECID (TBD Bytes ) Type (1Byte) CDMHS (1 Byte)

PACS IIDR 01/02 Mar 2001 On-Board Data Compression38 Compressed Entity/HK The compressed entity size is: –~ 31KB per SPU board in photometry –~ 13 KB per SPU board in spectroscopy The SPU HK: compression results/SPU status –SPU HK rate depends on the reset interval and compression efficiency –SPU HK size is 38 Bytes

PACS IIDR 01/02 Mar 2001 On-Board Data Compression39 Summary The concept of the on-board software finalized Compression of the science data according to the header DPU commands to load or dump from the SPU memory or to perform specific exercises HK to inform about the SPU status Two different set of compression modules according to: –Photometry to achieve at least a compression of 16 –Spectroscopy to achieve at least a compression of 40