1. Decompression Flaws in PI Historian
Md. Arif Khan
and
John W. Pierre
Department of Electrical and Computer Engineering
University of Wyoming
December 12, 2017

2. Outline Overview of topic
Compression/Decompression– swinging door algorithm used by some utilities in PI historians
Raw and decompressed PMU data from utility
Flaws in decompression algorithm implementation
Interpolation error
Phase wrapping error
Corrections
Should be corrected in compression/decompression code
Recovery algorithm for already corrupted data
Outline

3. Overview Raw: original measurements
Decompressed: the raw measurements went through the compression and decompression processes
Both of the data sets are from the utility
Overview

4. Overview Raw: original measurements
Decompressed: the raw measurements went through the compression and decompression processes
Both of the data sets are from the utility
Overview

5. PI System Compression Lossy! Yes! Real time operations.
Compression in PI PMU historian
A measurement has to pass two tests to be archived
Exception test
Compression test
Exception Test (PI interface)
Data
Snapshot table
Compression Test
Archive
Lossy!
Yes! Real time operations.
PI System Compression

6. Exception Test PI System Compression
Takes place on the PI interface node before the value is ever sent to the PI server
Discards small variations due to noise
Exception
time
value
Dead band
ExcDev
ExcDev
ExcDev <~ Instrument precision
Snapshot (current value sent to the PI server)
Raw value
Keep this
Discarded
Stored in Snapshot Table
PI System Compression

7. Exception Test PI System Compression
Takes place on the PI interface node before the value is ever sent to the PI server
Discards small variations due to noise
time
value
ExcDev
ExcDev
Value passed Exception test and sent to Snapshot table
Discarded
Snapshot (current value sent to the PI server)
PI System Compression

8. Exception Test PI System Compression
Takes place on the PI interface node before the value is ever sent to the PI server
Discards small variations due to noise
time
value
ExcDev
ExcDev
Value passed Exception test and sent to Snapshot table
Discarded
PI System Compression

9. Compression Test PI System Compression
Swinging door algorithm— by Edger H. Bristol in Yes! One of the owners of “The Foxboro Company.”
time
value
CompDev
CompDev
CompDev ≈ 2* ExcDev
Recently archived
Current snapshot
Archived
Incoming value
Discarded
PI System Compression

10. Compression Test PI System Compression
Swinging door algorithm (SDA)— by Edger H. Bristol in Yes! One of the owners of “The Foxboro Company.”
Deviation “blanket”
value
CompDev ≈ 2* ExcDev
time
Recently archived
Current snapshot
Archived
Incoming value
Discarded
PI System Compression

11. SDA Decompression Linear interpolation
𝑦 𝑛 = 𝑛 2 −𝑛 𝑦 𝑛 1 𝑛 2 − 𝑛 𝑛− 𝑛 1 𝑦( 𝑛 2 ) 𝑛 2 − 𝑛 1
𝑦 𝑛 2
value
𝑦 𝑛 1
Archived
Compression test discarded
time
Reconstructed
SDA Decompression

12. PMU Data Raw: original measurements
Decompressed: the raw measurements went through the compression and decompression processes
Both of the data sets are from the utility
PMU Data

13. PMU Data Raw: original measurements
Decompressed: the raw measurements went through the compression and decompression processes
Both of the data sets are from the utility
PMU Data

14. PMU Data Raw: original measurements
Decompressed: the raw measurements went through the compression and decompression processes
Both of the data sets are from the utility
Welch estimate, 30 min. frequency data, 50 sec. Von Hann window, 80% overlap
PMU Data

15. PMU Data

16. Two types of errors were observed in the decompressed data
Interpolation error
Angle wrapping error
Decompression Flaw

17. Interpolation Error m 𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 𝑚
𝑦 𝑛 = 𝑛 2 −𝑛 𝑦 𝑛 1 𝑛 2 − 𝑛 𝑛− 𝑛 1 𝑦( 𝑛 2 ) 𝑛 2 − 𝑛 1
𝑦 𝑛 2
value m
𝑦 𝑛 1
Archived
Discarded
time
𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 𝑚
Reconstructed
Interpolation Error

18. 𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 𝑚 𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 +1 𝑚 Interpolation Error
𝑦 𝑛 = 𝑛 2 −𝑛 𝑦 𝑛 1 𝑛 2 − 𝑛 𝑛− 𝑛 1 𝑦( 𝑛 2 ) 𝑛 2 − 𝑛 1
𝑦 𝑛 2
value 3m 2m m
𝑦 𝑛 1
Archived
Discarded
time
𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 𝑚
Reconstructed
𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 +1 𝑚
Interpolation Error

19. 𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 𝑚 𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 +1 𝑚 Interpolation Error
𝑦 𝑛 = 𝑛 2 −𝑛 𝑦 𝑛 1 𝑛 2 − 𝑛 𝑛− 𝑛 1 𝑦( 𝑛 2 ) 𝑛 2 − 𝑛 1
𝑦 𝑛 2
value 2m 3m 4m
𝑦 𝑛 1
Archived
Discarded
time
𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 𝑚
Reconstructed
𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 +1 𝑚
Interpolation Error

20. 𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 𝑚 𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 +1 𝑚 Interpolation Error
𝑦 𝑛 = 𝑛 2 −𝑛 𝑦 𝑛 1 𝑛 2 − 𝑛 𝑛− 𝑛 1 𝑦( 𝑛 2 ) 𝑛 2 − 𝑛 1
Equation 1
𝑦 𝑛 2
value
𝑦 𝑛 1
Archived
Discarded
time
𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 𝑚
Reconstructed
Flawed
Equation 2
𝑦 𝑛 =𝑦 𝑛 1 + 𝑛− 𝑛 1 +1 𝑚
Interpolation Error

21. Interpolation Error 𝑦 𝑛 = 𝑛 2 −𝑛 𝑦 𝑛 1 𝑛 2 − 𝑛 1
𝑵 𝒅 = 𝟑,𝟔,𝟕,𝟏𝟏,𝟏𝟐,𝟏𝟑
Equation 1
𝑦 𝑛 = 𝑛 2 −𝑛 𝑦 𝑛 1 𝑛 2 − 𝑛 1
+ 𝑛− 𝑛 1 𝑦( 𝑛 2 ) 𝑛 2 − 𝑛 1
Equation 2
𝑦 𝑛 =𝑦 𝑛 1
+ 𝑛− 𝑛 1 +1 𝑚
Interpolation Error

22. Phase Wrapping Error

23. Ideally, the utilities would correct the errors in the decompression code
For already corrupted data– two-stage three- sample algorithm (IEEE PES GM 2018)
Recovery

24. Two-stage three-sample algorithm
Recovery

25. Compression achieved 1.87%
Two-stage three-sample algorithm
Welch estimate, 30 min. frequency data, 50 sec. Von Hann window, 80% overlap
Compression achieved 1.87%
swinging door algorithm can achieve 90+% compression with significant loss of information, and the type of loss cannot be predicted precisely
Recovery

26. Offline compression using swinging door algorithm—achieved 92
Offline compression using swinging door algorithm—achieved 92.16% compression
Off-the-shelf open source lossless compression algorithms would provide 60—70% compression
Black raw data
Red decompressed data
Welch estimate, 30 min. frequency data, 50 sec. Von Hann window, 80% overlap
Ideal Compression

27. The compression/decompression processes in PI historian need to be investigated and corrected for any error
Already corrupted decompressed data can be recovered using the two-stage three-sample algorithm
Impacts of swinging door lossy compression of synchrophasor data need to be investigated
Conclusion

28. Thanks!
Questions?