1. Bounds on Redundancy in Constrained Delay Arithmetic Coding Ofer ShayevitzEado Meron Meir Feder Ram Zamir Tel Aviv University

2. Arithmetic Coding (AC) Background  Source coding with finite alphabet  Sequential coding at the entropy rate  Encoder -  Sequentially maps growing source strings into nested source intervals in [0,1)  Interval size = Probability of the sequence  Shrinking intervals  Converge to a source point  Outputs bits representing the smallest binary interval containing the source interval  Decoder –  Tracks binary interval  Decodes smallest source interval containing the current binary interval ! ( x 1 ) X = © 0 ; 1 ;:::; K ¡ 1 ª x n 4 = x 1 ;:::; x n

3. Setting and Notations  was binary encoded – Arbitrary coding scheme  However, only for was decoded  The delay (in symbols) at time n is defined as  Random prefix  The delay is a random variable  The redundancy-delay function for a source is Infimum is taken over all achievable compression ratios obeying the delay constraint D = D ( x n ) = n ¡ k ( x n ) x n x k k = k ( x n ) · n R ( d c ) 4 = i n f C ( C ¡ H ) s. t. D ( x n ) < d c + 1

4. Previous Work – Delay vs. Redundancy  [Jelinek 1968]  Memoryless source  B 2 V coding  Constant buffer size  Delay in encoded bits  Beautiful analysis of the tradeoff between buffer overflow probability and redundancy  However…  Delay in symbols = Time!  Errs when buffer overflows – not strictly lossless  B 2 V with a finite (symbol ) delay constraint: R ( d c ) = O ( 1 d c )

5. Delay in AC - An Example  Source symbols each with probability  Assume the source sequence is encoded f 0 ; 1 ; 2 g 1 3 3 8 01 1 3 2 3 01 1 2 1 4 3 4 1 3 2 3 01 4 9 11 27 Binary Interval Encoder’s Interval Decoder’s Interval Encoded symbols : Output Bits: Decoded Symbols: 0 12 011 1 1 ; 0 ; 2 ;::: 2 symbol delay at time n=3

6. Ternary Example Cont. Assume now the source sequence is encoded 01 1 3 2 3 01 1 2 1 4 3 4 1 3 2 3 01 4 9 Binary Interval Encoder’s Interval Decoder’s Interval Encoded Symbols : Output Bits: Decoded symbols: 1 1 ; 1 ; 1 ;::: 13 27 1 1 ::: 5 9 14 27 Source point Unbounded delay !!! ! ( x 1 ) = 1 2

7. Delay in AC  Origin – Discrepancy between source intervals and binary intervals  Infinite number of source sequences with unbounded delay exist  Bounded Delay  Must intervene in the normal AC flow!

8. Previous Work – The Delay’s Tail Probability  AC for a memoryless source  Bound on the delay’s tail probability [Shayevitz Zamir Feder 2006]  Resulting bound on the expected delay P n D ( x n ) > d o · 4 d ® d l og 1 ® + 4 ® d µ l og l n 2 + 1 l n 2 ¶ |{z} K ® 4 = max © p 0 ; p 2 ;:::; p K ¡ 1 ª E ( D ) · 1 + 4 ® ( K ( 1 ¡ ® ) + l og 1 ® ) ( 1 ¡ ® ) 2

9. Our New Result  Goal – Intervene with the AC flow to meet a finite delay constraint  Insertion algorithm – inserts fictitious symbols to nullify the delay  Constructive upper bound on  Bounding the redundancy-delay exponent R ( d c ) E r d 4 = l i m n ! 1 ¡ 1 d c l og R ( d c ) ¸ l og 1 = ® R ( d c ) · k ( ® ) ® d c ( 1 + d c l og 1 = ® ) 2 = O ( ® d c )

10. The Forbidden Points Concept  Ternary source, assume was encoded  No output bits not decoded yet  Condition for decoding : The binary interval must shrink into  Magnifying … £ 1 3 ; 2 3 ¢ Binary Interval 1 2 0 1 1 4 3 4 Encoder Source Interval 0 1 3 2 3 1 £ 1 3 ; 2 3 ¢ x 1 = 1 x 1 x 1

11.  not decoded if the Encoder’s interval:  Contains the point  Contains either or  And so on… We call those the forbidden points  Each “step” is the maximal binary step so that the interval ‘s edges are not crossed  To nullify the delay  Avoid the forbidden points! 1 2 3 8 3 8 5 8 5 8 1 4 11 32 21 32 11 32 21 32 1 32 1 2 1 3 2 3 1 32 x 1 1 8 1 8 1 6 1 6

12. Solution - Insertion Algorithm  Append two fictitious symbols to the source’s alphabet  Map each to an interval of length  Shrink all other intervals evenly  The encoder keeps track of decoder’s delay, and inserts a fictitious symbol when the delay constraint is breached  Can we guarantee a delay reduction? "

13. Solution – Cont.  Yes! – with a suitable fictitious symbol mapping Forbidden free interval Encoder’s Source Interval Binary interval

14. Analysis  A suitable mapping assures that one of the fictitious symbols is contained in a forbidden free interval  Inserting it will nullify the delay!  Origin of Redundancy  Expected codelength of fictitious symbols  Mismatch of source probabilities (Shrinkage) P ( D ¸ d c ) ¢ l og 1 = " l og 1 1 ¡ 2 "

15.  Optimizing for we get  And the exponent is bounded by Analysis – Cont. " R ( d c ) · k ( ® ) ® d c ( 1 + d c l og 1 = ® ) 2 E r d 4 = l i m n ! 1 ¡ 1 d c l og R ( d c ) ¸ l og 1 = ®

16. English Text Compression  Zero order model H ¼ 4 b i t s E ( D ) ¼ 3 : 5 sym b o l s B2VB2VB2VB2V d c > 100

17. Summary  Arithmetic coding scheme with an insertion algorithm can attain a finite delay  The redundancy in V 2 V coding decays exponentially with the delay constraint  In comparison, for B 2 V coding the redundancy decays like R ( d c ) = O ¡ ® d c ¢ R ( d c ) = O ( 1 d c )

18. Future Research  Sources with memory  Sharper lower bounds for the redundancy-delay exponent, depending on the entire source’s distribution  Upper bounds ?

19. Thank You !