Bounds on Redundancy in Constrained Delay Arithmetic Coding Ofer ShayevitzEado Meron Meir Feder Ram Zamir Tel Aviv University

Arithmetic Coding (AC) Background  Source coding with finite alphabet  Sequential coding at the entropy rate  Encoder -  Sequentially maps growing source strings into nested source intervals in [0,1)  Interval size = Probability of the sequence  Shrinking intervals  Converge to a source point  Outputs bits representing the smallest binary interval containing the source interval  Decoder –  Tracks binary interval  Decodes smallest source interval containing the current binary interval ! ( x 1 ) X = © 0 ; 1 ;:::; K ¡ 1 ª x n 4 = x 1 ;:::; x n

Setting and Notations  was binary encoded – Arbitrary coding scheme  However, only for was decoded  The delay (in symbols) at time n is defined as  Random prefix  The delay is a random variable  The redundancy-delay function for a source is Infimum is taken over all achievable compression ratios obeying the delay constraint D = D ( x n ) = n ¡ k ( x n ) x n x k k = k ( x n ) · n R ( d c ) 4 = i n f C ( C ¡ H ) s. t. D ( x n ) < d c + 1

Previous Work – Delay vs. Redundancy  [Jelinek 1968]  Memoryless source  B 2 V coding  Constant buffer size  Delay in encoded bits  Beautiful analysis of the tradeoff between buffer overflow probability and redundancy  However…  Delay in symbols = Time!  Errs when buffer overflows – not strictly lossless  B 2 V with a finite (symbol ) delay constraint: R ( d c ) = O ( 1 d c )

Delay in AC - An Example  Source symbols each with probability  Assume the source sequence is encoded f 0 ; 1 ; 2 g Binary Interval Encoder’s Interval Decoder’s Interval Encoded symbols : Output Bits: Decoded Symbols: ; 0 ; 2 ;::: 2 symbol delay at time n=3

Ternary Example Cont. Assume now the source sequence is encoded Binary Interval Encoder’s Interval Decoder’s Interval Encoded Symbols : Output Bits: Decoded symbols: 1 1 ; 1 ; 1 ;::: ::: Source point Unbounded delay !!! ! ( x 1 ) = 1 2

Delay in AC  Origin – Discrepancy between source intervals and binary intervals  Infinite number of source sequences with unbounded delay exist  Bounded Delay  Must intervene in the normal AC flow!

Previous Work – The Delay’s Tail Probability  AC for a memoryless source  Bound on the delay’s tail probability [Shayevitz Zamir Feder 2006]  Resulting bound on the expected delay P n D ( x n ) > d o · 4 d ® d l og 1 ® + 4 ® d µ l og l n l n 2 ¶ |{z} K ® 4 = max © p 0 ; p 2 ;:::; p K ¡ 1 ª E ( D ) · ® ( K ( 1 ¡ ® ) + l og 1 ® ) ( 1 ¡ ® ) 2

Our New Result  Goal – Intervene with the AC flow to meet a finite delay constraint  Insertion algorithm – inserts fictitious symbols to nullify the delay  Constructive upper bound on  Bounding the redundancy-delay exponent R ( d c ) E r d 4 = l i m n ! 1 ¡ 1 d c l og R ( d c ) ¸ l og 1 = ® R ( d c ) · k ( ® ) ® d c ( 1 + d c l og 1 = ® ) 2 = O ( ® d c )

The Forbidden Points Concept  Ternary source, assume was encoded  No output bits not decoded yet  Condition for decoding : The binary interval must shrink into  Magnifying … £ 1 3 ; 2 3 ¢ Binary Interval Encoder Source Interval £ 1 3 ; 2 3 ¢ x 1 = 1 x 1 x 1

 not decoded if the Encoder’s interval:  Contains the point  Contains either or  And so on… We call those the forbidden points  Each “step” is the maximal binary step so that the interval ‘s edges are not crossed  To nullify the delay  Avoid the forbidden points! x

Solution - Insertion Algorithm  Append two fictitious symbols to the source’s alphabet  Map each to an interval of length  Shrink all other intervals evenly  The encoder keeps track of decoder’s delay, and inserts a fictitious symbol when the delay constraint is breached  Can we guarantee a delay reduction? "

Solution – Cont.  Yes! – with a suitable fictitious symbol mapping Forbidden free interval Encoder’s Source Interval Binary interval

Analysis  A suitable mapping assures that one of the fictitious symbols is contained in a forbidden free interval  Inserting it will nullify the delay!  Origin of Redundancy  Expected codelength of fictitious symbols  Mismatch of source probabilities (Shrinkage) P ( D ¸ d c ) ¢ l og 1 = " l og 1 1 ¡ 2 "

 Optimizing for we get  And the exponent is bounded by Analysis – Cont. " R ( d c ) · k ( ® ) ® d c ( 1 + d c l og 1 = ® ) 2 E r d 4 = l i m n ! 1 ¡ 1 d c l og R ( d c ) ¸ l og 1 = ®

English Text Compression  Zero order model H ¼ 4 b i t s E ( D ) ¼ 3 : 5 sym b o l s B2VB2VB2VB2V d c > 100

Summary  Arithmetic coding scheme with an insertion algorithm can attain a finite delay  The redundancy in V 2 V coding decays exponentially with the delay constraint  In comparison, for B 2 V coding the redundancy decays like R ( d c ) = O ¡ ® d c ¢ R ( d c ) = O ( 1 d c )

Future Research  Sources with memory  Sharper lower bounds for the redundancy-delay exponent, depending on the entire source’s distribution  Upper bounds ?

Thank You !