1. Generalized Entropies
Renato Renner
Institute for Theoretical Physics
ETH Zurich, Switzerland

2. Collaborators Roger Colbeck (ETH Zurich) Nilanjana Datta (U Cambridge)
Oscar Dahlsten (ETH Zurich)
Patrick Hayden (McGill U, Montreal)
Robert König (Caltech)
Christian Schaffner (CWI, Amsterdam)
Valerio Scarani (NUS, Singapore)
Marco Tomamichel (ETH Zurich)
Ligong Wang (ETH Zurich)
Andreas Winter (U Bristol)
Stefan Wolf (ETH Zurich)
Jürg Wullschleger (U Bristol)

3. operational interpretation
Why is Shannon / von Neumann entropy so widely used in information theory?
operational interpretation
quantitative characterization of information-processing tasks
easy to handle
simple mathematical definition / intuitive entropy calculus

4. Operational interpretations of Shannon entropy (classical scenarios)
data compression rate for a source PX
rate = S(X)
transmission rate of a channel PY|X
rate = maxPX S(X) - S(X|Y)
secret-key rate for a correlated source PXYZ
rate ≥ S(X|Z) - S(X|Y)
many more …

5. Operational interpretations of von Neumann entropy (quantum scenarios)
data compression rate for a source ½A
rate = S(A)
state merging rate for a bipartite state ½AB
rate = - S(A|B)
randomness extraction rate for a cq-state ½XE
rate = S(X|E)
secret-key rate for a cqq-state ½XBE
rate ≥ S(X|E) - S(X|B) …

6. Why is von Neumann entropy so widely used in information theory?
operational meaning 
quantitative characterization of information-processing tasks
easy to handle
simple mathematical definition / intuitive entropy calculus

7. More useful facts about von Neumann entropy simple definition
S(½) := - tr(½ log ½)
S(A) := S(½A)
S(A | B) := S(A B) - S(B)
entropy calculus
Chain rule:
S(A B | C) = S(A | C) + S(B | A C)
Strong subadditivity:
S(A | B) ≥ S(A | B C)

8. Why is von Neumann entropy so widely used in information theory?
operational meaning 
quantitative characterization of information-processing tasks
easy to handle 
simple mathematical definition / intuitive entropy calculus

9. Limitations of the von Neumann entropy
Claim: Operational interpretations are only valid under certain assumptions.
Typical assumptions (e.g., for source coding)
i.i.d.: source emits n identical and independently distributed pieces of data
asymptotics: n is large (n  1)
Formally: PX1…Xn = (PX)£n for n  1

10. Can these assumptions be justified in realistic settings?
i.i.d. assumption
approximation justified by de Finetti’s theorem
(permutation symmetry implies i.i.d. structure on almost all subsystems)
problematic in certain cryptographic scenarios
(e.g., in the bounded storage model)
asymptotics
realistic settings are always finite
(small systems might be of particular interest for practice)
but might be OK if convergence is fast enough
(convergence often unknown, problematic in cryptography)

11. Is the i.i.d. assumption really needed?
Example PX
Fig.: k=4
Randomness extraction
Hextr(X) = 1 bit (depends on maximum prob.)
Data compression
Hcompr(X) ¸ k bits (depends on alphabet size)
Shannon entropy
S(X) = 1 + k/2
(provides the right answer if PX1…Xn = (PX)£n for n  1)

12. Features of von Neumann entropy
operational interpretations
hold asymptotically under the i.i.d. assumption
but generally invalid 
easy to handle
simple definition 
entropy calculus 
(Obvious) question
Is there a    -entropy measure?
Answer
Yes: Hmin

13. Generalized entropies
Definition: Generalized relative entropy for positive operators ½ and ¾.
Dmin(½ || ¾) := min {¸2R : ½ · 2¸¢¾}
Notation
½ · 2¸¢¾ means that the operator 2¸¢¾ - ½ is positive
Remarks
for two density operators ½ and ¾ Dmin(½ || ¾) ≥ 0 with equality iff ½ = ¾
Dmin is also defined for non-normalized operators

14. Classical case Let ½ and ¾ be classical states, i.e.,
Reminder: Dmin(½ || ¾) = min {¸: ½ · 2¸¢¾}
Let ½ and ¾ be classical states, i.e.,
½ = x P(x) |xihx|
¾ = x Q(x) |xihx|
Then
Dmin(P || Q) = min {¸: P(x) · 2¸¢Q(x), 8 x}
= min {¸: P(x) / Q(x) · 2¸, 8 x}
= log maxx P(x) / Q(x)

15. Classical case Let ½ and ¾ be classical states, i.e.,
½ = x P(x) |xihx|
¾ = x Q(x) |xihx|
Generalized relative entropy
Dmin(P || Q) = maxx log P(x) / Q(x)
Comparison: the standard relative entropy equals
S(P || Q) = x P(x) log P(x) / Q(x)

16. Min-entropy Definition: (Conditional) min-entropy of ½AB
Hmin(A | B) := - min Dmin(½AB || idA ­ ¾B)
where the minimum ranges over all states ¾B
Reminder: Dmin(½ || ¾) := min {¸: ½ · 2¸¢¾}
Remarks
von Neumann entropy S(A | B) := - min¾ S(½AB || idA ­ ¾B)
mutual information I (A : B) := min ¾A ¾B S(½AB || ¾A ­ ¾B)
hence, the min-mutual information may be defined by Imin(A : B) := min¾A ¾B Dmin(½AB || ¾A ­ ¾B)

17. Classical case Hmin(X|Y) = - log minQ maxx,y PXY(x,y) / QY(y)
For a classical probability distribution PXY
Hmin(X|Y) = - log minQ maxx,y PXY(x,y) / QY(y)
Optimization over Q = QY gives
Hmin(X|Y) = - log y PY maxx PX|Y(x,y)
Remarks
r.h.s. corresponds to -log of the average probability of correctly guessing X given Y.
this interpretation can be extended to the fully quantum domain [König, Schaffner, RR, 2008].
Hmin(X|Y) equiv. to entropy used in [Dodis, Smith]

18. Min-entropy without conditioning
Hmin(X) = - log maxx PX(x)

19. Smoothing Definition: Smooth entropy of PX H²(X) := maxX’ Hmin(X’)
maximum taken over all PX’ with || PX - PX’ || · ²

20. Smoothing Definition: Smooth relative entropy of ½ and ¾
D²(½ || ¾) := min½’ Dmin(½’ || ¾)
minimum taken over all ½ such that ||½ - ½’|| · ².
Definition: Smooth min-entropy of ½AB
H²(A | B) := - min¾B D²(½AB || idA ­ ¾B)

21. Von Neumann entropy as a special case
Consider an i.i.d. state ½A1 ... An B1 ... Bn := ½A B­n.
Lemma
S( A | B ) = lim²  1 limn  1 H²( A1 ... An | B1 ... Bn) / n
Remark
The lemma can be extended to spectral entropy rates (see [Han, Verdu] for classical distributions and [Hayashi, Nagaoka, Ogawa, Bowen, Datta] for quantum states).

22. Definition of a dual quantity
For ½ABC pure, the von Neumann entropy satisfies
S(A | B) = S(A B) - S(B) = S(C) - S(A C) = - S(A | C)
Definition: Smooth max-entropy of ½AB
Hmax(A | B) := - Hmin(A | C) for ½ABC pure.
Observation
Hmax(A) = H1/2(A)
Hence, Hmax(A) is a measure for the rank of ½A.

23. Operational interpretations of Hmin
extractable randomness
Hmin(X|E) uniform (relative to E) random bits can be extracted from a random variable X [König, RR]
for classical E, this result is known as left-over hashing [ILL]
state merging
- Hmin(A|B) bits of classical communication (from A to B) needed to merge A with B [Winter, RR]
data compression
Hmax(X|B) bits are needed to store X such that it can be recovered with the help of B [König]
key agreement
(at least) Hmin(X|E) - Hmax(X|B) secret key bits can be obtained from source of correlated randomness

24. Features of Hmin
operational meaning 
easy to handle ?

25. Min-entropy calculus Chain rules Strong subadditivity
Hmin(A B | C) ' Hmin(A | B C) + Hmin(B | C)
Hmin(A B | C) / Hmin(A | B C) + Hmax(B | C)
(cf. talk by Robert König)
Strong subadditivity
Hmin(A | B C) · Hmin(A | B)
... “usual” entropy calculus ...

26. Example: Proof of strong subadditivity
Lemma: Hmin(A | B C) · Hmin(A | B)
Proof: By definition, we need to show that
Dmin(½ABC || idA ­ ¾BC) ¸ Dmin(½AB || idA ­ ¾B)
But this inequality holds because
2¸ ¢ idA ­ ¾BC - ½ABC ¸ 0
implies
2¸ ¢ idA ­ ¾B - ½AB ¸ 0
Note: the lemma implies subadditivity for S

27. Features of Hmin operational meaning  easy to handle 
(often easier than von Neumann entropy)

28. Summary Hmin generalizes von Neumann entropy Main features
general operational interpretation
i.i.d. assumption not needed
no asymptotics
easy to handle
simple definition
simpler proofs (e.g., strong subadditivity)

29. quantum key distribution cryptography in the bounded storage model ...
Applications
quantum key distribution
min-entropy plays crucial role in general security proofs
cryptography in the bounded storage model
see talks by Christian Schaffner and by Robert König
...
Open questions
Additivity conjecture
Hmax corresponds to H1/2, for which the additivity conjecture still might hold
New entropy measures based on Hmin?

30. Thanks for your attention