1. Sketching and Embedding are Equivalent for Norms
Alex Andoni, Columbia
Robert Krauthgamer, Weizmann Institute
Ilya Razenshteyn, MIT
HALG 2016 in Paris [paper in STOC 2015, arXiv: ]
TexPoint fonts used in EMF.
Read the TexPoint manual before you delete this box.: AAA

2. Sketching and Embedding are Equivalent for Norms
“Compressing” large object to short summary
Example: dimension reduction
JL [Johnson-Lindenstrauss,1984]: approximating ℓ 2 -distances
PCA (truncated SVD): improve signal-to-noise, easier to visualize
high-dimensional vectors, matrices, graphs, …
lossy, functional n d
When is sketching possible?
Sketching and Embedding are Equivalent for Norms

3. Sketching and Embedding are Equivalent for Norms
Sketching Metrics 1 …
Fix a metric space 𝑀 (e.g. Euclidean)
Alice and Bob hold points 𝑥,𝑦∈𝑀
Each sends 𝑠-bit sketch to Charlie
Who should distinguish
If 𝑑 𝑀 (𝑥, 𝑦)≤𝑟
or 𝑑 𝑀 𝑥, 𝑦 >𝐷𝑟
(for threshold 𝑟>0, approx. 𝐷>1)
Shared randomness, error probability ≤1/3
Alice
Bob
Charlie 𝑥 𝑦
sketch(𝑥)
sketch(𝑦)
Output: “close” or “far”
Expect tradeoff between sketch size s and distortion D?
Sketching and Embedding are Equivalent for Norms

4. Sketching and Embedding are Equivalent for Norms
Why Sketch Metrics?
Useful algorithmic building block
Near-Neighbor Search algorithm with approximation 𝐷 and storage 𝑛 𝑂(𝑠)
Fast estimation of distances (filtering)
Linear sketches are powerful for streaming (and sparse recovery)
To identify current bottlenecks
Some algorithms for 𝑙 1 and 𝑙 2 use this approach (implicitly)
Based on information theory
Less sensitive to representation and transformations
In general, to measure the “complexity” of the metric
Sketching and Embedding are Equivalent for Norms

5. Example: Sketching for ℝ
Distinguish
|𝑥–𝑦|≤1 vs. |𝑥–𝑦|>1+𝜀
Break line into pieces of width 𝑤=1+𝜀 w/random shift and color Red/Blue at random
Pr[color(𝑥)=color(𝑦)] ?
if far: 1/2
if close: 𝜖+(1−𝜖)⋅1/2=1/2+𝜖/2
Repeat 𝑂(1/ 𝜀 2 ) times
Take (1/2+𝜖/4)-th percentile
Overall: 𝐷=1+𝜀 and size 𝑠=𝑂(1/ 𝜀 2 ) B R x y
Sketching and Embedding are Equivalent for Norms

6. Sketching and Embedding are Equivalent for Norms
Sketching ℓ 𝑝 norms
In Hamming space ( ℓ 1 )
Sample bits randomly + hashing [Indyk-Motwani, Kushilevitz-Ostrovsky-Rabani’98]
ℓ 2 :
Reduces it to the real line case via dimension reduction (JL lemma)
For Gaussian 𝑔 , then 𝑔 𝑇 𝑥− 𝑔 𝑇 𝑦 is distributed as ||𝑥−𝑦| | 2 times N(0,1)
ℓ 𝑝 for 0<𝑝≤2:
Projection using 𝑝-stable distributions [Indyk’00]
Achieves 𝐷=1+𝜀 using 𝑠=𝑂(1/𝜀2)
tight [Woodruff’04]
ℓ 𝑝 for 𝑝>2: sketching is harder [BarYossef-Jayram-Kumar-Sivakumar’02, Indyk-Woodruff’05]
Achieving 𝐷=𝑂(1) requires sketch size 𝑠= Θ 𝑑 1−2/𝑝
Other metrics/norms beyond ℓ 𝒑 ?
Sketching and Embedding are Equivalent for Norms

7. Reductions between Geometries
An embedding is a map 𝑓:𝑀→𝑁 of metric 𝑀 into 𝑁
It has distortion 𝐶>0 if
∀𝑥,𝑦∈𝑀, 1≤ 𝑑 𝑁 𝑓 𝑥 ,𝑓 𝑦 𝑑 𝑀 𝑥,𝑦 ≤𝐶 𝑀 𝑁
𝑓(𝑥)
𝑓(𝑦) 𝑓 𝑥 𝑦
Sketching of size s and approximation CD for M
Sketching of size s and approximation D for N
Sketching and Embedding are Equivalent for Norms

8. Goal: Efficient Sketching
Efficient = constant sketch size 𝑠 and approximation 𝐷
Classification?
Known: Metrics 𝑀 that admit efficient sketching are
𝑀 is ℓ 𝑝 for 𝑝≤2, and
𝑀 embeds into ℓ 𝑝 for 𝑝≤2 with distortion 𝑂(1).
Other metrics with efficient sketches?
Essentially NO!
Sketching and Embedding are Equivalent for Norms

9. Efficient Sketching vs. Embedding
Our Theorem: Every normed space 𝑋 with sketches of size 𝑠 and approximation 𝐷, embeds into ℓ 1−𝜖 with distortion 𝑂(𝑠𝐷/𝜀) (for every 0<𝜀<1)
Normed space: ℝ𝑑 equipped with “length” ⋅ 𝑋
Examples: ℓ 𝑝 for 𝑝≥1, matrix norms, Earthmover distance
embedding into ℓ 𝑝 , 𝑝≤2
[Kushilevitz-Ostrosvksy-Rabani’98]
[Indyk’00]
for norms
efficient sketching
Sketching and Embedding are Equivalent for Norms

10. Application: Sketching Lower Bounds
Non-embeddability implies lower bounds for sketches
In a black-box manner
Yields new results
No embedding with distortion O(1) into ℓ 1−𝜖
No sketches* of size and approximation O(1)
*in fact, no communication protocols (any number of rounds)
Sketching and Embedding are Equivalent for Norms

11. Example 1: Earth-Mover’s Distance
For 𝑥∈ ℝ 𝑛×𝑛 that sums to zero, ‖𝑥‖ 𝐸𝑀𝐷 is the minimum cost of moving positive part of 𝑥 to the negative part
Upper bounds: 𝐷-approximation with space 𝑠= 𝑛 𝑂(1/𝐷) [Charikar’02, Indyk-Thaper’03, Naor-Schechtman’05, Andoni-DoBa-Indyk-Woodruff’09]
Lower bound extends to the min-cost matching on finite subsets in 𝑛 ×[𝑛]
No embedding with distortion 𝑂(1) into ℓ 1−𝜖 [Naor-Schechtman’05]
No sketches with 𝑠=𝑂(1) and 𝐷=𝑂(1)
Sketching and Embedding are Equivalent for Norms

12. Sketching and Embedding are Equivalent for Norms
Example 2: Trace Norm
For matrix 𝐴∈ ℝ 𝑛×𝑛 , the trace norm ‖𝐴‖ is the sum of the singular values
aka nuclear norm or Schatten-1 norm
Previous lower bounds:
Only for restricted sketching algorithms [Li-Nguyen-Woodruff’14]
Recently, 𝐷=1+𝜖 requires 𝑠≈𝑛 [Li-Woodruff’16]
Embedding into ℓ 1−𝜖 requires distortion 𝑂( 𝑛 ) [Pisier’78]
Sketching requires 𝑠𝐷=Ω( 𝑛 / log 𝑛 )
Sketching and Embedding are Equivalent for Norms

13. Sketching and Embedding are Equivalent for Norms
Proof Outline
Linear embedding of 𝑋 into ℓ 1−𝜖
Fourier analysis [Aharoni-Maurey-Mityagin’85]
Good sketches for 𝑋
Good sketches for ℓ ∞ 𝑘(𝑋)
Uses that 𝑋 is a norm
𝐿 || 𝑥 1 − 𝑥 2 | | 𝑋 ≤||𝑔 𝑥 1 −𝑔 𝑥 2 ||≤𝑈(|| 𝑥 1 − 𝑥 2 | | 𝑋 )
𝐿 and 𝑈 are non-decreasing,
𝐿(𝑡)>0 for 𝑡>0
𝑈(𝑡)→0 as 𝑡→0
|| 𝑥1, …, 𝑥𝑘 | | ∞ = max i || 𝑥 𝑖 | | 𝑋
Uniform embedding 𝑔:𝑋→ ℓ 2
Lipschitz extension [Johnson-Randrianarivony’06]
Absence of certain Poincaré-type inequalities on 𝑋
Direct sum for Information Complexity [Andoni-Jayram-Pătraşcu’10]
|| 𝑥 1 − 𝑥 2 | | 𝑋 ≤1 ⇒ ||𝑓 𝑥 1 −𝑓 𝑥 2 ||≤1
|| 𝑥 1 − 𝑥 2 | | 𝑋 ≥𝑠𝐷⇒||𝑓 𝑥 1 −𝑓 𝑥 2 ||≥10
Weak embedding 𝑓:𝑋→ ℓ 2
Convex duality + compactness
Sketching and Embedding are Equivalent for Norms

14. Sketching and Embedding are Equivalent for Norms
Almost a Shortcut
Good sketches for 𝑋
Use [Andoni-K.’07] to get 1-bit sketch with “advantage” 2 −𝑠 , i.e. random 𝑓′:𝑋→{0,1} s.t.
|| 𝑥 1 − 𝑥 2 | | 𝑋 ≤1 ⇒𝔼 |𝑓′ 𝑥 1 −𝑓′ 𝑥 2 |≤1/2− 2 −𝑠
|| 𝑥 1 − 𝑥 2 | | 𝑋 ≥𝐷 ⇒𝔼 |𝑓′ 𝑥 1 −𝑓′ 𝑥 2 |≥1/2+ 2 −𝑠
Define 𝑓:𝑋→ ℓ 1 by “enumerating” randomness
|| 𝑥 1 − 𝑥 2 | | 𝑋 ≤1 ⇒ ||𝑓 𝑥 1 −𝑓 𝑥 2 ||≤1
|| 𝑥 1 − 𝑥 2 | | 𝑋 ≥𝑠𝐷⇒||𝑓 𝑥 1 −𝑓 𝑥 2 ||≥10
Weak embedding 𝑓:𝑋→ ℓ 2
Sketching and Embedding are Equivalent for Norms

15. Sketching and Embedding are Equivalent for Norms
Further Questions
Extension to ℓ 1 ?
Do sketches with 𝑠,𝐷=𝑂(1) imply embedding into ℓ 1 with distortion 𝑂(1)?
Equivalent to an old open problem in Functional Analysis [Kwapien’69]
Extension to all metrics?
Here, the ℓ 1 version is false: Heisenberg group [Lee-Naor’06, Cheeger-Kleiner’10, Cheeger-Kleiner-Naor’11]
Tradeoff between 𝑠,𝐷?
Large approximation? 1+𝜖 approximation?
Sketches → Linear sketches?
“Complexity” of metric spaces?
Thank You!
Sketching and Embedding are Equivalent for Norms