1. When Are LDCs a False Promise? Moni Naor Weizmann Institute of Science

2. Talk Based on: The Complexity of Online Memory Checking [Naor and Rothblum] Fault Tolerant Storage And Quorum Systems [Nadav and Naor] On the Compressibility of NP Instance and Cryptographic Applications [Harnik and Naor] Theme: cases where LDC should be helpful but Either provably not helpful Or open problem

3. Authentication Verifying a string has not been modified –Central problem in cryptography –Many variants Our Setting: User works on large file residing on a remote server User stores a small secret `fingerprint’ (hash) of file –Used to detect corruption What is the size of the fingerprint? –A well understood problem

4. Online Memory Checking Problem with the model: What if we don’t want to read the entire file? What if we only want small part? Read entire file?! Idea: Don’t verify the entire file, verify what you need! –How much of the file do you read per authenticated bit? –How large a fingerprint do you need?

5. Online Memory Checkers User makes store and retrieve requests to memory a vector in {0,1} n under adversary’s control Checker Checks: answer to retrieve = last stored value Checker: –Has secret reliable memory: space complexity s(n) –Makes its own reads/writes: query complexity q(n) Want small s(n) and small q(n) ! C memory checker User Public memory secret memory s(n) bits store(i,b)retrieve(i) b q(n) bits R/W

6. Memory Checker Requirements: For ANY sequence of user requests and ANY responses from public memory: Completeness: If every read from public memory = last write Guarantee: user retrieve = last store (w.h.p) Soundness: If some read from public memory ≠ last write Guarantee: user retrieve = last store or BUG (w.h.p) C memory checker User Public memory secret memory s(n) bits retrieve(i) bb or BUG

7. [Blum, Evans, Gemmel, Kannan and Naor 1991] Offline Memory Checkers: Detect errors only at end of long request sequence q(n)=O(1) (amortized) s(n)=O(log n) No Crypto assumptions! Online Memory Checkers: Past Results: With One-Way Functions q(n)=O(log n) s(n)=n  (for any  > 0) No Computational Assumpt. q(n) (any query complexity) s(n) = O(n/q(n)) Are they necessary?! s(n) x q(n) = O(n) Other Results: Optimal [Gemmel Naor 92] Must be invasive [Ajtai 2003] Very Simple (in chunks)

8. Authenticators Memory Checkers allow reliable local decodability, What about reliable local testability ? Authenticators: Encode the file x 2 {0,1} n into: a large public encoding p x a small secret encoding s x. Space complexity: s(n) Decoding Algorithm D : –Receives a public encoding p and decodes it into a vector x 2 {0,1} n Consistency verifier checks (repeatedly) public encoding was it (significantly) corrupted? reading only a few bits: t(n). –If not currupted: verifier should output “ Ok ” –If verifier outputs “ Ok ”, decoder can (whp) retrieve the file

9. Good example: Reed Solomon Pretty Good Authenticator with computational assumptions Idea: encode file X using a good error correcting code C –Actually erasures are more relevant –As long as a certain fraction of the symbols of C(X) is available, can decode X Add to each symbol a tag F k (a,i), a function of secret information k 2 {0,1} s, seed of a PRF symbol a 2  location i Verifiers picks random location i reads symbol ’a’ and tag t –Check whether t=F k (a,i) and rejects if not Decoding process removes all inappropriate tags and uses the decoding procedure of C

10. Memory Checker  Authenticator If there exists an online memory checker with – space complexity s(n) – query complexity t(n) then there exists an authenticator with – space complexity O(s(n)) – query complexity O(t(n)) Idea: Use a high-distance code

11. Improve the Information Theoretic Upper Bound(s)? Maybe we can use: Locally Decodable Codes? Locally Testable Codes? PCPs of proximity?

12. The Lower Bound Theorem 1 [Tight lower bound]: For any online memory checker secure against a computationally unbounded adversary s(n) x q(n) =  (n) True also for authenticators

13. Memory Checkers and One-Way Functions Breaking the lower bound implies one-way functions. Theorem 2 : If there exists an online memory checker: –Working in polynomial time –Secure against polynomial time adversaries –With query and space complexity: s(n) x q(n) 0 ) Then there exist functions that are hard to invert for infinitely many input lengths (“almost one-way” functions)

14. This Talk: Not say much about the proof –It is involved Initial insight: connection to the simultaneous message model

15. Simultaneous Messages Protocols [Yao 1979] For the equality function: –|m A | + |m B | =  (√n) [Newman Szegedy 1996] –|m A | x |m B | =  (n) [Babai Kimmel 1997] mBmB f(x,y) x  {0,1} n y  {0,1} n x=y? mAmA ALICE BOB CAROL

16. Ingredients for Full Proof: Consecutive Messages Model: Generalized communication complexity lower bound. Adversary “learns” public memory access distribution: Learning Adaptively Changing Distributions [NR06]. “Bait and Switch” technique: Handle adaptive checkers. One-Way functions: Breaking the generalized communication complexity lower bound in a computational setting requires one- way functions.

17. Conclusions for OMC Settled the complexity of online memory checking Characterized the computational assumptions required for good online memory checkers Open Questions: Do we need logarithmic query complexity for online memory checking with computational assumptions? Understanding relationships of crypto/complexity objects Quantum Memory Checkers? LDC

18. Talk Based on: The Complexity of Online Memory Checking [Naor and Rothblum] Fault Tolerant Storage And Quorum Systems [Nadav and Naor] On the Compressibility of NP Instance and Cryptographic Applications [Harnik and Naor] Theme: cases where LDC should be helpful but Either provably not helpful Or open problem

19. Goal Distributed file storage system –Peer-to-peer environment –Processors join and leave the system continuously Want to be able to store and retrieve files distributively Partial Solutions –Distributed File sharing applications [Gnutella, Kazaa] –Distributed Hash Tables [DH, Chord, Viceroy] Store (key, value) pairs and perform lookup on key

20. Fault-Tolerant Storage System Censor –Aims to eliminate access to some files –Can take down some servers Design Goal: –A reader should be able to reconstruct each file with high probability even after faults have occurred Probability taken over coins of the writer and reader

21. Adversarial Behavior How are the faulty processors chosen? What is the influence of the adversary Type of faults –Complete/Partial control

22. Adversarial Model Adversary chooses the set of processors to crash Different degrees of adaptiveness –Non adaptive adversary Choice of faulty processors is not based on their content –Adversary with a limited number of queries May query some processors fail-stop failures –We do not consider Byzantine failures

23. Other Fault Models Random faults model: – Examples: Distance Halving DHT, Chord –Standard technique: Replication to log(n) processors Assures survival with high probability Adversarial faults [Fiat, Saia] –Large fraction accessible after adversary crashes a linear fraction of the processors Still, a censor can target a specific file

24. Measures of Quality Read/Write complexity: –Average number of processors accessed during a read/write operation Number of rounds: – Number of rounds required from an adaptive reader Blowup Ratio: –Ratio between the total number of bits used for the storage of a file and its size

25. Connection to LDC If you are willing to have high write complexity: Can encode ALL the data with an LDC Parameters of the LDC determine how good the data storage is

26. Probabilistic Storage system based on  intersecting quorum system Storage System: –To store a file: pick a set of size uniformly at random replicate the file to all members of the quorum set –Retrieval: Choose a random set of size and probe its members –Intersection follows from the birthday paradox

27. Properties of the Probabilistic Storage System Pros : –Simplicity –Resilient against linear number of faults Even if the processors are chosen by the adversary adaptively –Adapted to a dynamic environment [Abraham, Malkhi] Want a storage system with better parameters Cons: High read/write complexity High blowup-ratio

28. Non-adaptive readers are wasteful! Non-adaptive reader: –Processors are chosen without accessing any processor Theorem: A fault tolerant storage system, in the non-adaptive reader model, resilient against  (n) faults, cannot do better than the  - intersecting storage system example. Read Complexity ¢ Write Complexity is  (n) Blowup Ratio is  (√n)

29. Open Question Do the lower bounds for the case when both the reader and the adversary are non-adaptive hold when both are fully adaptive? E For Effort

30. Talk Based on: The Complexity of Online Memory Checking [Naor and Rothblum] Fault Tolerant Storage And Quorum Systems [Nadav and Naor] On the Compressibility of NP Instance and Cryptographic Applications [Naor and Harnik] Theme: cases where LDC should be helpful but Either provably not helpful Or open problem

31. The Problem Is it possible to have an efficient procedure: Given CNF formulae  1 and  2 on same variables and same length come up with a CNF formula  that is: 1.Satisfiable if and only if  1 v  2 is satisfiable 2.Shorter than |  1 |+|  2 | If yes: There is a construction of Collision Resistant Hash functions from any one-way function No “black box” construction of CRH from OWF [Simon98] Construction uses the code of the one-way function If no: there is hope for: Efficient everlasting encryption in the hybrid bounded storage model Forward-Secure-Storage [Dziembowski] Derandomization of Sampling [Dubrov-Ishai] Sufficiently short to apply recursively (1-  ) (|  1 |+|  2 |)

32. No Witness Retrievable Compression Given CNF formulae  1 and  2 on same variables come up with a formula  that is: 1.Satisfiable if and only if  1 v  2 is satisfiable 2.Shorter than |  1 |+|  2 | Claim: if one-way functions exist, then a witness for either  1 or  2 cannot yield a witness for  efficiently. Most natural ideas are witness retrievable Satisfying assignment Proof intuition based on broadcast encryption lower bounds

33. Garey and Johnson, 1979 I can’t find an algorithm for the problem Maybe I can approximate it Solve it for some fixed parameters Find an algorithm that usually works? Solve it in time 2  n Could we just postpone it ? Approaches for dealing with NP-complete problems: Approximation algorithms Sub-exponential time algorithms Parameterized complexity Average case complexity Save it for the future

34. Verdict on LDCs? Uncompressed paper on compressibility: www.wisdom.weizmann.ac.il/~naor/PAPERS/compressibility.html www.wisdom.weizmann.ac.il/~naor/PAPERS/compressibility.html Compressed version FOCS 2006

35. THE END Thank You

36. Slides for the Proof of OMC

37. Simultaneous Theorem (lower bound for CM protocols): For any equality protocol, as long as |m P | ≤ n/100, |m A | x |m B | =  (n) mPmP x  {0,1} n y  {0,1} n mAmA mBmB x=y? ConsecutiveMessages Protocols ALICE BOB CAROL

38. Program for This Talk: Define online memory checkers Review some past results Describe new results Proof sketch: –Define communication complexity model –Sketch lower bound for a simple case –Ideas for extending to the general case

39. The Reduction Use online memory checker to construct a consecutive messages equality protocol Online Checker Space: s(n) Query: q(n) Equality Protocol Alice msg: s(n) Bob msg: O(q(n)) Reduction Conclusion: s(n) x q(n) = Ω(n) (From communication complexity lower bound)

40. Simplifying Assumption Assumption: checker chooses indices to read from public memory independently of secret memory Checker Operation: 1.Get an index i in the original file 2.Choose which indices to read from the public memory, and read them. 3.Get the secret memory 4.Retrieve i -th bit or say BUG (With loss of generality)

41. The Reduction: Outline Use online memory checker Construct “random index” protocol, Bob chooses random index i: If x = y, then Carol accepts If x i ≠ y i, then Carol rejects Use online checker to build this protocol Use error correcting code Go from “random index” to equality testing: Alice, Bob encode inputs and run “random index” protocol If Alice’s and Bob’s inputs different at even one index, encodings are different at many indices.

42. q(n)+1 bits s(n) bits CAROL BOB ALICE Checker Secret Memory S(x) Public Memory P(x) store(x) Get random index i x  {0,1} n S(x) y  {0,1} n i, y i Checker Secret Memory S(y) store (y) Public Memory P(y) retrieve(i) Bits for Carol C i = x i /BUG xixi Secret Memory S(x) Accept if y i = C i x=y accept x i ≠y i reject WANT: An adversary that can find bad x,y for protocol Can be used to find bad x,P(y),i for memory checker PROBLEM: Protocol adversary sees randomness! SOLUTION: Re-Randomize! Alice re-computes S(x) with different randomness, New S(x) independent of public randomness (given P(x) ) Requires exponential time Alice Conclusion [Weak Theorem]: For “restricted” online memory checkers s(n) x q(n) = Ω(n)

43. C i = x i /BUG CAROL BOB Checker Get random index i Accept if y i = C i Secret Memory S(x) Public Memory P(x) i x x  {0,1} n x=y accept x i ≠y i reject y  {0,1} n i, y i ALICE s(n) bits q(n)+1 bits randomness S(x) randomness Checker y Public Memory P(y) Bits for Carol WANT: An adversary that can find bad x,y for protocol Can be used to find bad x,P(y),i for memory checker PROBLEM: Protocol adversary sees randomness! SOLUTION: Rerandomize! For S(x) Alice selects (using secret coins) different real randomness giving same public memory P(x) ! Conclusion [Weak Theorem]: For “restricted” online memory checkers s(n) x q(n) = Ω(n)

44. Program for This Talk: Define online memory checkers Review some past results Describe new results Proof sketch: –Define communication complexity model –Sketch lower bound for a simple case –Ideas for extending to the general case

45. Recall Simplifying Assumption Assumption: checker chooses indices to read from public memory independently of secret memory Do we really need the assumption? Idea: If checker uses secret memory to choose indices, Adversary learns something about the secret memory from indices the checker reads.

46. Access Pattern Distribution For a retrieve request Access Pattern : Bits of public memory accessed by checker Access Pattern Distribution : Distribution of the checker’s access pattern (given its secret memory) Randomness: over checker’s coin tosses

47. Where Do We Go From Here? Observation: If adversary doesn’t know the access pattern distribution, then the checker is “home free”. Lesson for adversary: Activate checker many times, “learn” its access pattern distribution! [NR05]: Learning to Impersonate.

48. Learning The Access Pattern Distribution Theorem (Corollary from [NR05]) Learning algorithm for adversary: –Adversary stores x, secret memory s –Adversary makes O(s(n)) retrieves, p : Final public memory (after the stores and retrieves) –Adversary learns L, can generate distribution D L (p). –“Real” distribution is D S (p) Guarantee: With high probability, the distributions D L (p) and D S (p) are ε -close. L is of size O(q(n) x s(n)) bits. Guarantee is only for the public memory p reached by checker!

49. CAROL BOB ALICE Checker Secret Memory S(x) Public Memory P(x) store(x), retrieves Get random index i x  {0,1} n S(x) y  {0,1} n i, y i Checker Secret Memory S(y) store(y), retrieves Public Memory P(y) retrieve(i) Bits for Carol C i = x i /BUG Secret Memory S(x) Accept if y i = C i x=y accept x i ≠y i reject Learned L Run Learner with public coins Learned L Run Learner with same coins O(s(n)xq(n)) bits q(n)+1 bits s(n) bits L Completeness: Adversary that finds x s.t. Carol rejects when Alice AND Bob’s inputs are x, also fools memory checker Access pattern distributions by “real” S and “learned” L are close on P(x). Protocol adversary sees L, checker adversary learns it! Soundness: An adversary that finds x≠y s.t. Carol doesn’t reject, also fools memory checker Does this work??? PROBLEM: distributions by “real” S and “learned” L are close on original P(x) ! They may be very far on P(y) !

50. CAROL BOB Learner (activate Checker) Checker Get random index i Accept if y i = C i Initial Secret Mem. S Public Memory P(x) i x x  {0,1} n x=y accept x i ≠y i reject y  {0,1} n i, y i ALICE q(n)+1 bits randomness, Learned L S randomness Learner y Bits for Carol Learned L Public Memory P(y) C i = x i /BUG s(n) bits O(s(n)xq(n)) bits

51. CAROL BOB Learner (activate Checker) Checker Get random index i Accept if y i = C i Initial Secret Mem. S Public Memory P(x) i x x  {0,1} n x=y accept x i ≠y i reject y  {0,1} n i, y i ALICE q(n)+1 bits randomness, Learned L S randomness Learner y Bits for Carol Learned L Public Memory P(y) C i = x i /BUG s(n) bits O(s(n)xq(n)) bits Completeness: Adversary that finds x s.t. Carol rejects when Alice AND Bob’s inputs are x, also fools memory checker Access pattern distributions by “real” S and “learned” L are close on P(x). Protocol adversary sees L, checker adversary learns it! Soundness: An adversary that finds x≠y s.t. Carol doesn’t reject, also fools memory checker Does this work??? PROBLEM: distributions by “real” S and “learned” L are close on original P(x) ! They may be very far on P(y) !

52. Does it Work? Will the protocol work when y≠x ? No! Big problem for the adversary: Can learn access pattern distribution on correct and unmodified public memory… really wants the distribution on different modified memory! Learned information L may be: – Good on unmodified memory ( D L (P(x)), D S (P(x)) close ) – Bad on modified memory ( D L (P(y)), D S (P(y)) far ) Can’t hope to learn distribution on modified public memory

53. Bait and Switch Carol knows S and L, if only she could check whether D L (P(y)), D S (P(y)) are ε-close… If far: P(y)≠P(x) (not “real” public memory)! Reject! If close: OK for Bob to use L for access pattern! Bob always uses L to determine access pattern. This is a “weakening” of the checker.

54. Bait and Switch: Carol Approximates the Distance Main Observation: Carol (computationally unbounded) can compute probabilities of any access pattern for which all the bits read from P(y) are known. (Probabilities by both D L (P(y)) and D S (P(y)) ) Solution: Sample O(1) access patterns by D L (P(y)), use them to approximate distance between the distributions. In the protocol Bob sends these samples to Carol, she approximates the distance.

55. Putting It Together From any memory checker, we get a CM protocol for equality with: Public message:length O(s(n) x q(n)) Alice message:length s(n) Bob message:length O(q(n)) Conclusion: s(n) x q(n) =  (n)

56. Conclusion Settled the complexity of online memory checking Characterized the computational assumptions required for good online memory checkers Open Questions: Do we need logarithmic query complexity for online memory checking with computational assumptions? Understanding relationships of crypto/complexity objects Quantum Memory Checkers?