Presentation is loading. Please wait.

Presentation is loading. Please wait.

On-The-Fly Verification of Rateless Erasure Codes Max Krohn (MIT CSAIL) Michael Freedman and David Mazières (NYU)

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "On-The-Fly Verification of Rateless Erasure Codes Max Krohn (MIT CSAIL) Michael Freedman and David Mazières (NYU)"— Presentation transcript:

1 On-The-Fly Verification of Rateless Erasure Codes Max Krohn (MIT CSAIL) Michael Freedman and David Mazières (NYU)

2 On-The-Fly Verification of Rateless Erasure Codes Max Krohn (MIT CSAIL) Michael Freedman and David Mazières (NYU) Multicast Authentication: Dead/Exhausted

3 The Setting A large file F  Linux ISO (650MB) H(F) is available  signed by Publisher (RedHat) A handful of untrusted sources/mirrors S 1,…S 8

4 A Handful of Senders

5 The Setting A large file F  Linux ISO (650MB) H(F) is available  signed by Publisher (RedHat) A handful of untrusted sources S 1,…S 8  Their aggregate BW is limited A slew of receivers R 1,...,R 1,000,000  Version 81.3 just released! Want it Now!

6 Three Desirable Properties Sources Can Multicast Clients Get Fast Downloads Clients Can Verify Blocks On-the-Fly

7 Receivers Get Fast, Verifiable Downloads The trusted publisher (RedHat)  Splits up F into n blocks  Hashes all blocks  Signs all hashes (or hash tree) Receivers:  Download and verify hashes  Download needed file blocks in parallel

8 R1R1 S1S1 S3S3 S4S4 S2S2 R3R3 R2R2 R4R4 R5R5 R6R6 R7R7 R8R8 R9R9 R 10 R 11 R 12 R 13 Everyone for Themselves

9 Everyone For Themselves Sources Can Multicast Clients Get Fast Downloads Clients Can Verify Blocks On-the-Fly

10 R1R1 S1S1 S3S3 S4S4 S2S2 R3R3 R2R2 R4R4 R5R5 R6R6 R7R7 R8R8 R9R9 R 10 R 11 R 12 R 13 Verifiable Multicast (BitTorrent)

11 Sources Can Multicast Clients Get Fast Downloads Clients Can Verify Blocks On-the-Fly

12 Multicast With Erasure Codes Sources erasure encode the file F n blocks blocks F … ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? … … … …

13 Multicast With Erasure Codes Sources erasure encode the file F Receivers collect blocks and decode n blocks blocks ?? ?? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? 1.03n blocks n blocks F F … … … … …

14 R1R1 S1S1 S3S3 S4S4 S2S2 R3R3 R2R2 R4R4 R5R5 R6R6 R7R7 R8R8 R9R9 R 10 R 11 R 12 R 13 Multicast With Erasure Codes

15 Bullet [SOSP 2003] SplitStream [SOSP 2003] Big Downloads [IPTPS 2003] Informed Content Delivery [SIGCOMM 2002]

16 R1R1 S1S1 S3S3 S4S4 S2S2 Receivers Cannot Verify Content ? ? ? ? ? ? ? ? ? ? ? ? ? ?

17 R1R1 S1S1 S3S3 S4S4 S2S2 ? ? ? ? ? ? ? ? ? ?

18 Multicast With Erasure Codes Sources Can Multicast Clients Get Fast Downloads Clients Can Verify Blocks On-the-Fly

19 Multicast With Erasure Codes Sources Can Multicast Clients Get Fast Downloads Clients Can Verify Blocks On-the-Fly

20 What is the Attack Goal? To corrupt the file. To waste bandwidth. R S1S1 S3S3 S2S2

21 How To Attack? Send correct blocks but with skewed distributions.  “Distribution Attack” Send incorrect blocks  “Pollution Attack” Karlof et al. [NDSS ’04] R S1S1 S3S3 S2S2

22 Properties of a Solution to Pollution OK: Receivers can tell good from bad. Much better: Receivers can finger bad blocks as they arrive. R S1S1 S3S3 S2S2 CONTRIBUTION

23 Outline Introduction Review of LT Codes Strawman #1 Strawman #2 Efficiently Catching Bad Blocks as They Arrive

24 LT-Codes [Luby, FOCS 2002] b1b1 b2b2 b3b3 b4b4 b5b5 F= n=5 input blocks

25 LT-Codes – How To Encode b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 1.Pick degree d 1 from a pre-specified distribution. ( d 1 =2) 2.Select d 1 input blocks uniformly at random. (Pick b 1 and b 4 ) 3.Compute their sum. 4.Output E(F)= F=

26 LT-Codes – How To Encode (cont’d) b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F=

27 How To Decode b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F=

28 How To Decode b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F=

29 How To Decode b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F=

30 b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F= How To Decode

31 b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F= b5b5 b5b5 b5b5

32 How To Decode b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c5c5 c6c6 c7c7 E(F)= F= b5b5 c4c4 b5b5 b5b5

33 How To Decode b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c5c5 c6c6 c7c7 E(F)= F= b5b5 c4c4 b5b5 b5b5

34 How To Decode b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c5c5 c6c6 c7c7 E(F)= F= b5b5 c4c4 b5b5 b5b5

35 How To Decode b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c5c5 c6c6 c7c7 E(F)= F= b5b5 c4c4 b5b5 b5b5 b2b2 b2b2

36 How To Decode b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c5c5 c6c6 c7c7 E(F)= F= b5b5 c4c4 b5b5 b5b5 b2b2 b2b2

37 Outline Introduction Review of LT Codes Strawman #1  Simple Solution To Tell Good Blocks From Bad Strawman #2 Efficiently Catching Bad Blocks as They Arrive

38 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F=

39 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F=

40 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F=

41 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F=

42 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F=

43 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F= b5b5 b5b5 b5b5

44 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F= b5b5 b5b5 b5b5

45 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F= b5b5 b5b5 b5b5 X

46 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F= b5b5 b5b5 b5b5 X

47 “Smart Decoder” for LT-Codes b1b1 b2b2 b3b3 b4b4 b5b5 c1c1 c2c2 c3c3 c4c4 c5c5 c6c6 c7c7 E(F)= F= b5b5 b5b5 b5b5

48 “Smart Decoder:” Problem Data collected from 50 random Online encodings of a 10,000 block file.

49 Outline Introduction Review of LT Codes Strawman #1 Strawman #2  Hashing/Signing Encoded Blocks Efficiently Catching the Bad as They Arrive

50 Hashing/Signing Encoded Blocks Trusted Publisher (RedHat)  Picks e, computes e·n encoded blocks  Hashes all encoded blocks  Signs the hashes. n blocks e · n blocks ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? F

51 Hashing/Signing Encoded Blocks Expansion factor e should be big to avoid duplicate blocks. e should be small to make crypto overhead acceptable. Our analysis shows there’s no “sweet spot”.

52 Hashing/Signing Encoded Blocks Expansion factor e should be big to avoid duplicate blocks. e should be small to make crypto overhead acceptable. Our analysis shows there’s no “sweet spot”.  e.g., best case bandwidth requirements: +5%  e.g., generating hashes is very expensive as e gets large.

53 Outline Introduction Review of LT Codes Strawman #1 Strawman #2 Efficiently Catching the Bad as They Arrive

54 Best of Both Worlds Goal:  Crypto overhead of one hash for every block in the input file (Strawman #1)  Verify blocks as they arrive (Strawman #2) Idea:  Distribute hashes of file blocks, and use them to verify encoded blocks.  Need a better hash function.

55 Insight: Homomorphic Hashing Assume function h exists such that: 1. is homomorphic: 2. is a CRHF:

56 R knows: R wants proof that: Homomorphic Hashing: Intuition R receives the block c b2b2 b5b5 c

57 R knows: R wants proof that: Homomorphic Hashing: Intuition c b2b2 b5b5 c R receives the block

58 R knows: R wants proof that: Homomorphic Hashing: Intuition c b2b2 b5b5 c R receives the block

59 R knows: R wants proof that: Homomorphic Hashing: Intuition R receives the block

60 R knows: R wants proof that: Homomorphic Hashing: Intuition Property 1 R receives the block

61 R knows: R wants proof that: Homomorphic Hashing: Intuition Property 1 Property 2 R receives the block

62 Homomorphic Hashing: Protocol R receives the block  Compute h(c)  If Accept block; mark as valid  else Suspect sender of being bad guy, and switch.

63 Homomorphic Hashing: Protocol R receives the block  Compute h(c)  If Accept block; mark as valid  else Suspect sender of being bad guy, and switch. Can such an h possibly exist?

64 Homomorphic Hashing: Related Work DLog-Based CRHF  Pederson Commitment [CRYPTO ’91]  Chaum et al. [CRYPTO ’91] One-Way Accumulators  Benaloh and de Mare [EUROCRYPT ’93]  Barić and Pfitzmann [EUROCRYPT ’93] Incremental Hashing  Bellare et al. [CRYPTO ’94] Homomorphic Signatures  Micali and Rivest [RSA ’02]  Johnson et al. [RSA ’02]

65 Mechanics of Homomorphic Hashing Discrete Log Hash Pick 1024-bit prime p and 256-bit prime q, q divides (p-1) Pick from 512 generators of order q : Write F as elements in F= 256-bit “fragment” 16K “block”

66 How to Encode (example) Standard LT-Codes: Homomorphic Scheme:

67 How To DLog Hash Hashes are elements in (128 bytes big) Hash reduces 16K block by a factor of 128

68 How To DLog Hash Hashes are elements in (128 bytes big) Hash reduces 16K block by a factor of 128  +1% overhead

69 DLog-Hash: Key Property Note that:

70 DLog-Hash: Key Property Note that: Goal achieved!

71 “This Seems Really Expensive” Operation on a 16K Block Throughput (kB/sec) DLog Hash39 Arrival on 1.5Mbps DSL190 SHA1 Hash57,600

72 Key Optimizations Hash Generation  Each publisher picks her own parameters,  compute with 1 exponentiation (not 512) Hash Verification  Receiver verifies hashes probabilistically and in batches. Bellare et al. [EUROCRYPT ’98]

73 Much Better Operation on a 16K Block Throughput (MB/sec) Naïve DLog Hash0.038 Per-publisher Generation11.210 Batch Verification7.620 Arrival on 1.5 Mbps DSL0.186 SHA1 Hash56.250

74 Homomorphic Hashing: Key Points + Key Algebraic Feature + Homomorphism: Receivers can compose hashes the way encoders sum file blocks. + Can check encoded blocks as they arrive. + Fast + Can be optimized to achieve good generation and verification throughputs + Provably Secure + As hard as discrete log (SHA1/MD5 not needed)

75 Conclusion Sources Can Multicast Clients Get Fast Downloads Clients Can Verify Blocks On-the-Fly

76 Thank you. Now accepting questions.

Download ppt "On-The-Fly Verification of Rateless Erasure Codes Max Krohn (MIT CSAIL) Michael Freedman and David Mazières (NYU)"

Similar presentations

Chapter 3 Public Key Cryptography and Message authentication.

Chapter 3 Public Key Cryptography and Message authentication.
Signatures for Network Coding Denis Charles Kamal Jain Kristin Lauter Microsoft Research.

Signatures for Network Coding Denis Charles Kamal Jain Kristin Lauter Microsoft Research.
Efficient Signature Generation by Smart Cards 20103112 Suk Ki Kim 20103114 Sunyeong Kim.

Efficient Signature Generation by Smart Cards Suk Ki Kim Sunyeong Kim.
On-the-fly Verification of Erasure-Encoded File Transfers Mike Freedman & Max Krohn NYU Dept of Computer Science.

On-the-fly Verification of Erasure-Encoded File Transfers Mike Freedman & Max Krohn NYU Dept of Computer Science.
Spreading Alerts Quietly and the Subgroup Escape Problem Aleksandr Yampolskiy (Yale) Joint work with James Aspnes, Zoë Diamadi, Kristian Gjøsteen, and.

Spreading Alerts Quietly and the Subgroup Escape Problem Aleksandr Yampolskiy (Yale) Joint work with James Aspnes, Zoë Diamadi, Kristian Gjøsteen, and.
Advanced Security Constructions and Key Management Class 16.

Advanced Security Constructions and Key Management Class 16.
Authentication and Digital Signatures CSCI 5857: Encoding and Encryption.

Authentication and Digital Signatures CSCI 5857: Encoding and Encryption.
Data Persistence in Sensor Networks: Towards Optimal Encoding for Data Recovery in Partial Network Failures Abhinav Kamra, Jon Feldman, Vishal Misra and.

Data Persistence in Sensor Networks: Towards Optimal Encoding for Data Recovery in Partial Network Failures Abhinav Kamra, Jon Feldman, Vishal Misra and.
CS426Fall 2010/Lecture 351 Computer Security CS 426 Lecture 35 Commitment & Zero Knowledge Proofs.

CS426Fall 2010/Lecture 351 Computer Security CS 426 Lecture 35 Commitment & Zero Knowledge Proofs.
1 Authenticated key agreement without using one-way hash functions Harn, L.; Lin, H.-Y. Electronics Letters, Volume: 37 Issue: 10, 10 May 2001 Presented.

1 Authenticated key agreement without using one-way hash functions Harn, L.; Lin, H.-Y. Electronics Letters, Volume: 37 Issue: 10, 10 May 2001 Presented.
Security Chapters 14,15. The Security Environment Threats Security goals and threats.

Security Chapters 14,15. The Security Environment Threats Security goals and threats.
Network Coding for Large Scale Content Distribution Christos Gkantsidis Georgia Institute of Technology Pablo Rodriguez Microsoft Research IEEE INFOCOM.

Network Coding for Large Scale Content Distribution Christos Gkantsidis Georgia Institute of Technology Pablo Rodriguez Microsoft Research IEEE INFOCOM.
CNS2010handout 10 :: digital signatures1 computer and network security matt barrie.

CNS2010handout 10 :: digital signatures1 computer and network security matt barrie.
Dr Alejandra Flores-Mosri Message Authentication Internet Management & Security 06 Learning outcomes At the end of this session, you should be able to:

Dr Alejandra Flores-Mosri Message Authentication Internet Management & Security 06 Learning outcomes At the end of this session, you should be able to:
Fountain Codes Amin Shokrollahi EPFL and Digital Fountain, Inc.

Fountain Codes Amin Shokrollahi EPFL and Digital Fountain, Inc.
Introduction to Modern Cryptography Homework assignments.

Introduction to Modern Cryptography Homework assignments.
1 How to securely outsource cryptographic computations Susan Hohenberger and Anna Lysyanskaya TCC2005.

1 How to securely outsource cryptographic computations Susan Hohenberger and Anna Lysyanskaya TCC2005.
CMSC 414 Computer and Network Security Lecture 9 Jonathan Katz.

CMSC 414 Computer and Network Security Lecture 9 Jonathan Katz.
Secure Hashing and DSS Sultan Almuhammadi ICS 454 Principles of Cryptography.

Secure Hashing and DSS Sultan Almuhammadi ICS 454 Principles of Cryptography.
Electronic Voting Schemes and Other stuff. Requirements Only eligible voters can vote (once only) No one can tell how voter voted Publish who voted (?)

Electronic Voting Schemes and Other stuff. Requirements Only eligible voters can vote (once only) No one can tell how voter voted Publish who voted (?)

Similar presentations

Ads by Google