1. Hashing THEN AND NOW MIKE SMORUL – ADAPT PROJECT

2. Commodity Storage Performance  2003 JetStor III IDE-FC  62MB/s large block  2013  218MB/s workstation SSD  Perc 6/MD1000, 400MB/s+

3. Chip Speed  2003: Pentium 4  3.2Ghz  2013: Core i7 Extreme  3.5Ghz

4. Hashing Performance  SHA-256 Hashing  Java: 85MB/s  Crypto++: 111-134MB/s  Real World Penalty  Java: 20-40% penalty on slow seek disk

5. Implications  Flipped bottlenecks

6. How to overcome  Faster/weaker digests  Simultaneous transfers  Data locality, tape?  Improve single stream performance

7. Parallelize Single Stream  Independent IO and digest threads  Always have work for the digest algorithm.  Large files saw over 95% of algorithm potential.  Small files unchanged.

8. Securing Data in Motion ?

9. Where to apply fixity  Internal integrity services  At Transfer via manifests  End to End?

10. Operational Integrity  Internal Auditing  Prove your hardware  Error, not malice detection  Peer-Auditing  Prove your friends

11. Transporting Integrity  Manifest Lists  Transfer validation  Digital Signatures  Prove identity  Token Based  Prove time

12. Chronopolis Integrity  Current:  Producer supplied authoritative manifest  Peers locally monitor integrity  Manually trace back to point of ingest

13. Chronopolis Integrity  In-progress  Single integrity token back to ingest  Ideal  Tokens issued prior to arrival  ‘Prove’ the state of data to point before Chronopolis

14. Manifests 2.0  Beyond simple transfer list  Token manifests  Portable, embeddable  Python, etc

15. Cloud Integrity  Digests in a cloud validate transfer only  Http headers can pass extended integrity information  End-user verification

16. Integrity as provenance  Integrity checking forward in time  Consumer level verification of data  Integrity from object creation  Start integrity checking before archiving

17. Closing  Why are you hashing?  What do you want to prove?  Hashing Cost/performance

18. Contact Mike Smorul msmorul@sesync.org http://adapt.umiacs.umd.edu/ace