1. LZRW3 Decompressor dual semester project Characterization Presentation Students: Peleg Rosen Tal Czeizler Advisors: Moshe Porian Netanel Yamin 6.4.2014

2. Presentation Content Project Motivation and Goals Project Requirements Background – LZ77 algorithm LZRW3 algorithm Compressed file format Proposed Solution Decompression Example Project Top Block Diagram Planned GUI Project Schedule and Gantt

3. Project Motivation Why do we need an hardware data compression?

4. -Reduce storage capacity Project Motivation

5. Why do we need an hardware data compression? -Reduce storage capacity -Reduce amount of data to be handled Project Motivation

6. Why do we need an hardware data compression? -Reduce storage capacity -Reduce amount of data to be handled -Reduce communication costs Project Motivation

7. Why do we need an hardware data compression? -Reduce storage capacity -Reduce amount of data to be handled -Reduce communication costs -Speed Project Motivation

8. Project Goals

9. Implementation of LZRW3 data decompression core.

10. Implementation of a verification environment. Project Goals

11. Project Requirements

12. Part A: Core Requirements: – Process data at the speed of 1 Gbps. – Support data blocks with output of 2KB – 32KB. – Relay only on the FPGA’s internal memory. – VHDL Implementation.

13. Part A: Core Requirements: – Process data at the speed of 1 Gbps. – Support data blocks with output of 2KB – 32KB. – Relay only on the FPGA’s internal memory. – VHDL Implementation. Full simulation environment (golden model and checkers). Project Requirements

14. Part A: Core Requirements: – Process data at the speed of 1 Gbps. – Support data blocks with output of 2KB – 32KB. – Relay only on the FPGA’s internal memory. – VHDL Implementation. Full simulation environment (golden model and checkers). Part B: Synthesis & implementation of FPGA device (Xilinx Virtex-5). Project Requirements

15. Part A: Core Requirements: – Process data at the speed of 1 Gbps. – Support data blocks with output of 2KB – 32KB. – Relay only on the FPGA’s internal memory. – VHDL Implementation. Full simulation environment (golden model and checkers). Part B: Synthesis & implementation of FPGA device (Xilinx Virtex-5). GUI implementation in VisualStudio. Project Requirements

16. Background – LZ77 algorithm

17. Lossless data compression algorithm that was published by Abraham Lempel and Jacob Ziv in 1977.

18. Output item : (offset, length ) + next literal In this case Output item = (, ) Meaning : BABDAC ABDBCAA History bufferLook-ahead buffer 32451 5 123 3 B Background – LZ77 algorithm Lossless data compression algorithm that was published by Abraham Lempel and Jacob Ziv in 1977. -> Take 3 literals -> Next literal will be BGo 5 steps back

19. Output item (Copy item): [slot address, length ] In this case Output item = [, ] BABDACABDBCAA 2 1 3 0 3 LZRW3 compression algorithm Hash Function Hash Table ABD Slot address 1 45 6 BAB 0 ABD Slot address Send every 3 literals to the hash function Put offset in the hash table If the slot is occupied and the literals match - make copy item 6

20. Compressed file format Structure

21. File header (8 byte) Compressed file format

22. Structure File header (8 byte) Groups: Compressed file format

23. Structure File header (8 byte) Groups: - control bytes (2 bytes) Compressed file format

24. Structure File header (8 byte) Groups: - control bytes (2 bytes) - data bytes (16 - 32 bytes) * The last group might be smaller

25. Compressed file format File header contains information about the file size and whether it is compressed or not.

26. Compressed file format Control bytes Hold the information about the items type (copy or literal).

27. Compressed file format Literal items Single bytes of data in their original form.

28. Compressed file format Copy items Two bytes representing hash table slot holding the offset and length of a literal sequence.

29. Proposed Solution Header decoding Determine file size and if it is compressed or not.

30. Proposed Solution Header decoding Determine file size and if it is compressed or not. File decompressed! File not compressed

31. Proposed Solution Header decoding Determine file size and if it is compressed or not. Control bytes decoding Determine number and location of copy items. File decompressed! File compressed File not compressed

32. Proposed Solution Header decoding Determine file size and if it is compressed or not. Control bytes decoding Determine number and location of copy items. File decompressed! File compressed File not compressed Group decoding Literal items: 1. Store in hash table 2. Copy Copy items: 1. Decode 2. Replace 3. Copy

33. Proposed Solution Header decoding Determine file size and if it is compressed or not. Control bytes decoding Determine number and location of copy items. File decompressed! File compressed File not compressed Next group Group decoding Literal items: 1. Store in hash table 2. Copy Copy items: 1. Decode 2. Replace 3. Copy

34. Proposed Solution Header decoding Determine file size and if it is compressed or not. Control bytes decoding Determine number and location of copy items. Group decoding Literal items: 1. Store in hash table 2. Copy Copy items: 1. Decode 2. Replace 3. Copy EOF File decompressed! File compressed File not compressed Next group

35. Decompression example

36. Header decoding The header is composed of 8 bytes: Meaningless zeroes

37. Decompression example Header decoding The header is composed of 8 bytes: File size information: SIZE = (0 x 256) + 35 + 4 = 39 Byte 3 Byte 4

38. Decompression example Header decoding The header is composed of 8 bytes: Compression indicator: 0 = compressed 1 = not compressed

39. Decompression example Meaningless zeroes Header decoding The header is composed of 8 bytes:

40. Decompression example Control Bytes Analysis The control bytes are consisted of 2 bytes, each bit represent an item in the group:

41. Decompression example Control Bytes Analysis The control bytes are consisted of 2 bytes, each bit represent an item in the group: 0 – A literal item in the appropriate group position 1 - A copy item in the appropriate group position

42. Decompression example Control Bytes Analysis The control bytes are consisted of 2 bytes, each bit represent an item in the group: 0 – A literal item in the appropriate group position 1 - A copy item in the appropriate group position

43. Decompression example Control Bytes Analysis The control bytes are consisted of 2 bytes, each bit represent an item in the group: 0 – A literal item in the appropriate group position 1 - A copy item in the appropriate group position

44. Decompression example Data Bytes Analysis Bytes 11-22: literal item. Sent to hash function and copied as is to output file.

45. Decompression example Data Bytes Analysis Bytes 11-22: literal item. Sent to hash function and copied as is to output file.

46. Decompression example Data Bytes Analysis Bytes 11-22: literal item. Sent to hash function and copied as is to output file.

47. Compressed stream Hash func. Offset Hash table Decompression example Hashing The decompressor maintains a hash table identical to the one created by the compressor.

48. Decompression example Data Bytes Analysis Bytes 23-24: copy item

49. Decompression example Data Bytes Analysis Bytes 23-24: copy item

50. Decompression example Data Bytes Analysis Bytes 23-24: copy item

51. Decompression example Data Bytes Analysis Bytes 23-24: copy item Length = 0 + 3

52. Decompression example Data Bytes Analysis Bytes 23-24: copy item The number of copied literals is 3. The offset is found in slot 3645 in the hash table. Length = 0 + 3

53. Decompression example Data Bytes Analysis Bytes 23-24: copy item The number of copied literals is 3. The offset is found in slot 3645 in the hash table. Length = 0 + 3

54. Project Top Block Diagram LZRW3 DECOMPRESSION CORE

55. Project Top Block Diagram LZRW3 DECOMPRESSION CORE Serial compressed data 115200 bit/sec

56. Project Top Block Diagram LZRW3 DECOMPRESSION CORE Parallel data (bytes) wishbone protocol 125MHz REUSE

57. Project Top Block Diagram LZRW3 DECOMPRESSION CORE Compressed data bytes 125 MHz REUSE

58. Project Top Block Diagram LZRW3 DECOMPRESSION CORE Decompressed data bytes 125 MHz REUSE

59. Project Top Block Diagram LZRW3 DECOMPRESSION CORE Decompressed bytes wishbone protocol 125 MHz REUSE

60. Project Top Block Diagram LZRW3 DECOMPRESSION CORE Serial decompressed data 115200 bit/sec REUSE

61. Load uncompressed file PROCESS & COMPARE Input size x x Output (golden model): Messages (progress): (sending packets to FPGA/ decompressing/ getting files back/ comparing results) (browse) RAW INPUT COMPRESSED INPUT STATUS Planned GUI Load expected output file (browse) DECOMPRESS Load compressed file (browse) COMPARE Manually insert data PROCESS & COMPARE

62. DateGoals 21/3/2014 – 5/4/2014Project Characterization & Algorithm interpreting 6/4/2014Characterization Presentation 7/4/2014 – 20/4/2014Full Characterization of all blocks 21/4/2014 – 10/6/2014 System blocks VHDL Design 11/6/2014Mid presentation 11/6/2014 – 25/6/2014Building a simulation environment 26/6/2014 – 25/7/2014Work on project paused for exams Project Schedule 1/2

63. DateGoals 30/7/2014 – 3/9/2014Simulation run & debug 4/9/2014Part A - Final presentation 21/9/2014 – 5/10/2014FPGA synthesis & implementation 6/10/2014 – 23/10/2014GUI implementation 24/10/2014 – 8/11/2014Tests & debug 9/11/2014 – 23/11/2014Writing project portfolio 24/11/2014Final project presentation Project Schedule 2/2

64. Weeks: 0 - 5 6 – 12 13 – 19 20 - 26 27 - 32 Characterization & interpretation Characterization presentation Blocks characterization VHDL blocks implementation Mid presentation Simulation environment Exams Simulation – Cont. Part A - Final pres. FPGA synthesis GUI implementation Tests & debug Writing portfolio Final presentation ……...........………...…………… ….…….……………………………………………………….…............ 4 4 4 4 7 7 2 2 2 2 2 2 2 2 3 3 Project Gantt 2 2 ….…….……………………………………….