1. Platform Architecture Lab USB Performance Analysis of Bulk Traffic Brian Leete brian.a.leete@intel.com

2. Platform Architecture Lab 2 Introduction Bulk Traffic u Designed for reliable, highly variable data transfer u No guarantees are made in the specification for throughput u Is scheduled last after ISOC, Interrupt, and Control u Throughput is dependant on many factors

3. Platform Architecture Lab 3 Introduction We will look at Bulk Throughput from the following aspects u Distribution of Throughput for Various Packet Sizes and Endpoints u Low Bandwidth Performance u Small Endpoint Performance u Nak Performance u CPU Utilization u PCI bus Utilization

4. Platform Architecture Lab 4 Test Environment -- Hardware PII 233 (8522px) with 512 Bytes Cache Atlanta Motherboard with 440LX (PIX 4A) Chipset 32 Meg Memory Symbios OHCI Controller (for OHCI Measurements) Intel Lava Card as Test Device

5. Platform Architecture Lab 5 Test Environment -- Software Custom Driver and Application Test Started by IOCTL IOCTL allocates static memory structures, submits IRP to USBD Completion routine resubmits next buffer All processing done at ring 0, IRQL_DISPATCH

6. Platform Architecture Lab 6 Terminology A “Packet” is a Single Packet of Data on the Bus. It is determined by Max Packet Size of the Device u Valid numbers are 8, 16, 32, 64 A Buffer is the amount of data sent to USBD in a Single IRP. u In this presentation buffers range from 8 Bytes to 64K Bytes Unless otherwise specified, Most Data Taken at 64 Byte Max Packet Size, 15 Endpoints Configured in the System

7. Platform Architecture Lab 7 Host Controller Operation (UHCI)

8. Platform Architecture Lab 8 Single Endpoint Throughput Flat Throughput @ 512 and 1024 Byte Buffers Oscillations @ 256, 512 Byte Buffers Small Buffer Throughput

9. Platform Architecture Lab 9 Small Buffer Throughput For Buffer Sizes < Max Packet Size Host Controller sends 1 Buffer per Frame No Ability to Look Ahead and Schedule Another IRP Even Though Time Remains in the Frame Why is this?

10. Platform Architecture Lab 10 Interrupt Delay

11. Platform Architecture Lab 11 Single Endpoint Graph Flat Throughput @ 1024 and 512 Byte Graphs Single Ended Throughput for 64K Byte Buffers Below Theoretical Max of 1216000 Bytes per Second Both are explained by Looking at the Number of Packets per Frame

12. Platform Architecture Lab 12 Maximum Packets per Frame

13. Platform Architecture Lab 13 Throughput for Multiple Endpoints 512 Byte Buffers

14. Platform Architecture Lab 14 512 Byte Buffers 1 Endpoint 8 Packets * 64 Bytes per Packet = 512,000 B/S u 511986 Measured

15. Platform Architecture Lab 15 512 Byte Buffers 2 Endpoints 16 Packets * 64 Bytes per Packet = 1,024,000 B/S u 1,022,067 B/S Measured Notice that Interrupt Delay is not a factor here!

16. Platform Architecture Lab 16 512 Byte Buffer -- 3 Endpoints 24 Packets * 64 Bytes / 2 Frames = 768,000 B/S u 776,211 Measured

17. Platform Architecture Lab 17 High End Throughput 18 PPF VS 17 PPF Single Ended Throughput 900,000 VS 950,000 B/S Flat Throughput @ 512 and 1024 B Buffers Small Buffer Throughput Oscillations @ 256 and 512 B buffers

18. Platform Architecture Lab Minimal Endpoint Configuration

19. Platform Architecture Lab 19 Higher Single Endpoint Throughput 17 VS 15 PPF

20. Platform Architecture Lab 20 Host Controller Operation (UHCI)

21. Platform Architecture Lab 21

22. Platform Architecture Lab 22 Results We are working with Microsoft to remove unused endpoints from the Host Controller Data Structures

23. Platform Architecture Lab 23 Higher Single Endpoint Throughput More Endpoints get 18 Packets per Frame

24. Platform Architecture Lab Distribution of Throughput across Endpoints

25. Platform Architecture Lab 25

26. Platform Architecture Lab 26 Results We are working with Microsoft to get the Host Controller driver to start sending packets at the next endpoint rather than starting over at the beginning of the frame.

27. Platform Architecture Lab 27

28. Platform Architecture Lab Limited Bandwidth Operation

29. Platform Architecture Lab 29

30. Platform Architecture Lab 30

31. Platform Architecture Lab Small Endpoint Performance

32. Platform Architecture Lab 32

33. Platform Architecture Lab 33

34. Platform Architecture Lab 34

35. Platform Architecture Lab 35

36. Platform Architecture Lab 36 If you care about throughput…. Use 64 byte Max Packet Size Endpoints Use Large Buffers

37. Platform Architecture Lab Nak Performance

38. Platform Architecture Lab 38

39. Platform Architecture Lab 39 45 % Drop in Total Throughput

40. Platform Architecture Lab 40

41. Platform Architecture Lab 41

42. Platform Architecture Lab CPU Utilization

43. Platform Architecture Lab 43 CPU Utilization Idle process incrementing a counter in main memory u Designed to simulate a heavily CPU bound load Numbers indicate how much “work” the CPU could accomplish after servicing USB traffic u Higher numbers are better Small buffers and large numbers of Endpoints take more overhead u Software Stack Navigation Endpoint 0 is the Control -- No USB Traffic running

44. Platform Architecture Lab 44

45. Platform Architecture Lab 45

46. Platform Architecture Lab PCI Utilization

47. Platform Architecture Lab 47

48. Platform Architecture Lab 48 PCI Utilization (UHCI) 15 Endpoint Configuration For low numbers of active endpoints, Host Controller must poll memory for each unused endpoint, causing relatively high utilization. Removing unused endpoints will lower single endpoint PCI utilization for this configuration.

49. Platform Architecture Lab 49 Conclusions UHCI Host Controller Driver needs a few tweaks u Need to get Host Controller to start sending packets where it last left off rather than at endpoint 1. u Needs to remove unused endpoints from the list Performance Recommendations u Use 64 Byte Max Packet Size Endpoints u Large Buffers are better than small buffers u Reduce NAK’d traffic l Fast devices if possible

50. Platform Architecture Lab 50 Future Research Topics Multiple IRPS per Pipe USB needs to control throughput to the slow device u Small Endpoints aren’t good u Small Buffers aren’t good u NAKing isn’t good