1. PCI Express® 4.0 and 5.0: The Future of NVMe™ Technology 2019 NVMe™ Annual Members Meeting and Developer Day March 19, Prepared by Justin Wenck, Senior Technical Marketing Engineer at Intel, PCI-SIG

2. Agenda PCI-SIG Overview PCIe 5.0 Specification Form Factors Summary
NVMe & PCIe 3.0
PCIe 4.0 Specification
Today we’ll go over PCI-SIG and the markets the consortioum addresses.

3. PCI-SIG Snapshot 800+ member companies located worldwide.
Board of Directors
Gord Caruk Fellow Design Engineer
Greg Casey Senior Architect & Strategist, Server CTO Team
Michael Krause
VP and Fellow
Al Yanes
Distinguished Engineer
Debendra Das Sharma
Intel Fellow, Director of I/O Technology & Standards Group
Rick Eads
Principal Program Manager
Michael Diamond
Sr. Director of Strategic Marketing
Rick Wietfeldt
Senior Technologist
Richard Solomon
Technical Marketing Manager
Organization that defines the PCI Express® (PCIe®) I/O bus specifications and related form factors.
800+ member companies located worldwide.
Creating specifications and mechanisms to support compliance and interoperability.
Market categories of the newer PCI-SIG members:
Virtual reality
Automotive
Artificial intelligence
Telecommunications
Storage
SSD controllers
Tablet/PC R&D
Peripheral Component Interconnect Special Interest Group is an electronics industry consortium responsible for specifying the Peripheral Component Interconnect, PCI-X, and PCI Express computer buses. 
It has wide industry support, with over 800 member companies worldwide. PCI-SIG has about 3 decades of history of creating interoperable interfaces.

4. Market Segments Addressed by PCIe®
Automotive
High-performance
Reliability
Availability
Serviceability
Cloud
Scalable architecture
Increased performance
Reduced TCO
Artificial Intelligence
High-performance
High-bandwidth
Enterprise Servers
Redundancy/failover
Ubiquity
Power savings
PC/Mobile/IoT
Faster performance
Power efficiency
Low latency
Storage
Faster data transfer
Better user experience
Ubiquity
PCI-SIG Standards now target market segments that go well beyond the PC:
Virtual reality
Automotive
Artificial intelligence
Telecommunications
Storage
SSD controllers
Tablet/PC R&D

5. PCIe® Roadmap: Specification Completion
2019
2017
PCIe 5.0
@ 32GT/s
2010
PCIe 4.0
@ 16GT/s
2006
2003
PCIe 3.0
@ 8GT/s
PCIe 2.0
@ 5.0GT/s
Continuous improvement with data rate as well as usage models; doubling bandwidth and improving capabilities
PCIe 1.0
@ 2.5GT/s
To satisrfy more markets PCI-SIG has been eveolving the spec with lower features and BW doubling about every 3 years.
In Planning

6. PCIe*/NVMe*. Design for a Range of Objectives.
Low Power
Server Performance
High Performance
PCIe
Form Factors
CEM Add-in-card
M.2
U.2
42, 80, and 110mm lengths, smallest footprint of PCIe connector form factors, use for boot, for max storage density, for PXI/AXIe ecosystem
2.5in makes up the majority of SSDs sold today because of ease of deployment, hotplug, serviceability, and small form factor
Single-Port x4 or Dual-Port x2
Add-in-card (AIC) has maximum system compatibility with existing servers and most reliable compliance program. Higher power envelope, and options for height and length
PCI-SIG Develops a number of Form Factors to service those markets, from the low power M.2, to the Mainstream U.2, with over 6M U.2 SSDs shipped by 2018, over 20M by To the AIC which can scale to high performance, high power use cases.
Sgnaling rate of the various form factors and specification version are not linked. The form factors evolve based on industry demand.
In addition to the Form Factors developed by PCI-SIG. The EDSFF form factors will are PCIe compatible for Gen 3 and 4, and currently there no limitations for Gen 5 support. The EDSFF form factor allows for higher storage densities with better thermal performance
SFF-TA-1002: Protocol Agnostic Multi-Lane High Speed Connector
SFF-TA-1006: Enterprise and Datacenter 1U Short SSD Form Factor
SFF-TA-1007: Enterprise and Datacenter 1U Long SSD Form Factor
SFF-TA-1008: Enterprise and Datacenter 3” Media Device Form Factor
Flexible performance
Higher capacities
Optimized power and thermal management
Choose a drive for your design target
BW/TB – 1TB, 2TB
Capacity/RU – 16TB, 32TB, Ruler form factor
Thermal efficiency – Ruler
IOPS/Watt – 7mm U.2

7. NVM Express™ and PCI Express 3.0
NVMe and PCIe enable performance and latency scalability
PCI Express benefits include; scalable performance through lane utilization, lower power consumption, and reduced latency

8. PCIe/NVMe*. Extending Usable Capacity.1
PCIe
PCIe/NVMe interface accelerates workloads at larger capacities
2.8x
SATA*
4.3x
6.9x
At higher capacities, SATA interface limits performance capabilities
7.6x
Performance required by certain storage workloads2
Source – Intel: Intel® SSD D5-P4320 results are simulated and SATA* data is from Micron datasheet. PCIe* IOPS based on simulated 4K random read, queue depth of 256, performance estimates conducted by Intel for the Intel D5-P4320/D5-P4326 PCIe-based QLC SSD at different capacities: 3.84TB; 7.68TB; 15.36TB, and 30.72TB. SATA IOPs set to 100K IOPs for all capacity points based on 100K IOPS being the max possible for current competitive SATA base SSDs from Micron. The Micron 5200 Series* NAND Flash SSDs data sheet showing a max 4K random read QD32 IOPs of 95K IOPs for 3.84TB, and 7.68TB SKUs.Datasheet located here: Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit Performance results are based on testing as of July 2018 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure
Usable capacity threshold.
*Other names and brands may be claimed as property of others.
SATA cannot take advantage of large capacity TCO savings
TCO improves with SSD Capacity
SATA performance is capped causing performance per TB to decline to unusable levels with high capacity drives
PCIe performance continues to scale with capacity and is much higher overall enabling usable performance/TB at very high capacities and great TCO savings
PCIe enables large capacity TCO savings
Performance per capacity is much higher than SATA
Performance per capacity stays well above the usability threshold of storage workloads even at very high capacities
Based on 4TB, 8TB, 16TB, 32 TB P4320 QD 256 vs 4TB, 8TB, 16TB, 32TB S4510 QD32 . Extreme workload based off averages of 8192 IOPS/TB

9. PCIe*/NVMe*. Widening the Performance Advantage.
SATA and SAS interfaces are saturated
PCIe performance delivers scalability for emerging workloads and storage consolidation
26x faster
up to
6.5x faster
up to
Continued solution-wide innovation
Last increase in SATA was 2008
In fact both SATA and SAS in typical configurations are often saturated
Meanwhile, PCIe interfaces has continued to evolve
PCIe 3.1 is currently 6.5X faster than SATA
With PCIe 4.0 products that will begin coming to market in this coming year, the gap increases to 13X SATA
PCIe 5 products that will be introduced in 2021 or later further widen the advantage to 26X over SATA
Performance is important for a few reasons
PCIe will support larger capacities – SATA drops below the IOPS/TB required Weby some storage workloads (more on this later)
Emerging workloads such as AI and Big Data will drive even more stringent performance requirements
PCIe works across multiple media types – NAND and Optane
Current gen interfaces compare SATA 3.0, SAS 3.0 and PCIe 3.1 x 4 and interface speed is publically available. PCIe 4.0 specification is available at PCIe 5.0 performance is based on target from PCI-SIG.

10. Required BW for Uncompressed Video
Some might ask, who would ever need that much performance? The content creation, editing, and distribution would love that performance..yesterday. An example of how PCIe Gen 3, 4, and 5 will enable the explosive growth of video content is shown on this slid.e The line shows the data rate of uncompressed video for various video standards. We see here that SDTV requires 31 MB/s. A SATA drive can easy handle that data rate, same for HDT and even 2k. Once you get to UHD 4k and Digital Cinema 4k, SATA is no longer sufficient and PCIe Gen4 is reuired. Moving to UHD 8k would require either 16 lanes of Gen 3 or Gen 4 with 8 lanes. Moving to Digital Cinema 8K, Gen 3 is no longer enough and 16 or 8 lanes of Gen 4 and 5 are needed.
Source: T. Coughlin, "How Big Are Your Dreams? Gauging the Size of Future Content [The Art of Storage]," in IEEE Consumer Electronics Magazine, vol. 6, no. 2, pp , April 2017.

11. Transition is Here. Pricing Will Accelerate.
Data explosion is driving SSD adoption
SSD market CAGR of 14.8% during Source: IDC
PCIe SSD market to surpass a CAGR of 33% during Source: Technavio
PCIe adoption is accelerating
Forward Insights. SSD Insights Q4/18. November 2018.
Driven by tier one Cloud Service Providers that transition to PCIe is well underway.
According to research firm Forward Insights, PCIe shipments to the data center accounted for 50 percent in the third quarter of 2018.
And when you consider where pricing is headed we believe the transition will accelerate.
In the third quarter of 2018 industry average pricing for PCIe and SATA where at parity.
This transition is likely to accellerate thanks to improving price of PCIe NVMe SSD’s and improved performance that PCIe Gen 4 will bring which will bring even more TCO advantages to performance and latency sensitive applications
Source: SSD Insights Q1/18, Forward Insights

12. TCO Savings Over 5 Years Same Performance ~ 50GB/s
SATA- Intel S4510
PCIe 3.0- Intel P4510
PCIe 4.0- Future Drive
PCIe 5.0- Future Drive
SATA- Intel S4510
PCIe 3.0- Intel P4510
PCIe 4.0- Future Drive
PCIe 5.0- Future Drive
TCO Savings
Baseline
$62,117
$84,394
$97,198
Total Drives 90 16 8 4
Sequential Read (MB/s)
560
3,200
6,400
12,800
Power Consumption
- 8%
- 52%
- 75%
Test-estimator.intel.com/ssddc/
Assumptions:
SATA Intel S4510: TB ea., full load, $1037/drive
PCIe 3.0 Intel P4510: 16 8TB ea., full load, $2800/drive
PCIe 4.0 Future Drive: 8 8TB ea., full load, $3000/drive *Estimate
PCIe 5.0 Future Drive: 4 8TB ea., full load, $3000/drive *Estimate
In storage it can be common to only think of $/GB when it comes to value, but really, it’s total cost of ownership or TCO that matters, and this can vary greatly depending on your application. Typically PCIe based drives carry a premium compared to SSD’s on other interfaces. For performance heavy applications, that extra performance can result in significant TCO savings. Let’s take an application where you want to be able to stream 50GB/s of dataUsing sata SSD’s you would need 90 drives to satisfy that amount of read BW. Moving to PCIe based drives, which have much higher read BW, reduces the drive count to 16, which reduces the amount of Rackspace required and power and ultimately the cost. Looking forward PCIe Gen 4 and 5, even if we don’t get reduced costs of the drives, which is likely untrue, we see reduced cost even more. That’s even assuming a premium price on the drives. This is absolutely a contrived example, and I know there are plenty of specific examples where you can show better TCO of using HDD’s with SATA, which is why that use case is not disappearing any time soon. The point is that there are more and more applications where PCIe SSD’s will give TCO advantage.

13. PCI Express 4.0 Specification
Adoption Is Well Under Way
Key Features:
Delivers 16 GT/s
Maintains backward compatibility with PCIe 3.x, 2.x, and 1.x
Implements:
Extended tags and credits
Reduced system latency
Lane margining
Superior RAS capabilities
Scalability for added lanes and bandwidth
Improved I/O virtualization and platform integration
Compliance Status:
PCI-SIG Launched Official FYI Testing for PCIe 4.0 in December 2018
>50 participants in pre-FYI testing
Formal Compliance testing targeted for Q3 2019
Adoption:
Numerous vendors with 16GT/s PHYs and controllers in silicon
Test equipment from multiple vendors
Several member companies have publicly announced & exhibited PCIe 4.0 products
The Gen 4 Spec was finalized and published in Oct 2017 and doubles the per lane speed to 16 GT/s while still maintaining backwards compatibility. There are numerous silicon proven PHYs at this speed and Major IP vendors are currently offering controllers for easy integration into products. Also, the PCI-SIG Compliance Workshops have recently had FYI testing on serveral Gen 4 Hosts and Endpoints. with formal compliance testing targeting Q3 of this year.
In addition to increased speed, additionl features include
Extended tags and credits
Reduced system latency
Lane margining
Superior RAS capabilities
Scalability for added lanes and bandwidth
Improved I/O virtualization and platform integration
Improved Retimer ssupport

14. PCI Express 5.0 Specification
PCI Express 5.0, Rev 0.9 Released in November 2018; Rev 1.0 on Target for Q1 2019
Key Features:
Delivers 32 GT/s
Maintains backward compatibility with PCIe 4.0, 3.x, 2.x, and 1.x
Changes limited primarily to speed upgrade
Protocol already supports higher speed via extended tags and credits
Electrical changes to improve signal integrity and mechanical performance of connectors
CEM connector backwards compatible for add-in cards
RAW
BIT RATE
LINK BW
BW/
LANE/WAY
TOTAL
BI DIR BW X16
PCIe 5.0
32GT/s
32Gb/s
~4GB/s
~128GB/s
PCIe 4.0
16GT/s
16Gb/s
~2GB/s
~64GB/s
PCIe 3.x
8.0GT/s
8Gb/s
~1GB/s
~32GB/s
PCIe 2.x
5GT/s
4Gb/s
500MB/s
16GB/s
PCIe 1.x
2.5GT/s
2Gb/s
250MB/s
8GB/s
Much of what’s needed to enable the 32GT/s speed has already been put into the Gen 4 protocol, and PHY’s for other interconnect technologies already run at 28GT/s and 56GT/s. This has enabled the role out of Gen 5 to be accelerated compared to the gen 3 to 4 timeline. The 5.0 Spec was released in November of 2018 and the 1.0 spec is on target for being released by the end of the month.
In 27 years PCI-SIG has gone from 133/MBs to 32GB/s with 16 lanes in Gen 4, a 240X increase! With Gen 5 this doubles again to 64GB/s peak 1 way BW.
Leverages and adds to the PCIe 4.0 specification and its support for higher speeds via extended tags and credits Implements electrical changes to improve signal integrity and mechanical performance of connectors Includes new backwards compatible CEM connector targeted for add-in cards Maintains backwards compatibility with PCIe 4.0, 3.x, 2.x and 1.x   The new specification increases performance in the high-performance markets including artificial intelligence, machine learning, gaming, visual computing, storage and networking

15. The Future of PCIe and NVMe™
PCIe 4.0 and 5.0 continue the trend of doubling bandwidth every generation, while maintaining backward compatibility
PCIe 4.0 & 5.0 add new capabilities and bring additional TCO savings to use cases that are performance-critical and latency-sensitive
NVMe/PCIe integration benefit a variety of markets, including data centers, IoT, autonomous driving, machine learning and more
This needs some work here

16. Join the Conversation with PCI-SIG!
Follow. Watch. Like.
Join the Conversation with PCI-SIG!
PCI-SIG
@PCI_SIG
linkedin.com/company/pcisig/

17. Questions?

18. Back-Up Slides

19. PCI Express 3.0 Specification
Specification Completed: November 2010
Target Applications:
Meets demand for increased storage I/O speeds
Compliance Program:
Multiple companies have added PCIe-compliant products to the PCI-SIG Integrators List
CEM Add-in Cards: 298 products
Systems: 20
Components: 226
U.2 Form Factors: 2
Key Features:
Delivers 8 GT/s
Implements:
New encoding scheme from 8b/10b to 128b/130b allows near 100% complete efficiency, enabling bandwidth doubling
Dynamic Feedback Equalization (DFE) to improve the PCIe bus signal integrity
New low power features: L1 Substates
Maintains backward compatibility with PCIe 2.X and 1.X
Visit: