1. PQI vs. NVMe® Queuing Comparison
T10 CAP discussion for PQI
Ie-Wei Njoo, PMC-Sierra
5 May,2011

2. Reference and acknowledgement
All references to of NVMe® entities are based on the publicly available NMVe specification:

3. PQI device Inbound – the direction from the PQI host to the PQI device
Definitions
PQI device Inbound – the direction from the PQI host to the PQI device
IQ – Inbound Queue
PQI device Outbound – the direction from the PQI device to the PQI host
OQ – Outbound Queue
NVMe Submission – the direction from the NVMe host to the NVMe device
SQ – Submission Queue
NVMe Completion – the direction from the NVMe device to the NVMe host
CQ – Completion Queue

4. High Level Comparison Similarity
Subject
PQI
NVMe
Circular queue with memory buffer in host
Yes
Queuing priority
Queue mapping to interrupt (MSI-x)
Interrupt Coalescing
Administrator function
PCIe register set for interrupt mask function
Interrupt Mask Set
Interrupt Mask Clear
Interrupt Mask register
Interrupt Mask Clear register
PCIe register for controller status
PQI device Status register
Controller Status register
PCIe register set for circular queue buffer in host memory
Admin IQ PQI host Memory Address
register
Admin OQ PQI host Memory Address
Admin Submission Queue Base Address
Admin Completion Queue Base Address

5. High Level Comparison Differences – part 1 of 2
Subject
PQI
NVMe
PCIe device register for capabilities
Register level: not defined yet.
Most capability features are reported through
Administrator command
Controller Capabilities register
PCIe register for version
No plan to be in register format
Version identification is returned through
Administrator command (future expandability)
Version register
PCIe register to configure the controller
Configuration is done using Administrator
command (future expandability)
Controller Configuration register
PCIe register for Administrator queue attribute
Admin queue element size and depth register:
ADMIN IQ DEPTH
ADMIN OQ DEPTH
ADMIN IQ ELEMENT SIZE
ADMIN OQ ELEMENT SIZE
Admin Queue Attributes register:
Admin Completion Queue Size (depth)
Admin Submission Queue Size (depth)
PCIe register for circular queue indexes
No. All PI and CI are soft registers.
Submission Queue 0 Tail Doorbell (Admin)
Completion Queue 0 Head Doorbell (Admin)
Submission Queue 1 Tail Doorbell
Completion Queue 1 Head Doorbell …
Submission Queue y Tail Doorbell
Completion Queue y Head Doorbell
PCIe register for triggering the creation of administrator queue
Admin Process Command register
Any update to Admin Queue Attributes,
Admin Submission Queue Base Address,
Admin Completion Queue Base Address registers.
PCIe register for supporting legacy INT-x
Interrupt register
Interrupt Clear register No
PCIe registers for passing parameters during administrator queue creation
Admin IQ CI PQI host Memory Address reg
Admin OQ PI PQI host Memory Address reg
Admin IQ PI register
Admin OQ CI register

6. High Level Comparison Differences – part 2 of 2
Subject
PQI
NVMe
Queue element size
Configurable
SQ=64 bytes, CQ=16 bytes
Queue mapping
Any IQ to any OQ mapping during I/O operation
SQ to CQ (1:n) mapping set during initialization.
Fixed SQ to CQ mapping during I/O.
Data (DMA) descriptor
SGL
PRP
Asynchronous/unsolicited message from device to host
Yes. Host could receive any amount of asynchronous/unsolicited messages as long as entries available in QO)
No. Required one pre-submitted SQ entry for every asynchronous/unsolicited message in CQ
Interrupt support
MSI-X (required), legacy INT-x optional
MSI and MSI-X, no legacy INT-x support
Queue element chaining
Yes
No, from queuing perspective
Yes, from command set perspective in the form of fused-command
Coalescing good completions in one queue element No
Support for separate memory for completion status (e.g. SENSE DATA)
Non-contiguous memory for circular queue buffer
Yes. Using PRP list based table in host that provides pointers to actual physical memory chunk
Queue index PCIe location
PI/CI for IQ/OQ are soft registers negotiated during initialization:
IQ PI in PQI device space
OQ CI in PQI device space
IQ CI in PQI host space
OQ PI in PQI host space
SQ Tail in NMVe device register (fixed)
CQ Head in NMVe device register (fixed)
No SQ Head register accessible to host
No CQ Tail register accessible to host
See detail discussion in the next few slides.

7. PQI Circular Queue Basic Mechanics Queue Full and Queue Empty
Circular queues are arrays in PQI host memory with two associated indices, a Producer Index (PI) and a Consumer Index (CI).
The queue is empty when the PI and CI are equal.
The queue is full when the PI is exactly one position behind the CI.

8. PQI Circular Queue Basic Mechanics Host and PQI device handshaking
Handshaking via posted writes only
Motivation: PCIe posted write is more efficient than reading across the other side of the PCIe bus
PI resides in consumer space and is cached (locally copied) by producer
CI resides in producer space and is cached (locally copied) by consumer
Both PQI host and PQI device are not required to read across the PCIe bus to get their relevant PI and CI
Both PQI host and PQI device have full knowledge of the exactly status of a particular queue
Motivation: basic requirement for a general purpose queuing interface for flexibility

9. PQI device IQ and OQ Example (HBA Initiator)
PQI device Inbound Queue
PQI host prepares request IU using the IQ element;
PQI host increments IQ PI in PQI device;
PQI device consumes request IU;
Controller increments IQ CI in PQI host; IQ element is available for re-used.
Through IQ PI and IQ CI, PQI host knows exactly how many free IQ queue elements available at any time.
PQI device Outbound Queue
PQI device prepares and DMAs response IU to the OQ element;
PQI device increments OQ PI in PQI host;
PQI device triggers interrupt to host;
PQI host consumes response IU;
PQI host increments OQ CI in PQI device OQ element is available for re-used.

10. NVMe Circular Queue Basic Mechanics (contiguous queue memory shown)
Some similarities as in PQI with different name for the queue and different name for the index
NMVe Submission Queue (SQ) is equivalent to PQI IQ
NMVe Completion Queue (CQ) is equivalent to PQI OQ
NVMe SQ Tail is equivalent to PQI IQ PI
NMVe CQ Head is equivalent to PQI OQ CI
NVMe SQ Head is not directly accessible by host (see below)
NVMe CQ Tail is not directly accessible by host (see below)
NVMe does not have the direct equivalent of PQI IQ CI
NVMe indicates the consumption of SQ entry indirectly through the completion path on the CQ
NMVe provides the current (at the time completion was posted to CQ) SQ head pointer in the completion status in CQ
No mechanism to provide direct and exact SQ head pointer to host at any given time
NVMe does not have the direct equivalent of PQI OQ PI
NVMe indicates the producing of a new Completion queue entry indirectly through the host reading of a “Phase Tag” flag while processing one completion
No mechanism to provide direct and exact CQ tail pointer to host at any given time
Host needs to traverse the CQ one entry at a time
NVMe motivation: eliminate the host requirement to read SQ Head and CQ tail