PQI vs. NVMe® Queuing Comparison T10 CAP discussion for PQI Ie-Wei Njoo, PMC-Sierra (IeWei_Njoo@pmc-sierra.com) 5 May,2011

Reference and acknowledgement All references to of NVMe® entities are based on the publicly available NMVe specification: http://nvmexpress.org

PQI device Inbound – the direction from the PQI host to the PQI device Definitions PQI device Inbound – the direction from the PQI host to the PQI device IQ – Inbound Queue PQI device Outbound – the direction from the PQI device to the PQI host OQ – Outbound Queue NVMe Submission – the direction from the NVMe host to the NVMe device SQ – Submission Queue NVMe Completion – the direction from the NVMe device to the NVMe host CQ – Completion Queue

High Level Comparison Similarity Subject PQI NVMe Circular queue with memory buffer in host Yes Queuing priority Queue mapping to interrupt (MSI-x) Interrupt Coalescing Administrator function PCIe register set for interrupt mask function Interrupt Mask Set Interrupt Mask Clear Interrupt Mask register Interrupt Mask Clear register PCIe register for controller status PQI device Status register Controller Status register PCIe register set for circular queue buffer in host memory Admin IQ PQI host Memory Address register Admin OQ PQI host Memory Address Admin Submission Queue Base Address Admin Completion Queue Base Address

High Level Comparison Differences – part 1 of 2 Subject PQI NVMe PCIe device register for capabilities Register level: not defined yet. Most capability features are reported through Administrator command Controller Capabilities register PCIe register for version No plan to be in register format Version identification is returned through Administrator command (future expandability) Version register PCIe register to configure the controller Configuration is done using Administrator command (future expandability) Controller Configuration register PCIe register for Administrator queue attribute Admin queue element size and depth register: ADMIN IQ DEPTH ADMIN OQ DEPTH ADMIN IQ ELEMENT SIZE ADMIN OQ ELEMENT SIZE Admin Queue Attributes register: Admin Completion Queue Size (depth) Admin Submission Queue Size (depth) PCIe register for circular queue indexes No. All PI and CI are soft registers. Submission Queue 0 Tail Doorbell (Admin) Completion Queue 0 Head Doorbell (Admin) Submission Queue 1 Tail Doorbell Completion Queue 1 Head Doorbell … Submission Queue y Tail Doorbell Completion Queue y Head Doorbell PCIe register for triggering the creation of administrator queue Admin Process Command register Any update to Admin Queue Attributes, Admin Submission Queue Base Address, Admin Completion Queue Base Address registers. PCIe register for supporting legacy INT-x Interrupt register Interrupt Clear register No PCIe registers for passing parameters during administrator queue creation Admin IQ CI PQI host Memory Address reg Admin OQ PI PQI host Memory Address reg Admin IQ PI register Admin OQ CI register

High Level Comparison Differences – part 2 of 2 Subject PQI NVMe Queue element size Configurable SQ=64 bytes, CQ=16 bytes Queue mapping Any IQ to any OQ mapping during I/O operation SQ to CQ (1:n) mapping set during initialization. Fixed SQ to CQ mapping during I/O. Data (DMA) descriptor SGL PRP Asynchronous/unsolicited message from device to host Yes. Host could receive any amount of asynchronous/unsolicited messages as long as entries available in QO) No. Required one pre-submitted SQ entry for every asynchronous/unsolicited message in CQ Interrupt support MSI-X (required), legacy INT-x optional MSI and MSI-X, no legacy INT-x support Queue element chaining Yes No, from queuing perspective Yes, from command set perspective in the form of fused-command Coalescing good completions in one queue element No Support for separate memory for completion status (e.g. SENSE DATA) Non-contiguous memory for circular queue buffer Yes. Using PRP list based table in host that provides pointers to actual physical memory chunk Queue index PCIe location PI/CI for IQ/OQ are soft registers negotiated during initialization: IQ PI in PQI device space OQ CI in PQI device space IQ CI in PQI host space OQ PI in PQI host space SQ Tail in NMVe device register (fixed) CQ Head in NMVe device register (fixed) No SQ Head register accessible to host No CQ Tail register accessible to host See detail discussion in the next few slides.

PQI Circular Queue Basic Mechanics Queue Full and Queue Empty Circular queues are arrays in PQI host memory with two associated indices, a Producer Index (PI) and a Consumer Index (CI). The queue is empty when the PI and CI are equal. The queue is full when the PI is exactly one position behind the CI.

PQI Circular Queue Basic Mechanics Host and PQI device handshaking Handshaking via posted writes only Motivation: PCIe posted write is more efficient than reading across the other side of the PCIe bus PI resides in consumer space and is cached (locally copied) by producer CI resides in producer space and is cached (locally copied) by consumer Both PQI host and PQI device are not required to read across the PCIe bus to get their relevant PI and CI Both PQI host and PQI device have full knowledge of the exactly status of a particular queue Motivation: basic requirement for a general purpose queuing interface for flexibility

PQI device IQ and OQ Example (HBA Initiator) PQI device Inbound Queue PQI host prepares request IU using the IQ element; PQI host increments IQ PI in PQI device; PQI device consumes request IU; Controller increments IQ CI in PQI host; IQ element is available for re-used. Through IQ PI and IQ CI, PQI host knows exactly how many free IQ queue elements available at any time. PQI device Outbound Queue PQI device prepares and DMAs response IU to the OQ element; PQI device increments OQ PI in PQI host; PQI device triggers interrupt to host; PQI host consumes response IU; PQI host increments OQ CI in PQI device. OQ element is available for re-used.

NVMe Circular Queue Basic Mechanics (contiguous queue memory shown) Some similarities as in PQI with different name for the queue and different name for the index NMVe Submission Queue (SQ) is equivalent to PQI IQ NMVe Completion Queue (CQ) is equivalent to PQI OQ NVMe SQ Tail is equivalent to PQI IQ PI NMVe CQ Head is equivalent to PQI OQ CI NVMe SQ Head is not directly accessible by host (see below) NVMe CQ Tail is not directly accessible by host (see below) NVMe does not have the direct equivalent of PQI IQ CI NVMe indicates the consumption of SQ entry indirectly through the completion path on the CQ NMVe provides the current (at the time completion was posted to CQ) SQ head pointer in the completion status in CQ No mechanism to provide direct and exact SQ head pointer to host at any given time NVMe does not have the direct equivalent of PQI OQ PI NVMe indicates the producing of a new Completion queue entry indirectly through the host reading of a “Phase Tag” flag while processing one completion No mechanism to provide direct and exact CQ tail pointer to host at any given time Host needs to traverse the CQ one entry at a time NVMe motivation: eliminate the host requirement to read SQ Head and CQ tail