1. NVMe

2. Introduction
NVMe defines an optimized queuing interface, command set, and feature set for PCIe SSDs
PCIe (Peripheral Component Interconnect Express)
Serial expansion bus standard for connecting a computer to one or more peripheral devices.
***PCIe removes controller latency and NVMe reduces software latency.
Benefits of PCIe as an SSD Interface
PCIe is High Performance
Low latency (no HBA overhead or protocol translation)
Scalable port width (x1 to x16)
Scalable link speed (2.5 GTps, 5 GTps, 8 GTps) (Gigatransfers per second)
PCIe is Low Cost
Direct attach to CPU subsystem eliminates HBA cost
PCIe Provides Effective Power Management
A host bus adapter (HBA) is a circuit board and/or integrated circuit adapter that provides input/output (I/O) processing and physical connectivity between a host system, or server, and a storage and/or network device.
With PCIe, data is transferred over two signal pairs: two wires for transmitting and two wires for receiving. Each set of signal pairs is called a "lane," and each lane is capable of sending and receiving eight-bit data packets simultaneously between two points.

3. NVMe It has: Standard drivers Consistent feature set
Eliminates need for OEMs to qualify a driver for each SSD
Enables broad adoption across a wide range of industry standard and
Consistent feature set
All SSDs implement required features
Optional features are implemented in a consistent manner
Industry ecosystem
Development tools
Original equipment manufacturer (OEM) is a company that makes a part or subsystem that is used in another company's end product.

4. NVM Express Specification
Specification cooperatively developed by more than 80 companies within the NVMe Work Group
NVMe Work Group is directed by a multi-member Promoter Group of Companies consisting of Cisco, Dell, EMC, IDT, Intel, Micron, NetApp, Oracle, SandForce and STEC

5. NVMe Structure

6. NVMe Architecture NVM Subsystem
one or more controllers, one or more namespaces, one or more PCI Express ports, a non-volatile memory storage medium, and an interface between the controller(s) and non-volatile memory storage medium

7. NVMe Queues
NVMe uses circular queues to pass messages (e.g., commands and command completion notifications.)
Typically queues are located in host memory
Queues may consist of a contiguous block of physical memory or optionally a non-contiguous set of physical memory pages.
A Queue consists of set of fixed sized elements
Tail
Points to next free element
If an element is added to the element pointed to by the tail, the tail is incremented to point to next free element taking wrapping into consideration.
Head
Points to next entry to be pulled off, if queue is not empty
If an element is removed from the element pointed to by the head, the head is incremented to point to the next element taking wrapping into consideration

8. Queue Size (Usable) Queue Empty Queue Full
Number of entries in the queue - 1
Minimum size is 2, Maximum is ~ 64K for I/O Queues and 4K for Admin Queue
Queue Empty
Head == Tail
Queue Full
Head == Tail + 1 mod # Of Queue Entries.

9. Types of Queues Admin Queue for Admin Command Set
One per NVMe controller with up to 4K elements per queue
Used to configure IO Queues and controller/feature management
I/O Queues for IO Command Sets (e.g., NVM command set)
Up to 64K queues per NVMe controller with up to 64K elements per queue
Used to submit/complete IO commands

10. Submission Queues (SQ)
Each type has:
Submission Queues (SQ)
Queues messages from host to controller
Used to submit commands
Identified by SQID
Completion Queues (CQ)
Queues messages from controller to host
Used to post command completions
Identified by CQID
May have an independent MSI-X(Message Signalled Interrupts) interrupt per completion queue

11. NVMe Command Execution
Command Submission:
Host writes command to submission queue.
Host writes updated submission queue tail pointer to doorbell.
Command Processing:
Controller fetches command.
Controller processes command.
Command Completion:
Controller writes completion to Completion queue.
Controller generates interrupt.
Host processes completion.
Host writes updated Completion Queue head pointer to doorbell.
TLP : Transaction layer packets.
Doorbell is often used to describe a mechanism whereby a software system can signal or notify a computer hardware device that there is some work to be done.
Typically, the software system will place data in some well known and mutually agreed upon memory location(s), and "ring the doorbell" by writing to a different memory location. This different memory location is often called the doorbell region.

12. Scalable Queuing Interface
Per core: One or more submission queues, one completion queue, and one MSI-X interrupt.
High performance and low latency command issue
No locking between cores
Up to ~2^32 outstanding commands
Support for up to ~ 64K I/O submission and completion queues
Each queue supports up to ~ 64K outstanding commands

13. Simple Command Set Admin Commands NVM I/O Commands
Create I/O Submission Queue
Delete I/O Submission Queue
Create I/O Completion Queue
Delete I/O Completion Queue
Get Log Page
Identify
Abort
Set Features
Get Features
… etc
NVM I/O Commands
Read
Write
Flush
...etc
More details about the commands and their parameters:

14. RocksDB

24. Thank You!!!