man.bsd.lv manual page server

DESCRIPTION

The driver supports the following features:

• controller and namespace configuration and management using nvmectl(8)
• highly parallel I/O using per-CPU I/O queues
• PCI MSI/MSI-X attachment, and INTx for legacy systems

On systems supporting MSI/MSI-X, the nvme driver uses per-CPU IO queue pairs for lockless and highly parallelized I/O. Interrupt handlers are scheduled on distinct CPUs. The driver allocates as many interrupt vectors as available, up to number of CPUs + 1. MSI supports up to 32 interrupt vectors within the system, MSI-X can have up to 2k. Each I/O queue pair has a separate command circular buffer. The nvme specification allows up to 64k commands per queue, the driver currently allocates 1024 entries per queue, or controller maximum, whatever is smaller. Command submissions are done always on the current CPU, the command completion interrupt is handled on the CPU corresponding to the I/O queue ID - first I/O queue on CPU0, second I/O queue on CPU1, etc. Admin queue command completion is handled by CPU0 by default. To keep lock contention to minimum, it is recommended to keep this assignment, even though it is possible to reassign the interrupt handlers differently using intrctl(8).

On systems without MSI, the driver uses a single HW interrupt handler for both admin and standard I/O commands. Command submissions are done on the current CPU, the command completion interrupt is handled on CPU0 by default. This leads to some lock contention, especially on command ccbs.

The driver offloads command completion processing to soft interrupt, in order to increase the total system I/O capacity and throughput.