1 .. SPDX-License-Identifier: GPL-2.0+
2
3 ==============================================================
4 Linux kernel driver for Compute Engine Virtual Ethernet (gve):
5 ==============================================================
6
7 Supported Hardware
8 ===================
9 The GVE driver binds to a single PCI device id used by the virtual
10 Ethernet device found in some Compute Engine VMs.
11
12 +--------------+----------+---------+
13 |Field | Value | Comments|
14 +==============+==========+=========+
15 |Vendor ID | `0x1AE0` | Google |
16 +--------------+----------+---------+
17 |Device ID | `0x0042` | |
18 +--------------+----------+---------+
19 |Sub-vendor ID | `0x1AE0` | Google |
20 +--------------+----------+---------+
21 |Sub-device ID | `0x0058` | |
22 +--------------+----------+---------+
23 |Revision ID | `0x0` | |
24 +--------------+----------+---------+
25 |Device Class | `0x200` | Ethernet|
26 +--------------+----------+---------+
27
28 PCI Bars
29 ========
30 The gVNIC PCI device exposes three 32-bit memory BARS:
31 - Bar0 - Device configuration and status registers.
32 - Bar1 - MSI-X vector table
33 - Bar2 - IRQ, RX and TX doorbells
34
35 Device Interactions
36 ===================
37 The driver interacts with the device in the following ways:
38 - Registers
39 - A block of MMIO registers
40 - See gve_register.h for more detail
41 - Admin Queue
42 - See description below
43 - Reset
44 - At any time the device can be reset
45 - Interrupts
46 - See supported interrupts below
47 - Transmit and Receive Queues
48 - See description below
49
50 Descriptor Formats
51 ------------------
52 GVE supports two descriptor formats: GQI and DQO. These two formats have
53 entirely different descriptors, which will be described below.
54
55 Addressing Mode
56 ------------------
57 GVE supports two addressing modes: QPL and RDA.
58 QPL ("queue-page-list") mode communicates data through a set of
59 pre-registered pages.
60
61 For RDA ("raw DMA addressing") mode, the set of pages is dynamic.
62 Therefore, the packet buffers can be anywhere in guest memory.
63
64 Registers
65 ---------
66 All registers are MMIO.
67
68 The registers are used for initializing and configuring the device as well as
69 querying device status in response to management interrupts.
70
71 Endianness
72 ----------
73 - Admin Queue messages and registers are all Big Endian.
74 - GQI descriptors and datapath registers are Big Endian.
75 - DQO descriptors and datapath registers are Little Endian.
76
77 Admin Queue (AQ)
78 ----------------
79 The Admin Queue is a PAGE_SIZE memory block, treated as an array of AQ
80 commands, used by the driver to issue commands to the device and set up
81 resources.The driver and the device maintain a count of how many commands
82 have been submitted and executed. To issue AQ commands, the driver must do
83 the following (with proper locking):
84
85 1) Copy new commands into next available slots in the AQ array
86 2) Increment its counter by he number of new commands
87 3) Write the counter into the GVE_ADMIN_QUEUE_DOORBELL register
88 4) Poll the ADMIN_QUEUE_EVENT_COUNTER register until it equals
89 the value written to the doorbell, or until a timeout.
90
91 The device will update the status field in each AQ command reported as
92 executed through the ADMIN_QUEUE_EVENT_COUNTER register.
93
94 Device Resets
95 -------------
96 A device reset is triggered by writing 0x0 to the AQ PFN register.
97 This causes the device to release all resources allocated by the
98 driver, including the AQ itself.
99
100 Interrupts
101 ----------
102 The following interrupts are supported by the driver:
103
104 Management Interrupt
105 ~~~~~~~~~~~~~~~~~~~~
106 The management interrupt is used by the device to tell the driver to
107 look at the GVE_DEVICE_STATUS register.
108
109 The handler for the management irq simply queues the service task in
110 the workqueue to check the register and acks the irq.
111
112 Notification Block Interrupts
113 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
114 The notification block interrupts are used to tell the driver to poll
115 the queues associated with that interrupt.
116
117 The handler for these irqs schedule the napi for that block to run
118 and poll the queues.
119
120 GQI Traffic Queues
121 ------------------
122 GQI queues are composed of a descriptor ring and a buffer and are assigned to a
123 notification block.
124
125 The descriptor rings are power-of-two-sized ring buffers consisting of
126 fixed-size descriptors. They advance their head pointer using a __be32
127 doorbell located in Bar2. The tail pointers are advanced by consuming
128 descriptors in-order and updating a __be32 counter. Both the doorbell
129 and the counter overflow to zero.
130
131 Each queue's buffers must be registered in advance with the device as a
132 queue page list, and packet data can only be put in those pages.
133
134 Transmit
135 ~~~~~~~~
136 gve maps the buffers for transmit rings into a FIFO and copies the packets
137 into the FIFO before sending them to the NIC.
138
139 Receive
140 ~~~~~~~
141 The buffers for receive rings are put into a data ring that is the same
142 length as the descriptor ring and the head and tail pointers advance over
143 the rings together.
144
145 DQO Traffic Queues
146 ------------------
147 - Every TX and RX queue is assigned a notification block.
148
149 - TX and RX buffers queues, which send descriptors to the device, use MMIO
150 doorbells to notify the device of new descriptors.
151
152 - RX and TX completion queues, which receive descriptors from the device, use a
153 "generation bit" to know when a descriptor was populated by the device. The
154 driver initializes all bits with the "current generation". The device will
155 populate received descriptors with the "next generation" which is inverted
156 from the current generation. When the ring wraps, the current/next generation
157 are swapped.
158
159 - It's the driver's responsibility to ensure that the RX and TX completion
160 queues are not overrun. This can be accomplished by limiting the number of
161 descriptors posted to HW.
162
163 - TX packets have a 16 bit completion_tag and RX buffers have a 16 bit
164 buffer_id. These will be returned on the TX completion and RX queues
165 respectively to let the driver know which packet/buffer was completed.
166
167 Transmit
168 ~~~~~~~~
169 A packet's buffers are DMA mapped for the device to access before transmission.
170 After the packet was successfully transmitted, the buffers are unmapped.
171
172 Receive
173 ~~~~~~~
174 The driver posts fixed sized buffers to HW on the RX buffer queue. The packet
175 received on the associated RX queue may span multiple descriptors.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.