1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 .. include:: <isonum.txt> 2 .. include:: <isonum.txt>
3 3
4 =========================================== 4 ===========================================
5 The PCI Express Port Bus Driver Guide HOWTO 5 The PCI Express Port Bus Driver Guide HOWTO
6 =========================================== 6 ===========================================
7 7
8 :Author: Tom L Nguyen tom.l.nguyen@intel.com 1 8 :Author: Tom L Nguyen tom.l.nguyen@intel.com 11/03/2004
9 :Copyright: |copy| 2004 Intel Corporation 9 :Copyright: |copy| 2004 Intel Corporation
10 10
11 About this guide 11 About this guide
12 ================ 12 ================
13 13
14 This guide describes the basics of the PCI Exp 14 This guide describes the basics of the PCI Express Port Bus driver
15 and provides information on how to enable the 15 and provides information on how to enable the service drivers to
16 register/unregister with the PCI Express Port 16 register/unregister with the PCI Express Port Bus Driver.
17 17
18 18
19 What is the PCI Express Port Bus Driver 19 What is the PCI Express Port Bus Driver
20 ======================================= 20 =======================================
21 21
22 A PCI Express Port is a logical PCI-PCI Bridge 22 A PCI Express Port is a logical PCI-PCI Bridge structure. There
23 are two types of PCI Express Port: the Root Po 23 are two types of PCI Express Port: the Root Port and the Switch
24 Port. The Root Port originates a PCI Express l 24 Port. The Root Port originates a PCI Express link from a PCI Express
25 Root Complex and the Switch Port connects PCI 25 Root Complex and the Switch Port connects PCI Express links to
26 internal logical PCI buses. The Switch Port, w 26 internal logical PCI buses. The Switch Port, which has its secondary
27 bus representing the switch's internal routing 27 bus representing the switch's internal routing logic, is called the
28 switch's Upstream Port. The switch's Downstrea 28 switch's Upstream Port. The switch's Downstream Port is bridging from
29 switch's internal routing bus to a bus represe 29 switch's internal routing bus to a bus representing the downstream
30 PCI Express link from the PCI Express Switch. 30 PCI Express link from the PCI Express Switch.
31 31
32 A PCI Express Port can provide up to four dist 32 A PCI Express Port can provide up to four distinct functions,
33 referred to in this document as services, depe 33 referred to in this document as services, depending on its port type.
34 PCI Express Port's services include native hot 34 PCI Express Port's services include native hotplug support (HP),
35 power management event support (PME), advanced 35 power management event support (PME), advanced error reporting
36 support (AER), and virtual channel support (VC 36 support (AER), and virtual channel support (VC). These services may
37 be handled by a single complex driver or be in 37 be handled by a single complex driver or be individually distributed
38 and handled by corresponding service drivers. 38 and handled by corresponding service drivers.
39 39
40 Why use the PCI Express Port Bus Driver? 40 Why use the PCI Express Port Bus Driver?
41 ======================================== 41 ========================================
42 42
43 In existing Linux kernels, the Linux Device Dr 43 In existing Linux kernels, the Linux Device Driver Model allows a
44 physical device to be handled by only a single 44 physical device to be handled by only a single driver. The PCI
45 Express Port is a PCI-PCI Bridge device with m 45 Express Port is a PCI-PCI Bridge device with multiple distinct
46 services. To maintain a clean and simple solut 46 services. To maintain a clean and simple solution each service
47 may have its own software service driver. In t 47 may have its own software service driver. In this case several
48 service drivers will compete for a single PCI- 48 service drivers will compete for a single PCI-PCI Bridge device.
49 For example, if the PCI Express Root Port nati 49 For example, if the PCI Express Root Port native hotplug service
50 driver is loaded first, it claims a PCI-PCI Br 50 driver is loaded first, it claims a PCI-PCI Bridge Root Port. The
51 kernel therefore does not load other service d 51 kernel therefore does not load other service drivers for that Root
52 Port. In other words, it is impossible to have 52 Port. In other words, it is impossible to have multiple service
53 drivers load and run on a PCI-PCI Bridge devic 53 drivers load and run on a PCI-PCI Bridge device simultaneously
54 using the current driver model. 54 using the current driver model.
55 55
56 To enable multiple service drivers running sim 56 To enable multiple service drivers running simultaneously requires
57 having a PCI Express Port Bus driver, which ma 57 having a PCI Express Port Bus driver, which manages all populated
58 PCI Express Ports and distributes all provided 58 PCI Express Ports and distributes all provided service requests
59 to the corresponding service drivers as requir 59 to the corresponding service drivers as required. Some key
60 advantages of using the PCI Express Port Bus d 60 advantages of using the PCI Express Port Bus driver are listed below:
61 61
62 - Allow multiple service drivers to run simu 62 - Allow multiple service drivers to run simultaneously on
63 a PCI-PCI Bridge Port device. 63 a PCI-PCI Bridge Port device.
64 64
65 - Allow service drivers implemented in an in 65 - Allow service drivers implemented in an independent
66 staged approach. 66 staged approach.
67 67
68 - Allow one service driver to run on multipl 68 - Allow one service driver to run on multiple PCI-PCI Bridge
69 Port devices. 69 Port devices.
70 70
71 - Manage and distribute resources of a PCI-P 71 - Manage and distribute resources of a PCI-PCI Bridge Port
72 device to requested service drivers. 72 device to requested service drivers.
73 73
74 Configuring the PCI Express Port Bus Driver vs 74 Configuring the PCI Express Port Bus Driver vs. Service Drivers
75 ============================================== 75 ===============================================================
76 76
77 Including the PCI Express Port Bus Driver Supp 77 Including the PCI Express Port Bus Driver Support into the Kernel
78 ---------------------------------------------- 78 -----------------------------------------------------------------
79 79
80 Including the PCI Express Port Bus driver depe 80 Including the PCI Express Port Bus driver depends on whether the PCI
81 Express support is included in the kernel conf 81 Express support is included in the kernel config. The kernel will
82 automatically include the PCI Express Port Bus 82 automatically include the PCI Express Port Bus driver as a kernel
83 driver when the PCI Express support is enabled 83 driver when the PCI Express support is enabled in the kernel.
84 84
85 Enabling Service Driver Support 85 Enabling Service Driver Support
86 ------------------------------- 86 -------------------------------
87 87
88 PCI device drivers are implemented based on Li 88 PCI device drivers are implemented based on Linux Device Driver Model.
89 All service drivers are PCI device drivers. As 89 All service drivers are PCI device drivers. As discussed above, it is
90 impossible to load any service driver once the 90 impossible to load any service driver once the kernel has loaded the
91 PCI Express Port Bus Driver. To meet the PCI E 91 PCI Express Port Bus Driver. To meet the PCI Express Port Bus Driver
92 Model requires some minimal changes on existin 92 Model requires some minimal changes on existing service drivers that
93 imposes no impact on the functionality of exis 93 imposes no impact on the functionality of existing service drivers.
94 94
95 A service driver is required to use the two AP 95 A service driver is required to use the two APIs shown below to
96 register its service with the PCI Express Port 96 register its service with the PCI Express Port Bus driver (see
97 section 5.2.1 & 5.2.2). It is important that a 97 section 5.2.1 & 5.2.2). It is important that a service driver
98 initializes the pcie_port_service_driver data 98 initializes the pcie_port_service_driver data structure, included in
99 header file /include/linux/pcieport_if.h, befo 99 header file /include/linux/pcieport_if.h, before calling these APIs.
100 Failure to do so will result an identity misma 100 Failure to do so will result an identity mismatch, which prevents
101 the PCI Express Port Bus driver from loading a 101 the PCI Express Port Bus driver from loading a service driver.
102 102
103 pcie_port_service_register 103 pcie_port_service_register
104 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 104 ~~~~~~~~~~~~~~~~~~~~~~~~~~
105 :: 105 ::
106 106
107 int pcie_port_service_register(struct pcie_p 107 int pcie_port_service_register(struct pcie_port_service_driver *new)
108 108
109 This API replaces the Linux Driver Model's pci 109 This API replaces the Linux Driver Model's pci_register_driver API. A
110 service driver should always calls pcie_port_s 110 service driver should always calls pcie_port_service_register at
111 module init. Note that after service driver be 111 module init. Note that after service driver being loaded, calls
112 such as pci_enable_device(dev) and pci_set_mas 112 such as pci_enable_device(dev) and pci_set_master(dev) are no longer
113 necessary since these calls are executed by th 113 necessary since these calls are executed by the PCI Port Bus driver.
114 114
115 pcie_port_service_unregister 115 pcie_port_service_unregister
116 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 116 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
117 :: 117 ::
118 118
119 void pcie_port_service_unregister(struct pci 119 void pcie_port_service_unregister(struct pcie_port_service_driver *new)
120 120
121 pcie_port_service_unregister replaces the Linu 121 pcie_port_service_unregister replaces the Linux Driver Model's
122 pci_unregister_driver. It's always called by s 122 pci_unregister_driver. It's always called by service driver when a
123 module exits. 123 module exits.
124 124
125 Sample Code 125 Sample Code
126 ~~~~~~~~~~~ 126 ~~~~~~~~~~~
127 127
128 Below is sample service driver code to initial 128 Below is sample service driver code to initialize the port service
129 driver data structure. 129 driver data structure.
130 :: 130 ::
131 131
132 static struct pcie_port_service_id service_i 132 static struct pcie_port_service_id service_id[] = { {
133 .vendor = PCI_ANY_ID, 133 .vendor = PCI_ANY_ID,
134 .device = PCI_ANY_ID, 134 .device = PCI_ANY_ID,
135 .port_type = PCIE_RC_PORT, 135 .port_type = PCIE_RC_PORT,
136 .service_type = PCIE_PORT_SERVICE_AER, 136 .service_type = PCIE_PORT_SERVICE_AER,
137 }, { /* end: all zeroes */ } 137 }, { /* end: all zeroes */ }
138 }; 138 };
139 139
140 static struct pcie_port_service_driver root_ 140 static struct pcie_port_service_driver root_aerdrv = {
141 .name = (char *)device_name, 141 .name = (char *)device_name,
142 .id_table = service_id, !! 142 .id_table = &service_id[0],
143 143
144 .probe = aerdrv_load, 144 .probe = aerdrv_load,
145 .remove = aerdrv_unload, 145 .remove = aerdrv_unload,
146 146
147 .suspend = aerdrv_suspend, 147 .suspend = aerdrv_suspend,
148 .resume = aerdrv_resume, 148 .resume = aerdrv_resume,
149 }; 149 };
150 150
151 Below is a sample code for registering/unregis 151 Below is a sample code for registering/unregistering a service
152 driver. 152 driver.
153 :: 153 ::
154 154
155 static int __init aerdrv_service_init(void) 155 static int __init aerdrv_service_init(void)
156 { 156 {
157 int retval = 0; 157 int retval = 0;
158 158
159 retval = pcie_port_service_register(&root_ 159 retval = pcie_port_service_register(&root_aerdrv);
160 if (!retval) { 160 if (!retval) {
161 /* 161 /*
162 * FIX ME 162 * FIX ME
163 */ 163 */
164 } 164 }
165 return retval; 165 return retval;
166 } 166 }
167 167
168 static void __exit aerdrv_service_exit(void) 168 static void __exit aerdrv_service_exit(void)
169 { 169 {
170 pcie_port_service_unregister(&root_aerdrv) 170 pcie_port_service_unregister(&root_aerdrv);
171 } 171 }
172 172
173 module_init(aerdrv_service_init); 173 module_init(aerdrv_service_init);
174 module_exit(aerdrv_service_exit); 174 module_exit(aerdrv_service_exit);
175 175
176 Possible Resource Conflicts 176 Possible Resource Conflicts
177 =========================== 177 ===========================
178 178
179 Since all service drivers of a PCI-PCI Bridge 179 Since all service drivers of a PCI-PCI Bridge Port device are
180 allowed to run simultaneously, below lists a f 180 allowed to run simultaneously, below lists a few of possible resource
181 conflicts with proposed solutions. 181 conflicts with proposed solutions.
182 182
183 MSI and MSI-X Vector Resource 183 MSI and MSI-X Vector Resource
184 ----------------------------- 184 -----------------------------
185 185
186 Once MSI or MSI-X interrupts are enabled on a 186 Once MSI or MSI-X interrupts are enabled on a device, it stays in this
187 mode until they are disabled again. Since ser 187 mode until they are disabled again. Since service drivers of the same
188 PCI-PCI Bridge port share the same physical de 188 PCI-PCI Bridge port share the same physical device, if an individual
189 service driver enables or disables MSI/MSI-X m 189 service driver enables or disables MSI/MSI-X mode it may result
190 unpredictable behavior. 190 unpredictable behavior.
191 191
192 To avoid this situation all service drivers ar 192 To avoid this situation all service drivers are not permitted to
193 switch interrupt mode on its device. The PCI E 193 switch interrupt mode on its device. The PCI Express Port Bus driver
194 is responsible for determining the interrupt m 194 is responsible for determining the interrupt mode and this should be
195 transparent to service drivers. Service driver 195 transparent to service drivers. Service drivers need to know only
196 the vector IRQ assigned to the field irq of st 196 the vector IRQ assigned to the field irq of struct pcie_device, which
197 is passed in when the PCI Express Port Bus dri 197 is passed in when the PCI Express Port Bus driver probes each service
198 driver. Service drivers should use (struct pci 198 driver. Service drivers should use (struct pcie_device*)dev->irq to
199 call request_irq/free_irq. In addition, the in 199 call request_irq/free_irq. In addition, the interrupt mode is stored
200 in the field interrupt_mode of struct pcie_dev 200 in the field interrupt_mode of struct pcie_device.
201 201
202 PCI Memory/IO Mapped Regions 202 PCI Memory/IO Mapped Regions
203 ---------------------------- 203 ----------------------------
204 204
205 Service drivers for PCI Express Power Manageme 205 Service drivers for PCI Express Power Management (PME), Advanced
206 Error Reporting (AER), Hot-Plug (HP) and Virtu 206 Error Reporting (AER), Hot-Plug (HP) and Virtual Channel (VC) access
207 PCI configuration space on the PCI Express por 207 PCI configuration space on the PCI Express port. In all cases the
208 registers accessed are independent of each oth 208 registers accessed are independent of each other. This patch assumes
209 that all service drivers will be well behaved 209 that all service drivers will be well behaved and not overwrite
210 other service driver's configuration settings. 210 other service driver's configuration settings.
211 211
212 PCI Config Registers 212 PCI Config Registers
213 -------------------- 213 --------------------
214 214
215 Each service driver runs its PCI config operat 215 Each service driver runs its PCI config operations on its own
216 capability structure except the PCI Express ca 216 capability structure except the PCI Express capability structure,
217 that is shared between many drivers including 217 that is shared between many drivers including the service drivers.
218 RMW Capability accessors (pcie_capability_clea 218 RMW Capability accessors (pcie_capability_clear_and_set_word(),
219 pcie_capability_set_word(), and pcie_capabilit 219 pcie_capability_set_word(), and pcie_capability_clear_word()) protect
220 a selected set of PCI Express Capability Regis 220 a selected set of PCI Express Capability Registers (Link Control
221 Register and Root Control Register). Any chang 221 Register and Root Control Register). Any change to those registers
222 should be performed using RMW accessors to avo 222 should be performed using RMW accessors to avoid problems due to
223 concurrent updates. For the up-to-date list of 223 concurrent updates. For the up-to-date list of protected registers,
224 see pcie_capability_clear_and_set_word(). 224 see pcie_capability_clear_and_set_word().
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.