1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ====================================== 3 ======================================
4 HiSilicon PCIe Tune and Trace device 4 HiSilicon PCIe Tune and Trace device
5 ====================================== 5 ======================================
6 6
7 Introduction 7 Introduction
8 ============ 8 ============
9 9
10 HiSilicon PCIe tune and trace device (PTT) is 10 HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex
11 integrated Endpoint (RCiEP) device, providing 11 integrated Endpoint (RCiEP) device, providing the capability
12 to dynamically monitor and tune the PCIe link' 12 to dynamically monitor and tune the PCIe link's events (tune),
13 and trace the TLP headers (trace). The two fun 13 and trace the TLP headers (trace). The two functions are independent,
14 but is recommended to use them together to ana 14 but is recommended to use them together to analyze and enhance the
15 PCIe link's performance. 15 PCIe link's performance.
16 16
17 On Kunpeng 930 SoC, the PCIe Root Complex is c 17 On Kunpeng 930 SoC, the PCIe Root Complex is composed of several
18 PCIe cores. Each PCIe core includes several Ro 18 PCIe cores. Each PCIe core includes several Root Ports and a PTT
19 RCiEP, like below. The PTT device is capable o 19 RCiEP, like below. The PTT device is capable of tuning and
20 tracing the links of the PCIe core. 20 tracing the links of the PCIe core.
21 :: 21 ::
22 22
23 +--------------Core 0-------+ 23 +--------------Core 0-------+
24 | | [ PTT ] | 24 | | [ PTT ] |
25 | | [Root Port]---[Endpo 25 | | [Root Port]---[Endpoint]
26 | | [Root Port]---[Endpo 26 | | [Root Port]---[Endpoint]
27 | | [Root Port]---[Endpo 27 | | [Root Port]---[Endpoint]
28 Root Complex |------Core 1-------+ 28 Root Complex |------Core 1-------+
29 | | [ PTT ] | 29 | | [ PTT ] |
30 | | [Root Port]---[ Swit 30 | | [Root Port]---[ Switch ]---[Endpoint]
31 | | [Root Port]---[Endpo 31 | | [Root Port]---[Endpoint] `-[Endpoint]
32 | | [Root Port]---[Endpo 32 | | [Root Port]---[Endpoint]
33 +---------------------------+ 33 +---------------------------+
34 34
35 The PTT device driver registers one PMU device 35 The PTT device driver registers one PMU device for each PTT device.
36 The name of each PTT device is composed of 'hi 36 The name of each PTT device is composed of 'hisi_ptt' prefix with
37 the id of the SICL and the Core where it locat 37 the id of the SICL and the Core where it locates. The Kunpeng 930
38 SoC encapsulates multiple CPU dies (SCCL, Supe 38 SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and
39 IO dies (SICL, Super I/O Cluster), where there 39 IO dies (SICL, Super I/O Cluster), where there's one PCIe Root
40 Complex for each SICL. 40 Complex for each SICL.
41 :: 41 ::
42 42
43 /sys/bus/event_source/devices/hisi_ptt<sic !! 43 /sys/devices/hisi_ptt<sicl_id>_<core_id>
44 44
45 Tune 45 Tune
46 ==== 46 ====
47 47
48 PTT tune is designed for monitoring and adjust 48 PTT tune is designed for monitoring and adjusting PCIe link parameters (events).
49 Currently we support events in 2 classes. The 49 Currently we support events in 2 classes. The scope of the events
50 covers the PCIe core to which the PTT device b 50 covers the PCIe core to which the PTT device belongs.
51 51
52 Each event is presented as a file under $(PTT 52 Each event is presented as a file under $(PTT PMU dir)/tune, and
53 a simple open/read/write/close cycle will be u 53 a simple open/read/write/close cycle will be used to tune the event.
54 :: 54 ::
55 55
56 $ cd /sys/bus/event_source/devices/hisi_pt !! 56 $ cd /sys/devices/hisi_ptt<sicl_id>_<core_id>/tune
57 $ ls 57 $ ls
58 qos_tx_cpl qos_tx_np qos_tx_p 58 qos_tx_cpl qos_tx_np qos_tx_p
59 tx_path_rx_req_alloc_buf_level 59 tx_path_rx_req_alloc_buf_level
60 tx_path_tx_req_alloc_buf_level 60 tx_path_tx_req_alloc_buf_level
61 $ cat qos_tx_dp 61 $ cat qos_tx_dp
62 1 62 1
63 $ echo 2 > qos_tx_dp 63 $ echo 2 > qos_tx_dp
64 $ cat qos_tx_dp 64 $ cat qos_tx_dp
65 2 65 2
66 66
67 Current value (numerical value) of the event c 67 Current value (numerical value) of the event can be simply read
68 from the file, and the desired value written t 68 from the file, and the desired value written to the file to tune.
69 69
70 1. Tx Path QoS Control 70 1. Tx Path QoS Control
71 ------------------------ 71 ------------------------
72 72
73 The following files are provided to tune the Q 73 The following files are provided to tune the QoS of the tx path of
74 the PCIe core. 74 the PCIe core.
75 75
76 - qos_tx_cpl: weight of Tx completion TLPs 76 - qos_tx_cpl: weight of Tx completion TLPs
77 - qos_tx_np: weight of Tx non-posted TLPs 77 - qos_tx_np: weight of Tx non-posted TLPs
78 - qos_tx_p: weight of Tx posted TLPs 78 - qos_tx_p: weight of Tx posted TLPs
79 79
80 The weight influences the proportion of certai 80 The weight influences the proportion of certain packets on the PCIe link.
81 For example, for the storage scenario, increas 81 For example, for the storage scenario, increase the proportion
82 of the completion packets on the link to enhan 82 of the completion packets on the link to enhance the performance as
83 more completions are consumed. 83 more completions are consumed.
84 84
85 The available tune data of these events is [0, 85 The available tune data of these events is [0, 1, 2].
86 Writing a negative value will return an error, 86 Writing a negative value will return an error, and out of range
87 values will be converted to 2. Note that the e 87 values will be converted to 2. Note that the event value just
88 indicates a probable level, but is not precise 88 indicates a probable level, but is not precise.
89 89
90 2. Tx Path Buffer Control 90 2. Tx Path Buffer Control
91 ------------------------- 91 -------------------------
92 92
93 Following files are provided to tune the buffe 93 Following files are provided to tune the buffer of tx path of the PCIe core.
94 94
95 - rx_alloc_buf_level: watermark of Rx requeste 95 - rx_alloc_buf_level: watermark of Rx requested
96 - tx_alloc_buf_level: watermark of Tx requeste 96 - tx_alloc_buf_level: watermark of Tx requested
97 97
98 These events influence the watermark of the bu 98 These events influence the watermark of the buffer allocated for each
99 type. Rx means the inbound while Tx means outb 99 type. Rx means the inbound while Tx means outbound. The packets will
100 be stored in the buffer first and then transmi 100 be stored in the buffer first and then transmitted either when the
101 watermark reached or when timed out. For a bus 101 watermark reached or when timed out. For a busy direction, you should
102 increase the related buffer watermark to avoid 102 increase the related buffer watermark to avoid frequently posting and
103 thus enhance the performance. In most cases ju 103 thus enhance the performance. In most cases just keep the default value.
104 104
105 The available tune data of above events is [0, 105 The available tune data of above events is [0, 1, 2].
106 Writing a negative value will return an error, 106 Writing a negative value will return an error, and out of range
107 values will be converted to 2. Note that the e 107 values will be converted to 2. Note that the event value just
108 indicates a probable level, but is not precise 108 indicates a probable level, but is not precise.
109 109
110 Trace 110 Trace
111 ===== 111 =====
112 112
113 PTT trace is designed for dumping the TLP head 113 PTT trace is designed for dumping the TLP headers to the memory, which
114 can be used to analyze the transactions and us 114 can be used to analyze the transactions and usage condition of the PCIe
115 Link. You can choose to filter the traced head 115 Link. You can choose to filter the traced headers by either Requester ID,
116 or those downstream of a set of Root Ports on 116 or those downstream of a set of Root Ports on the same core of the PTT
117 device. It's also supported to trace the heade 117 device. It's also supported to trace the headers of certain type and of
118 certain direction. 118 certain direction.
119 119
120 You can use the perf command `perf record` to 120 You can use the perf command `perf record` to set the parameters, start
121 trace and get the data. It's also supported to 121 trace and get the data. It's also supported to decode the trace
122 data with `perf report`. The control parameter 122 data with `perf report`. The control parameters for trace is inputted
123 as event code for each events, which will be f 123 as event code for each events, which will be further illustrated later.
124 An example usage is like 124 An example usage is like
125 :: 125 ::
126 126
127 $ perf record -e hisi_ptt0_2/filter=0x8000 127 $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1,
128 format=1/ -- sleep 5 128 format=1/ -- sleep 5
129 129
130 This will trace the TLP headers downstream roo 130 This will trace the TLP headers downstream root port 0000:00:10.1 (event
131 code for event 'filter' is 0x80001) with type 131 code for event 'filter' is 0x80001) with type of posted TLP requests,
132 direction of inbound and traced data format of 132 direction of inbound and traced data format of 8DW.
133 133
134 1. Filter 134 1. Filter
135 --------- 135 ---------
136 136
137 The TLP headers to trace can be filtered by th 137 The TLP headers to trace can be filtered by the Root Ports or the Requester ID
138 of the Endpoint, which are located on the same 138 of the Endpoint, which are located on the same core of the PTT device. You can
139 set the filter by specifying the `filter` para 139 set the filter by specifying the `filter` parameter which is required to start
140 the trace. The parameter value is 20 bit. Bit 140 the trace. The parameter value is 20 bit. Bit 19 indicates the filter type.
141 1 for Root Port filter and 0 for Requester fil 141 1 for Root Port filter and 0 for Requester filter. Bit[15:0] indicates the
142 filter value. The value for a Root Port is a m 142 filter value. The value for a Root Port is a mask of the core port id which is
143 calculated from its PCI Slot ID as (slotid & 7 143 calculated from its PCI Slot ID as (slotid & 7) * 2. The value for a Requester
144 is the Requester ID (Device ID of the PCIe fun 144 is the Requester ID (Device ID of the PCIe function). Bit[18:16] is currently
145 reserved for extension. 145 reserved for extension.
146 146
147 For example, if the desired filter is Endpoint 147 For example, if the desired filter is Endpoint function 0000:01:00.1 the filter
148 value will be 0x00101. If the desired filter i 148 value will be 0x00101. If the desired filter is Root Port 0000:00:10.0 then
149 then filter value is calculated as 0x80001. 149 then filter value is calculated as 0x80001.
150 150
151 The driver also presents every supported Root <<
152 sysfs. Each filter will be an individual file <<
153 device name (domain:bus:device.function). The <<
154 under $(PTT PMU dir)/root_port_filters and fil <<
155 are under $(PTT PMU dir)/requester_filters. <<
156 <<
157 Note that multiple Root Ports can be specified 151 Note that multiple Root Ports can be specified at one time, but only one
158 Endpoint function can be specified in one trac 152 Endpoint function can be specified in one trace. Specifying both Root Port
159 and function at the same time is not supported 153 and function at the same time is not supported. Driver maintains a list of
160 available filters and will check the invalid i 154 available filters and will check the invalid inputs.
161 155
162 The available filters will be dynamically upda !! 156 Currently the available filters are detected in driver's probe. If the supported
163 get correct filter information when hotplug ev !! 157 devices are removed/added after probe, you may need to reload the driver to update
164 remove/rescan the devices. !! 158 the filters.
165 159
166 2. Type 160 2. Type
167 ------- 161 -------
168 162
169 You can trace the TLP headers of certain types 163 You can trace the TLP headers of certain types by specifying the `type`
170 parameter, which is required to start the trac 164 parameter, which is required to start the trace. The parameter value is
171 8 bit. Current supported types and related val 165 8 bit. Current supported types and related values are shown below:
172 166
173 - 8'b00000001: posted requests (P) 167 - 8'b00000001: posted requests (P)
174 - 8'b00000010: non-posted requests (NP) 168 - 8'b00000010: non-posted requests (NP)
175 - 8'b00000100: completions (CPL) 169 - 8'b00000100: completions (CPL)
176 170
177 You can specify multiple types when tracing in 171 You can specify multiple types when tracing inbound TLP headers, but can only
178 specify one when tracing outbound TLP headers. 172 specify one when tracing outbound TLP headers.
179 173
180 3. Direction 174 3. Direction
181 ------------ 175 ------------
182 176
183 You can trace the TLP headers from certain dir 177 You can trace the TLP headers from certain direction, which is relative
184 to the Root Port or the PCIe core, by specifyi 178 to the Root Port or the PCIe core, by specifying the `direction` parameter.
185 This is optional and the default parameter is 179 This is optional and the default parameter is inbound. The parameter value
186 is 4 bit. When the desired format is 4DW, dire 180 is 4 bit. When the desired format is 4DW, directions and related values
187 supported are shown below: 181 supported are shown below:
188 182
189 - 4'b0000: inbound TLPs (P, NP, CPL) 183 - 4'b0000: inbound TLPs (P, NP, CPL)
190 - 4'b0001: outbound TLPs (P, NP, CPL) 184 - 4'b0001: outbound TLPs (P, NP, CPL)
191 - 4'b0010: outbound TLPs (P, NP, CPL) and inbo 185 - 4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)
192 - 4'b0011: outbound TLPs (P, NP, CPL) and inbo 186 - 4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)
193 187
194 When the desired format is 8DW, directions and 188 When the desired format is 8DW, directions and related values supported are
195 shown below: 189 shown below:
196 190
197 - 4'b0000: reserved 191 - 4'b0000: reserved
198 - 4'b0001: outbound TLPs (P, NP, CPL) 192 - 4'b0001: outbound TLPs (P, NP, CPL)
199 - 4'b0010: inbound TLPs (P, NP, CPL B) 193 - 4'b0010: inbound TLPs (P, NP, CPL B)
200 - 4'b0011: inbound TLPs (CPL A) 194 - 4'b0011: inbound TLPs (CPL A)
201 195
202 Inbound completions are classified into two ty 196 Inbound completions are classified into two types:
203 197
204 - completion A (CPL A): completion of CHI/DMA/ 198 - completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B
205 - completion B (CPL B): completion of DMA remo 199 - completion B (CPL B): completion of DMA remote2local and P2P non-posted requests
206 200
207 4. Format 201 4. Format
208 -------------- 202 --------------
209 203
210 You can change the format of the traced TLP he 204 You can change the format of the traced TLP headers by specifying the
211 `format` parameter. The default format is 4DW. 205 `format` parameter. The default format is 4DW. The parameter value is 4 bit.
212 Current supported formats and related values a 206 Current supported formats and related values are shown below:
213 207
214 - 4'b0000: 4DW length per TLP header 208 - 4'b0000: 4DW length per TLP header
215 - 4'b0001: 8DW length per TLP header 209 - 4'b0001: 8DW length per TLP header
216 210
217 The traced TLP header format is different from 211 The traced TLP header format is different from the PCIe standard.
218 212
219 When using the 8DW data format, the entire TLP 213 When using the 8DW data format, the entire TLP header is logged
220 (Header DW0-3 shown below). For example, the T 214 (Header DW0-3 shown below). For example, the TLP header for Memory
221 Reads with 64-bit addresses is shown in PCIe r 215 Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;
222 the header for Configuration Requests is shown 216 the header for Configuration Requests is shown in Figure 2.20, etc.
223 217
224 In addition, 8DW trace buffer entries contain 218 In addition, 8DW trace buffer entries contain a timestamp and
225 possibly a prefix for a PASID TLP prefix (see 219 possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).
226 Otherwise this field will be all 0. 220 Otherwise this field will be all 0.
227 221
228 The bit[31:11] of DW0 is always 0x1fffff, whic 222 The bit[31:11] of DW0 is always 0x1fffff, which can be
229 used to distinguish the data format. 8DW forma 223 used to distinguish the data format. 8DW format is like
230 :: 224 ::
231 225
232 bits [ 31:11 226 bits [ 31:11 ][ 10:0 ]
233 |------------------------------------ 227 |---------------------------------------|-------------------|
234 DW0 [ 0x1fffff 228 DW0 [ 0x1fffff ][ Reserved (0x7ff) ]
235 DW1 [ Prefix 229 DW1 [ Prefix ]
236 DW2 [ Header DW0 230 DW2 [ Header DW0 ]
237 DW3 [ Header DW1 231 DW3 [ Header DW1 ]
238 DW4 [ Header DW2 232 DW4 [ Header DW2 ]
239 DW5 [ Header DW3 233 DW5 [ Header DW3 ]
240 DW6 [ Reserved (0x0) 234 DW6 [ Reserved (0x0) ]
241 DW7 [ Time 235 DW7 [ Time ]
242 236
243 When using the 4DW data format, DW0 of the tra 237 When using the 4DW data format, DW0 of the trace buffer entry
244 contains selected fields of DW0 of the TLP, to 238 contains selected fields of DW0 of the TLP, together with a
245 timestamp. DW1-DW3 of the trace buffer entry 239 timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3
246 directly from the TLP header. 240 directly from the TLP header.
247 241
248 4DW format is like 242 4DW format is like
249 :: 243 ::
250 244
251 bits [31:30] [ 29:25 ][24][23][22][21][ 245 bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ]
252 |-----|---------|---|---|---|---|---- 246 |-----|---------|---|---|---|---|-------------|-------------|
253 DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ 247 DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ]
254 DW1 [ Header DW1 248 DW1 [ Header DW1 ]
255 DW2 [ Header DW2 249 DW2 [ Header DW2 ]
256 DW3 [ Header DW3 250 DW3 [ Header DW3 ]
257 251
258 5. Memory Management 252 5. Memory Management
259 -------------------- 253 --------------------
260 254
261 The traced TLP headers will be written to the 255 The traced TLP headers will be written to the memory allocated
262 by the driver. The hardware accepts 4 DMA addr 256 by the driver. The hardware accepts 4 DMA address with same size,
263 and writes the buffer sequentially like below. 257 and writes the buffer sequentially like below. If DMA addr 3 is
264 finished and the trace is still on, it will re 258 finished and the trace is still on, it will return to addr 0.
265 :: 259 ::
266 260
267 +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2 261 +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+
268 +----------------------------------------- 262 +---------------------------------------------------------+
269 263
270 Driver will allocate each DMA buffer of 4MiB. 264 Driver will allocate each DMA buffer of 4MiB. The finished buffer
271 will be copied to the perf AUX buffer allocate 265 will be copied to the perf AUX buffer allocated by the perf core.
272 Once the AUX buffer is full while the trace is 266 Once the AUX buffer is full while the trace is still on, driver
273 will commit the AUX buffer first and then appl 267 will commit the AUX buffer first and then apply for a new one with
274 the same size. The size of AUX buffer is defau 268 the same size. The size of AUX buffer is default to 16MiB. User can
275 adjust the size by specifying the `-m` paramet 269 adjust the size by specifying the `-m` parameter of the perf command.
276 270
277 6. Decoding 271 6. Decoding
278 ----------- 272 -----------
279 273
280 You can decode the traced data with `perf repo 274 You can decode the traced data with `perf report -D` command (currently
281 only support to dump the raw trace data). The 275 only support to dump the raw trace data). The traced data will be decoded
282 according to the format described previously ( 276 according to the format described previously (take 8DW as an example):
283 :: 277 ::
284 278
285 [...perf headers and other information] 279 [...perf headers and other information]
286 . ... HISI PTT data: size 4194304 bytes 280 . ... HISI PTT data: size 4194304 bytes
287 . 00000000: 00 00 00 00 281 . 00000000: 00 00 00 00 Prefix
288 . 00000004: 01 00 00 60 282 . 00000004: 01 00 00 60 Header DW0
289 . 00000008: 0f 1e 00 01 283 . 00000008: 0f 1e 00 01 Header DW1
290 . 0000000c: 04 00 00 00 284 . 0000000c: 04 00 00 00 Header DW2
291 . 00000010: 40 00 81 02 285 . 00000010: 40 00 81 02 Header DW3
292 . 00000014: 33 c0 04 00 286 . 00000014: 33 c0 04 00 Time
293 . 00000020: 00 00 00 00 287 . 00000020: 00 00 00 00 Prefix
294 . 00000024: 01 00 00 60 288 . 00000024: 01 00 00 60 Header DW0
295 . 00000028: 0f 1e 00 01 289 . 00000028: 0f 1e 00 01 Header DW1
296 . 0000002c: 04 00 00 00 290 . 0000002c: 04 00 00 00 Header DW2
297 . 00000030: 40 00 81 02 291 . 00000030: 40 00 81 02 Header DW3
298 . 00000034: 02 00 00 00 292 . 00000034: 02 00 00 00 Time
299 . 00000040: 00 00 00 00 293 . 00000040: 00 00 00 00 Prefix
300 . 00000044: 01 00 00 60 294 . 00000044: 01 00 00 60 Header DW0
301 . 00000048: 0f 1e 00 01 295 . 00000048: 0f 1e 00 01 Header DW1
302 . 0000004c: 04 00 00 00 296 . 0000004c: 04 00 00 00 Header DW2
303 . 00000050: 40 00 81 02 297 . 00000050: 40 00 81 02 Header DW3
304 [...] 298 [...]
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.