1 .. SPDX-License-Identifier: GPL-2.0 2 3 ====================================== 4 HiSilicon PCIe Tune and Trace device 5 ====================================== 6 7 Introduction 8 ============ 9 10 HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex 11 integrated Endpoint (RCiEP) device, providing the capability 12 to dynamically monitor and tune the PCIe link's events (tune), 13 and trace the TLP headers (trace). The two functions are independent, 14 but is recommended to use them together to analyze and enhance the 15 PCIe link's performance. 16 17 On Kunpeng 930 SoC, the PCIe Root Complex is composed of several 18 PCIe cores. Each PCIe core includes several Root Ports and a PTT 19 RCiEP, like below. The PTT device is capable of tuning and 20 tracing the links of the PCIe core. 21 :: 22 23 +--------------Core 0-------+ 24 | | [ PTT ] | 25 | | [Root Port]---[Endpoint] 26 | | [Root Port]---[Endpoint] 27 | | [Root Port]---[Endpoint] 28 Root Complex |------Core 1-------+ 29 | | [ PTT ] | 30 | | [Root Port]---[ Switch ]---[Endpoint] 31 | | [Root Port]---[Endpoint] `-[Endpoint] 32 | | [Root Port]---[Endpoint] 33 +---------------------------+ 34 35 The PTT device driver registers one PMU device for each PTT device. 36 The name of each PTT device is composed of 'hisi_ptt' prefix with 37 the id of the SICL and the Core where it locates. The Kunpeng 930 38 SoC encapsulates multiple CPU dies (SCCL, Super CPU Cluster) and 39 IO dies (SICL, Super I/O Cluster), where there's one PCIe Root 40 Complex for each SICL. 41 :: 42 43 /sys/bus/event_source/devices/hisi_ptt<sicl_id>_<core_id> 44 45 Tune 46 ==== 47 48 PTT tune is designed for monitoring and adjusting PCIe link parameters (events). 49 Currently we support events in 2 classes. The scope of the events 50 covers the PCIe core to which the PTT device belongs. 51 52 Each event is presented as a file under $(PTT PMU dir)/tune, and 53 a simple open/read/write/close cycle will be used to tune the event. 54 :: 55 56 $ cd /sys/bus/event_source/devices/hisi_ptt<sicl_id>_<core_id>/tune 57 $ ls 58 qos_tx_cpl qos_tx_np qos_tx_p 59 tx_path_rx_req_alloc_buf_level 60 tx_path_tx_req_alloc_buf_level 61 $ cat qos_tx_dp 62 1 63 $ echo 2 > qos_tx_dp 64 $ cat qos_tx_dp 65 2 66 67 Current value (numerical value) of the event can be simply read 68 from the file, and the desired value written to the file to tune. 69 70 1. Tx Path QoS Control 71 ------------------------ 72 73 The following files are provided to tune the QoS of the tx path of 74 the PCIe core. 75 76 - qos_tx_cpl: weight of Tx completion TLPs 77 - qos_tx_np: weight of Tx non-posted TLPs 78 - qos_tx_p: weight of Tx posted TLPs 79 80 The weight influences the proportion of certain packets on the PCIe link. 81 For example, for the storage scenario, increase the proportion 82 of the completion packets on the link to enhance the performance as 83 more completions are consumed. 84 85 The available tune data of these events is [0, 1, 2]. 86 Writing a negative value will return an error, and out of range 87 values will be converted to 2. Note that the event value just 88 indicates a probable level, but is not precise. 89 90 2. Tx Path Buffer Control 91 ------------------------- 92 93 Following files are provided to tune the buffer of tx path of the PCIe core. 94 95 - rx_alloc_buf_level: watermark of Rx requested 96 - tx_alloc_buf_level: watermark of Tx requested 97 98 These events influence the watermark of the buffer allocated for each 99 type. Rx means the inbound while Tx means outbound. The packets will 100 be stored in the buffer first and then transmitted either when the 101 watermark reached or when timed out. For a busy direction, you should 102 increase the related buffer watermark to avoid frequently posting and 103 thus enhance the performance. In most cases just keep the default value. 104 105 The available tune data of above events is [0, 1, 2]. 106 Writing a negative value will return an error, and out of range 107 values will be converted to 2. Note that the event value just 108 indicates a probable level, but is not precise. 109 110 Trace 111 ===== 112 113 PTT trace is designed for dumping the TLP headers to the memory, which 114 can be used to analyze the transactions and usage condition of the PCIe 115 Link. You can choose to filter the traced headers by either Requester ID, 116 or those downstream of a set of Root Ports on the same core of the PTT 117 device. It's also supported to trace the headers of certain type and of 118 certain direction. 119 120 You can use the perf command `perf record` to set the parameters, start 121 trace and get the data. It's also supported to decode the trace 122 data with `perf report`. The control parameters for trace is inputted 123 as event code for each events, which will be further illustrated later. 124 An example usage is like 125 :: 126 127 $ perf record -e hisi_ptt0_2/filter=0x80001,type=1,direction=1, 128 format=1/ -- sleep 5 129 130 This will trace the TLP headers downstream root port 0000:00:10.1 (event 131 code for event 'filter' is 0x80001) with type of posted TLP requests, 132 direction of inbound and traced data format of 8DW. 133 134 1. Filter 135 --------- 136 137 The TLP headers to trace can be filtered by the Root Ports or the Requester ID 138 of the Endpoint, which are located on the same core of the PTT device. You can 139 set the filter by specifying the `filter` parameter which is required to start 140 the trace. The parameter value is 20 bit. Bit 19 indicates the filter type. 141 1 for Root Port filter and 0 for Requester filter. Bit[15:0] indicates the 142 filter value. The value for a Root Port is a mask of the core port id which is 143 calculated from its PCI Slot ID as (slotid & 7) * 2. The value for a Requester 144 is the Requester ID (Device ID of the PCIe function). Bit[18:16] is currently 145 reserved for extension. 146 147 For example, if the desired filter is Endpoint function 0000:01:00.1 the filter 148 value will be 0x00101. If the desired filter is Root Port 0000:00:10.0 then 149 then filter value is calculated as 0x80001. 150 151 The driver also presents every supported Root Port and Requester filter through 152 sysfs. Each filter will be an individual file with name of its related PCIe 153 device name (domain:bus:device.function). The files of Root Port filters are 154 under $(PTT PMU dir)/root_port_filters and files of Requester filters 155 are under $(PTT PMU dir)/requester_filters. 156 157 Note that multiple Root Ports can be specified at one time, but only one 158 Endpoint function can be specified in one trace. Specifying both Root Port 159 and function at the same time is not supported. Driver maintains a list of 160 available filters and will check the invalid inputs. 161 162 The available filters will be dynamically updated, which means you will always 163 get correct filter information when hotplug events happen, or when you manually 164 remove/rescan the devices. 165 166 2. Type 167 ------- 168 169 You can trace the TLP headers of certain types by specifying the `type` 170 parameter, which is required to start the trace. The parameter value is 171 8 bit. Current supported types and related values are shown below: 172 173 - 8'b00000001: posted requests (P) 174 - 8'b00000010: non-posted requests (NP) 175 - 8'b00000100: completions (CPL) 176 177 You can specify multiple types when tracing inbound TLP headers, but can only 178 specify one when tracing outbound TLP headers. 179 180 3. Direction 181 ------------ 182 183 You can trace the TLP headers from certain direction, which is relative 184 to the Root Port or the PCIe core, by specifying the `direction` parameter. 185 This is optional and the default parameter is inbound. The parameter value 186 is 4 bit. When the desired format is 4DW, directions and related values 187 supported are shown below: 188 189 - 4'b0000: inbound TLPs (P, NP, CPL) 190 - 4'b0001: outbound TLPs (P, NP, CPL) 191 - 4'b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B) 192 - 4'b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A) 193 194 When the desired format is 8DW, directions and related values supported are 195 shown below: 196 197 - 4'b0000: reserved 198 - 4'b0001: outbound TLPs (P, NP, CPL) 199 - 4'b0010: inbound TLPs (P, NP, CPL B) 200 - 4'b0011: inbound TLPs (CPL A) 201 202 Inbound completions are classified into two types: 203 204 - completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B 205 - completion B (CPL B): completion of DMA remote2local and P2P non-posted requests 206 207 4. Format 208 -------------- 209 210 You can change the format of the traced TLP headers by specifying the 211 `format` parameter. The default format is 4DW. The parameter value is 4 bit. 212 Current supported formats and related values are shown below: 213 214 - 4'b0000: 4DW length per TLP header 215 - 4'b0001: 8DW length per TLP header 216 217 The traced TLP header format is different from the PCIe standard. 218 219 When using the 8DW data format, the entire TLP header is logged 220 (Header DW0-3 shown below). For example, the TLP header for Memory 221 Reads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17; 222 the header for Configuration Requests is shown in Figure 2.20, etc. 223 224 In addition, 8DW trace buffer entries contain a timestamp and 225 possibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0). 226 Otherwise this field will be all 0. 227 228 The bit[31:11] of DW0 is always 0x1fffff, which can be 229 used to distinguish the data format. 8DW format is like 230 :: 231 232 bits [ 31:11 ][ 10:0 ] 233 |---------------------------------------|-------------------| 234 DW0 [ 0x1fffff ][ Reserved (0x7ff) ] 235 DW1 [ Prefix ] 236 DW2 [ Header DW0 ] 237 DW3 [ Header DW1 ] 238 DW4 [ Header DW2 ] 239 DW5 [ Header DW3 ] 240 DW6 [ Reserved (0x0) ] 241 DW7 [ Time ] 242 243 When using the 4DW data format, DW0 of the trace buffer entry 244 contains selected fields of DW0 of the TLP, together with a 245 timestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3 246 directly from the TLP header. 247 248 4DW format is like 249 :: 250 251 bits [31:30] [ 29:25 ][24][23][22][21][ 20:11 ][ 10:0 ] 252 |-----|---------|---|---|---|---|-------------|-------------| 253 DW0 [ Fmt ][ Type ][T9][T8][TH][SO][ Length ][ Time ] 254 DW1 [ Header DW1 ] 255 DW2 [ Header DW2 ] 256 DW3 [ Header DW3 ] 257 258 5. Memory Management 259 -------------------- 260 261 The traced TLP headers will be written to the memory allocated 262 by the driver. The hardware accepts 4 DMA address with same size, 263 and writes the buffer sequentially like below. If DMA addr 3 is 264 finished and the trace is still on, it will return to addr 0. 265 :: 266 267 +->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-+ 268 +---------------------------------------------------------+ 269 270 Driver will allocate each DMA buffer of 4MiB. The finished buffer 271 will be copied to the perf AUX buffer allocated by the perf core. 272 Once the AUX buffer is full while the trace is still on, driver 273 will commit the AUX buffer first and then apply for a new one with 274 the same size. The size of AUX buffer is default to 16MiB. User can 275 adjust the size by specifying the `-m` parameter of the perf command. 276 277 6. Decoding 278 ----------- 279 280 You can decode the traced data with `perf report -D` command (currently 281 only support to dump the raw trace data). The traced data will be decoded 282 according to the format described previously (take 8DW as an example): 283 :: 284 285 [...perf headers and other information] 286 . ... HISI PTT data: size 4194304 bytes 287 . 00000000: 00 00 00 00 Prefix 288 . 00000004: 01 00 00 60 Header DW0 289 . 00000008: 0f 1e 00 01 Header DW1 290 . 0000000c: 04 00 00 00 Header DW2 291 . 00000010: 40 00 81 02 Header DW3 292 . 00000014: 33 c0 04 00 Time 293 . 00000020: 00 00 00 00 Prefix 294 . 00000024: 01 00 00 60 Header DW0 295 . 00000028: 0f 1e 00 01 Header DW1 296 . 0000002c: 04 00 00 00 Header DW2 297 . 00000030: 40 00 81 02 Header DW3 298 . 00000034: 02 00 00 00 Time 299 . 00000040: 00 00 00 00 Prefix 300 . 00000044: 01 00 00 60 Header DW0 301 . 00000048: 0f 1e 00 01 Header DW1 302 . 0000004c: 04 00 00 00 Header DW2 303 . 00000050: 40 00 81 02 Header DW3 304 [...]
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.