1 What: /sys/kernel/debug/iommu/intel/ 1 What: /sys/kernel/debug/iommu/intel/iommu_regset
2 Date: December 2023 2 Date: December 2023
3 Contact: Jingqi Liu <Jingqi.liu@intel.co 3 Contact: Jingqi Liu <Jingqi.liu@intel.com>
4 Description: 4 Description:
5 This file dumps all the regist 5 This file dumps all the register contents for each IOMMU device.
6 6
7 Example in Kabylake: 7 Example in Kabylake:
8 8
9 :: 9 ::
10 10
11 $ sudo cat /sys/kernel/debug/ 11 $ sudo cat /sys/kernel/debug/iommu/intel/iommu_regset
12 12
13 IOMMU: dmar0 Register Base Ad 13 IOMMU: dmar0 Register Base Address: 26be37000
14 14
15 Name Offse 15 Name Offset Contents
16 VER 0x00 16 VER 0x00 0x0000000000000010
17 GCMD 0x18 17 GCMD 0x18 0x0000000000000000
18 GSTS 0x1c 18 GSTS 0x1c 0x00000000c7000000
19 FSTS 0x34 19 FSTS 0x34 0x0000000000000000
20 FECTL 0x38 20 FECTL 0x38 0x0000000000000000
21 21
22 [...] 22 [...]
23 23
24 IOMMU: dmar1 Register Base Ad 24 IOMMU: dmar1 Register Base Address: fed90000
25 25
26 Name Offse 26 Name Offset Contents
27 VER 0x00 27 VER 0x00 0x0000000000000010
28 GCMD 0x18 28 GCMD 0x18 0x0000000000000000
29 GSTS 0x1c 29 GSTS 0x1c 0x00000000c7000000
30 FSTS 0x34 30 FSTS 0x34 0x0000000000000000
31 FECTL 0x38 31 FECTL 0x38 0x0000000000000000
32 32
33 [...] 33 [...]
34 34
35 IOMMU: dmar2 Register Base Ad 35 IOMMU: dmar2 Register Base Address: fed91000
36 36
37 Name Offse 37 Name Offset Contents
38 VER 0x00 38 VER 0x00 0x0000000000000010
39 GCMD 0x18 39 GCMD 0x18 0x0000000000000000
40 GSTS 0x1c 40 GSTS 0x1c 0x00000000c7000000
41 FSTS 0x34 41 FSTS 0x34 0x0000000000000000
42 FECTL 0x38 42 FECTL 0x38 0x0000000000000000
43 43
44 [...] 44 [...]
45 45
46 What: /sys/kernel/debug/iommu/intel/ 46 What: /sys/kernel/debug/iommu/intel/ir_translation_struct
47 Date: December 2023 47 Date: December 2023
48 Contact: Jingqi Liu <Jingqi.liu@intel.co 48 Contact: Jingqi Liu <Jingqi.liu@intel.com>
49 Description: 49 Description:
50 This file dumps the table entr 50 This file dumps the table entries for Interrupt
51 remapping and Interrupt postin 51 remapping and Interrupt posting.
52 52
53 Example in Kabylake: 53 Example in Kabylake:
54 54
55 :: 55 ::
56 56
57 $ sudo cat /sys/kernel/debug/ 57 $ sudo cat /sys/kernel/debug/iommu/intel/ir_translation_struct
58 58
59 Remapped Interrupt supported 59 Remapped Interrupt supported on IOMMU: dmar0
60 IR table address:100900000 60 IR table address:100900000
61 61
62 Entry SrcID DstID Vct IR 62 Entry SrcID DstID Vct IRTE_high IRTE_low
63 0 00:0a.0 00000080 24 00 63 0 00:0a.0 00000080 24 0000000000040050 000000800024000d
64 1 00:0a.0 00000001 ef 00 64 1 00:0a.0 00000001 ef 0000000000040050 0000000100ef000d
65 65
66 Remapped Interrupt supported 66 Remapped Interrupt supported on IOMMU: dmar1
67 IR table address:100300000 67 IR table address:100300000
68 Entry SrcID DstID Vct IR 68 Entry SrcID DstID Vct IRTE_high IRTE_low
69 0 00:02.0 00000002 26 00 69 0 00:02.0 00000002 26 0000000000040010 000000020026000d
70 70
71 [...] 71 [...]
72 72
73 **** 73 ****
74 74
75 Posted Interrupt supported on 75 Posted Interrupt supported on IOMMU: dmar0
76 IR table address:100900000 76 IR table address:100900000
77 Entry SrcID PDA_high PDA_lo 77 Entry SrcID PDA_high PDA_low Vct IRTE_high IRTE_low
78 78
79 What: /sys/kernel/debug/iommu/intel/ 79 What: /sys/kernel/debug/iommu/intel/dmar_translation_struct
80 Date: December 2023 80 Date: December 2023
81 Contact: Jingqi Liu <Jingqi.liu@intel.co 81 Contact: Jingqi Liu <Jingqi.liu@intel.com>
82 Description: 82 Description:
83 This file dumps Intel IOMMU DM 83 This file dumps Intel IOMMU DMA remapping tables, such
84 as root table, context table, 84 as root table, context table, PASID directory and PASID
85 table entries in debugfs. For 85 table entries in debugfs. For legacy mode, it doesn't
86 support PASID, and hence PASID 86 support PASID, and hence PASID field is defaulted to
87 '-1' and other PASID related f 87 '-1' and other PASID related fields are invalid.
88 88
89 Example in Kabylake: 89 Example in Kabylake:
90 90
91 :: 91 ::
92 92
93 $ sudo cat /sys/kernel/debug/ 93 $ sudo cat /sys/kernel/debug/iommu/intel/dmar_translation_struct
94 94
95 IOMMU dmar1: Root Table Addre 95 IOMMU dmar1: Root Table Address: 0x103027000
96 B.D.F Root_entry 96 B.D.F Root_entry
97 00:02.0 0x0000000000000000:0x 97 00:02.0 0x0000000000000000:0x000000010303e001
98 98
99 Context_entry 99 Context_entry
100 0x0000000000000102:0x00000001 100 0x0000000000000102:0x000000010303f005
101 101
102 PASID PASID_table_entry 102 PASID PASID_table_entry
103 -1 0x0000000000000000:0x 103 -1 0x0000000000000000:0x0000000000000000:0x0000000000000000
104 104
105 IOMMU dmar0: Root Table Addre 105 IOMMU dmar0: Root Table Address: 0x103028000
106 B.D.F Root_entry 106 B.D.F Root_entry
107 00:0a.0 0x0000000000000000:0x 107 00:0a.0 0x0000000000000000:0x00000001038a7001
108 108
109 Context_entry 109 Context_entry
110 0x0000000000000000:0x00000001 110 0x0000000000000000:0x0000000103220e7d
111 111
112 PASID PASID_table_entry 112 PASID PASID_table_entry
113 0 0x0000000000000000:0x 113 0 0x0000000000000000:0x0000000000800002:0x00000001038a5089
114 114
115 [...] 115 [...]
116 116
117 What: /sys/kernel/debug/iommu/intel/ 117 What: /sys/kernel/debug/iommu/intel/invalidation_queue
118 Date: December 2023 118 Date: December 2023
119 Contact: Jingqi Liu <Jingqi.liu@intel.co 119 Contact: Jingqi Liu <Jingqi.liu@intel.com>
120 Description: 120 Description:
121 This file exports invalidation 121 This file exports invalidation queue internals of each
122 IOMMU device. 122 IOMMU device.
123 123
124 Example in Kabylake: 124 Example in Kabylake:
125 125
126 :: 126 ::
127 127
128 $ sudo cat /sys/kernel/debug/ 128 $ sudo cat /sys/kernel/debug/iommu/intel/invalidation_queue
129 129
130 Invalidation queue on IOMMU: 130 Invalidation queue on IOMMU: dmar0
131 Base: 0x10022e000 Head: 131 Base: 0x10022e000 Head: 20 Tail: 20
132 Index qw0 132 Index qw0 qw1 qw2
133 0 0000000000000014 133 0 0000000000000014 0000000000000000 0000000000000000
134 1 0000000200000025 134 1 0000000200000025 0000000100059c04 0000000000000000
135 2 0000000000000014 135 2 0000000000000014 0000000000000000 0000000000000000
136 136
137 qw3 137 qw3 status
138 0000000000000000 138 0000000000000000 0000000000000000
139 0000000000000000 139 0000000000000000 0000000000000000
140 0000000000000000 140 0000000000000000 0000000000000000
141 141
142 [...] 142 [...]
143 143
144 Invalidation queue on IOMMU: 144 Invalidation queue on IOMMU: dmar1
145 Base: 0x10026e000 Head: 145 Base: 0x10026e000 Head: 32 Tail: 32
146 Index qw0 146 Index qw0 qw1 status
147 0 0000000000000004 147 0 0000000000000004 0000000000000000 0000000000000000
148 1 0000000200000025 148 1 0000000200000025 0000000100059804 0000000000000000
149 2 0000000000000011 149 2 0000000000000011 0000000000000000 0000000000000000
150 150
151 [...] 151 [...]
152 152
153 What: /sys/kernel/debug/iommu/intel/ 153 What: /sys/kernel/debug/iommu/intel/dmar_perf_latency
154 Date: December 2023 154 Date: December 2023
155 Contact: Jingqi Liu <Jingqi.liu@intel.co 155 Contact: Jingqi Liu <Jingqi.liu@intel.com>
156 Description: 156 Description:
157 This file is used to control a 157 This file is used to control and show counts of
158 execution time ranges for vari 158 execution time ranges for various types per DMAR.
159 159
160 Firstly, write a value to 160 Firstly, write a value to
161 /sys/kernel/debug/iommu/intel/ 161 /sys/kernel/debug/iommu/intel/dmar_perf_latency
162 to enable sampling. 162 to enable sampling.
163 163
164 The possible values are as fol 164 The possible values are as follows:
165 165
166 * 0 - disable sampling all lat 166 * 0 - disable sampling all latency data
167 167
168 * 1 - enable sampling IOTLB in 168 * 1 - enable sampling IOTLB invalidation latency data
169 169
170 * 2 - enable sampling devTLB i 170 * 2 - enable sampling devTLB invalidation latency data
171 171
172 * 3 - enable sampling intr ent 172 * 3 - enable sampling intr entry cache invalidation latency data
173 173
174 Next, read /sys/kernel/debug/i 174 Next, read /sys/kernel/debug/iommu/intel/dmar_perf_latency gives
175 a snapshot of sampling result 175 a snapshot of sampling result of all enabled monitors.
176 176
177 Examples in Kabylake: 177 Examples in Kabylake:
178 178
179 :: 179 ::
180 180
181 1) Disable sampling all laten 181 1) Disable sampling all latency data:
182 182
183 $ sudo echo 0 > /sys/kernel/d 183 $ sudo echo 0 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
184 184
185 2) Enable sampling IOTLB inva 185 2) Enable sampling IOTLB invalidation latency data
186 186
187 $ sudo echo 1 > /sys/kernel/d 187 $ sudo echo 1 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
188 188
189 $ sudo cat /sys/kernel/debug/ 189 $ sudo cat /sys/kernel/debug/iommu/intel/dmar_perf_latency
190 190
191 IOMMU: dmar0 Register Base Ad 191 IOMMU: dmar0 Register Base Address: 26be37000
192 <0.1us 0.1u 192 <0.1us 0.1us-1us 1us-10us 10us-100us 100us-1ms
193 inv_iotlb 0 193 inv_iotlb 0 0 0 0 0
194 194
195 1ms-10ms 195 1ms-10ms >=10ms min(us) max(us) average(us)
196 inv_iotlb 0 196 inv_iotlb 0 0 0 0 0
197 197
198 [...] 198 [...]
199 199
200 IOMMU: dmar2 Register Base Ad 200 IOMMU: dmar2 Register Base Address: fed91000
201 <0.1us 0.1u 201 <0.1us 0.1us-1us 1us-10us 10us-100us 100us-1ms
202 inv_iotlb 0 202 inv_iotlb 0 0 18 0 0
203 203
204 1ms-10ms 204 1ms-10ms >=10ms min(us) max(us) average(us)
205 inv_iotlb 0 205 inv_iotlb 0 0 2 2 2
206 206
207 3) Enable sampling devTLB inv 207 3) Enable sampling devTLB invalidation latency data
208 208
209 $ sudo echo 2 > /sys/kernel/d 209 $ sudo echo 2 > /sys/kernel/debug/iommu/intel/dmar_perf_latency
210 210
211 $ sudo cat /sys/kernel/debug/ 211 $ sudo cat /sys/kernel/debug/iommu/intel/dmar_perf_latency
212 212
213 IOMMU: dmar0 Register Base Ad 213 IOMMU: dmar0 Register Base Address: 26be37000
214 <0.1us 0.1u 214 <0.1us 0.1us-1us 1us-10us 10us-100us 100us-1ms
215 inv_devtlb 0 215 inv_devtlb 0 0 0 0 0
216 216
217 >=10ms mi 217 >=10ms min(us) max(us) average(us)
218 inv_devtlb 0 218 inv_devtlb 0 0 0 0
219 219
220 [...] 220 [...]
221 221
222 What: /sys/kernel/debug/iommu/intel/ 222 What: /sys/kernel/debug/iommu/intel/<bdf>/domain_translation_struct
223 Date: December 2023 223 Date: December 2023
224 Contact: Jingqi Liu <Jingqi.liu@intel.co 224 Contact: Jingqi Liu <Jingqi.liu@intel.com>
225 Description: 225 Description:
226 This file dumps a specified pa 226 This file dumps a specified page table of Intel IOMMU
227 in legacy mode or scalable mod 227 in legacy mode or scalable mode.
228 228
229 For a device that only support 229 For a device that only supports legacy mode, dump its
230 page table by the debugfs file 230 page table by the debugfs file in the debugfs device
231 directory. e.g. 231 directory. e.g.
232 /sys/kernel/debug/iommu/intel/ 232 /sys/kernel/debug/iommu/intel/0000:00:02.0/domain_translation_struct.
233 233
234 For a device that supports sca 234 For a device that supports scalable mode, dump the
235 page table of specified pasid 235 page table of specified pasid by the debugfs file in
236 the debugfs pasid directory. e 236 the debugfs pasid directory. e.g.
237 /sys/kernel/debug/iommu/intel/ 237 /sys/kernel/debug/iommu/intel/0000:00:02.0/1/domain_translation_struct.
238 238
239 Examples in Kabylake: 239 Examples in Kabylake:
240 240
241 :: 241 ::
242 242
243 1) Dump the page table of dev 243 1) Dump the page table of device "0000:00:02.0" that only supports legacy mode.
244 244
245 $ sudo cat /sys/kernel/debug/ 245 $ sudo cat /sys/kernel/debug/iommu/intel/0000:00:02.0/domain_translation_struct
246 246
247 Device 0000:00:02.0 @0x1017f8 247 Device 0000:00:02.0 @0x1017f8000
248 IOVA_PFN PML5E 248 IOVA_PFN PML5E PML4E
249 0x000000008d800 | 0x000 249 0x000000008d800 | 0x0000000000000000 0x00000001017f9003
250 0x000000008d801 | 0x000 250 0x000000008d801 | 0x0000000000000000 0x00000001017f9003
251 0x000000008d802 | 0x000 251 0x000000008d802 | 0x0000000000000000 0x00000001017f9003
252 252
253 PDPE PDE 253 PDPE PDE PTE
254 0x00000001017fa003 0x000 254 0x00000001017fa003 0x00000001017fb003 0x000000008d800003
255 0x00000001017fa003 0x000 255 0x00000001017fa003 0x00000001017fb003 0x000000008d801003
256 0x00000001017fa003 0x000 256 0x00000001017fa003 0x00000001017fb003 0x000000008d802003
257 257
258 [...] 258 [...]
259 259
260 2) Dump the page table of dev 260 2) Dump the page table of device "0000:00:0a.0" with PASID "1" that
261 supports scalable mode. 261 supports scalable mode.
262 262
263 $ sudo cat /sys/kernel/debug/ 263 $ sudo cat /sys/kernel/debug/iommu/intel/0000:00:0a.0/1/domain_translation_struct
264 264
265 Device 0000:00:0a.0 with pasi 265 Device 0000:00:0a.0 with pasid 1 @0x10c112000
266 IOVA_PFN PML5E 266 IOVA_PFN PML5E PML4E
267 0x0000000000000 | 0x000 267 0x0000000000000 | 0x0000000000000000 0x000000010df93003
268 0x0000000000001 | 0x000 268 0x0000000000001 | 0x0000000000000000 0x000000010df93003
269 0x0000000000002 | 0x000 269 0x0000000000002 | 0x0000000000000000 0x000000010df93003
270 270
271 PDPE PDE 271 PDPE PDE PTE
272 0x0000000106ae6003 0x000 272 0x0000000106ae6003 0x0000000104b38003 0x0000000147c00803
273 0x0000000106ae6003 0x000 273 0x0000000106ae6003 0x0000000104b38003 0x0000000147c01803
274 0x0000000106ae6003 0x000 274 0x0000000106ae6003 0x0000000104b38003 0x0000000147c02803
275 275
276 [...] 276 [...]
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.