1 =================================== 1 ===================================
2 In-kernel memory-mapped I/O tracing 2 In-kernel memory-mapped I/O tracing
3 =================================== 3 ===================================
4 4
5 5
6 Home page and links to optional user space too 6 Home page and links to optional user space tools:
7 7
8 https://nouveau.freedesktop.org/wiki/M !! 8 http://nouveau.freedesktop.org/wiki/MmioTrace
9 9
10 MMIO tracing was originally developed by Intel 10 MMIO tracing was originally developed by Intel around 2003 for their Fault
11 Injection Test Harness. In Dec 2006 - Jan 2007 11 Injection Test Harness. In Dec 2006 - Jan 2007, using the code from Intel,
12 Jeff Muizelaar created a tool for tracing MMIO 12 Jeff Muizelaar created a tool for tracing MMIO accesses with the Nouveau
13 project in mind. Since then many people have c 13 project in mind. Since then many people have contributed.
14 14
15 Mmiotrace was built for reverse engineering an 15 Mmiotrace was built for reverse engineering any memory-mapped IO device with
16 the Nouveau project as the first real user. On 16 the Nouveau project as the first real user. Only x86 and x86_64 architectures
17 are supported. 17 are supported.
18 18
19 Out-of-tree mmiotrace was originally modified 19 Out-of-tree mmiotrace was originally modified for mainline inclusion and
20 ftrace framework by Pekka Paalanen <pq@iki.fi>. 20 ftrace framework by Pekka Paalanen <pq@iki.fi>.
21 21
22 22
23 Preparation 23 Preparation
24 ----------- 24 -----------
25 25
26 Mmiotrace feature is compiled in by the CONFIG 26 Mmiotrace feature is compiled in by the CONFIG_MMIOTRACE option. Tracing is
27 disabled by default, so it is safe to have thi 27 disabled by default, so it is safe to have this set to yes. SMP systems are
28 supported, but tracing is unreliable and may m 28 supported, but tracing is unreliable and may miss events if more than one CPU
29 is on-line, therefore mmiotrace takes all but 29 is on-line, therefore mmiotrace takes all but one CPU off-line during run-time
30 activation. You can re-enable CPUs by hand, bu 30 activation. You can re-enable CPUs by hand, but you have been warned, there
31 is no way to automatically detect if you are l 31 is no way to automatically detect if you are losing events due to CPUs racing.
32 32
33 33
34 Usage Quick Reference 34 Usage Quick Reference
35 --------------------- 35 ---------------------
36 :: 36 ::
37 37
38 $ mount -t debugfs debugfs /sys/kernel 38 $ mount -t debugfs debugfs /sys/kernel/debug
39 $ echo mmiotrace > /sys/kernel/tracing !! 39 $ echo mmiotrace > /sys/kernel/debug/tracing/current_tracer
40 $ cat /sys/kernel/tracing/trace_pipe > !! 40 $ cat /sys/kernel/debug/tracing/trace_pipe > mydump.txt &
41 Start X or whatever. 41 Start X or whatever.
42 $ echo "X is up" > /sys/kernel/tracing !! 42 $ echo "X is up" > /sys/kernel/debug/tracing/trace_marker
43 $ echo nop > /sys/kernel/tracing/curre !! 43 $ echo nop > /sys/kernel/debug/tracing/current_tracer
44 Check for lost events. 44 Check for lost events.
45 45
46 46
47 Usage 47 Usage
48 ----- 48 -----
49 49
50 Make sure debugfs is mounted to /sys/kernel/de 50 Make sure debugfs is mounted to /sys/kernel/debug.
51 If not (requires root privileges):: 51 If not (requires root privileges)::
52 52
53 $ mount -t debugfs debugfs /sys/kernel 53 $ mount -t debugfs debugfs /sys/kernel/debug
54 54
55 Check that the driver you are about to trace i 55 Check that the driver you are about to trace is not loaded.
56 56
57 Activate mmiotrace (requires root privileges): 57 Activate mmiotrace (requires root privileges)::
58 58
59 $ echo mmiotrace > /sys/kernel/tracing !! 59 $ echo mmiotrace > /sys/kernel/debug/tracing/current_tracer
60 60
61 Start storing the trace:: 61 Start storing the trace::
62 62
63 $ cat /sys/kernel/tracing/trace_pipe > !! 63 $ cat /sys/kernel/debug/tracing/trace_pipe > mydump.txt &
64 64
65 The 'cat' process should stay running (sleepin 65 The 'cat' process should stay running (sleeping) in the background.
66 66
67 Load the driver you want to trace and use it. 67 Load the driver you want to trace and use it. Mmiotrace will only catch MMIO
68 accesses to areas that are ioremapped while mm 68 accesses to areas that are ioremapped while mmiotrace is active.
69 69
70 During tracing you can place comments (markers 70 During tracing you can place comments (markers) into the trace by
71 $ echo "X is up" > /sys/kernel/tracing/trace_m !! 71 $ echo "X is up" > /sys/kernel/debug/tracing/trace_marker
72 This makes it easier to see which part of the 72 This makes it easier to see which part of the (huge) trace corresponds to
73 which action. It is recommended to place descr 73 which action. It is recommended to place descriptive markers about what you
74 do. 74 do.
75 75
76 Shut down mmiotrace (requires root privileges) 76 Shut down mmiotrace (requires root privileges)::
77 77
78 $ echo nop > /sys/kernel/tracing/curre !! 78 $ echo nop > /sys/kernel/debug/tracing/current_tracer
79 79
80 The 'cat' process exits. If it does not, kill 80 The 'cat' process exits. If it does not, kill it by issuing 'fg' command and
81 pressing ctrl+c. 81 pressing ctrl+c.
82 82
83 Check that mmiotrace did not lose events due t 83 Check that mmiotrace did not lose events due to a buffer filling up. Either::
84 84
85 $ grep -i lost mydump.txt 85 $ grep -i lost mydump.txt
86 86
87 which tells you exactly how many events were l 87 which tells you exactly how many events were lost, or use::
88 88
89 $ dmesg 89 $ dmesg
90 90
91 to view your kernel log and look for "mmiotrac 91 to view your kernel log and look for "mmiotrace has lost events" warning. If
92 events were lost, the trace is incomplete. You 92 events were lost, the trace is incomplete. You should enlarge the buffers and
93 try again. Buffers are enlarged by first seein 93 try again. Buffers are enlarged by first seeing how large the current buffers
94 are:: 94 are::
95 95
96 $ cat /sys/kernel/tracing/buffer_size_ !! 96 $ cat /sys/kernel/debug/tracing/buffer_size_kb
97 97
98 gives you a number. Approximately double this 98 gives you a number. Approximately double this number and write it back, for
99 instance:: 99 instance::
100 100
101 $ echo 128000 > /sys/kernel/tracing/bu !! 101 $ echo 128000 > /sys/kernel/debug/tracing/buffer_size_kb
102 102
103 Then start again from the top. 103 Then start again from the top.
104 104
105 If you are doing a trace for a driver project, 105 If you are doing a trace for a driver project, e.g. Nouveau, you should also
106 do the following before sending your results:: 106 do the following before sending your results::
107 107
108 $ lspci -vvv > lspci.txt 108 $ lspci -vvv > lspci.txt
109 $ dmesg > dmesg.txt 109 $ dmesg > dmesg.txt
110 $ tar zcf pciid-nick-mmiotrace.tar.gz 110 $ tar zcf pciid-nick-mmiotrace.tar.gz mydump.txt lspci.txt dmesg.txt
111 111
112 and then send the .tar.gz file. The trace comp 112 and then send the .tar.gz file. The trace compresses considerably. Replace
113 "pciid" and "nick" with the PCI ID or model na 113 "pciid" and "nick" with the PCI ID or model name of your piece of hardware
114 under investigation and your nickname. 114 under investigation and your nickname.
115 115
116 116
117 How Mmiotrace Works 117 How Mmiotrace Works
118 ------------------- 118 -------------------
119 119
120 Access to hardware IO-memory is gained by mapp 120 Access to hardware IO-memory is gained by mapping addresses from PCI bus by
121 calling one of the ioremap_*() functions. Mmio 121 calling one of the ioremap_*() functions. Mmiotrace is hooked into the
122 __ioremap() function and gets called whenever 122 __ioremap() function and gets called whenever a mapping is created. Mapping is
123 an event that is recorded into the trace log. 123 an event that is recorded into the trace log. Note that ISA range mappings
124 are not caught, since the mapping always exist 124 are not caught, since the mapping always exists and is returned directly.
125 125
126 MMIO accesses are recorded via page faults. Ju 126 MMIO accesses are recorded via page faults. Just before __ioremap() returns,
127 the mapped pages are marked as not present. An 127 the mapped pages are marked as not present. Any access to the pages causes a
128 fault. The page fault handler calls mmiotrace 128 fault. The page fault handler calls mmiotrace to handle the fault. Mmiotrace
129 marks the page present, sets TF flag to achiev 129 marks the page present, sets TF flag to achieve single stepping and exits the
130 fault handler. The instruction that faulted is 130 fault handler. The instruction that faulted is executed and debug trap is
131 entered. Here mmiotrace again marks the page a 131 entered. Here mmiotrace again marks the page as not present. The instruction
132 is decoded to get the type of operation (read/ 132 is decoded to get the type of operation (read/write), data width and the value
133 read or written. These are stored to the trace 133 read or written. These are stored to the trace log.
134 134
135 Setting the page present in the page fault han 135 Setting the page present in the page fault handler has a race condition on SMP
136 machines. During the single stepping other CPU 136 machines. During the single stepping other CPUs may run freely on that page
137 and events can be missed without a notice. Re- 137 and events can be missed without a notice. Re-enabling other CPUs during
138 tracing is discouraged. 138 tracing is discouraged.
139 139
140 140
141 Trace Log Format 141 Trace Log Format
142 ---------------- 142 ----------------
143 143
144 The raw log is text and easily filtered with e 144 The raw log is text and easily filtered with e.g. grep and awk. One record is
145 one line in the log. A record starts with a ke 145 one line in the log. A record starts with a keyword, followed by keyword-
146 dependent arguments. Arguments are separated b 146 dependent arguments. Arguments are separated by a space, or continue until the
147 end of line. The format for version 20070824 i 147 end of line. The format for version 20070824 is as follows:
148 148
149 Explanation Keyword Space-separated argume 149 Explanation Keyword Space-separated arguments
150 ---------------------------------------------- 150 ---------------------------------------------------------------------------
151 151
152 read event R width, timestamp, map 152 read event R width, timestamp, map id, physical, value, PC, PID
153 write event W width, timestamp, map 153 write event W width, timestamp, map id, physical, value, PC, PID
154 ioremap event MAP timestamp, map id, phy 154 ioremap event MAP timestamp, map id, physical, virtual, length, PC, PID
155 iounmap event UNMAP timestamp, map id, PC, 155 iounmap event UNMAP timestamp, map id, PC, PID
156 marker MARK timestamp, text 156 marker MARK timestamp, text
157 version VERSION the string "20070824" 157 version VERSION the string "20070824"
158 info for reader LSPCI one line from lspci -v 158 info for reader LSPCI one line from lspci -v
159 PCI address map PCIDEV space-separated /proc/ 159 PCI address map PCIDEV space-separated /proc/bus/pci/devices data
160 unk. opcode UNKNOWN timestamp, map id, phy 160 unk. opcode UNKNOWN timestamp, map id, physical, data, PC, PID
161 161
162 Timestamp is in seconds with decimals. Physica 162 Timestamp is in seconds with decimals. Physical is a PCI bus address, virtual
163 is a kernel virtual address. Width is the data 163 is a kernel virtual address. Width is the data width in bytes and value is the
164 data value. Map id is an arbitrary id number i 164 data value. Map id is an arbitrary id number identifying the mapping that was
165 used in an operation. PC is the program counte 165 used in an operation. PC is the program counter and PID is process id. PC is
166 zero if it is not recorded. PID is always zero 166 zero if it is not recorded. PID is always zero as tracing MMIO accesses
167 originating in user space memory is not yet su 167 originating in user space memory is not yet supported.
168 168
169 For instance, the following awk filter will pa 169 For instance, the following awk filter will pass all 32-bit writes that target
170 physical addresses in the range [0xfb73ce40, 0 170 physical addresses in the range [0xfb73ce40, 0xfb800000]
171 :: 171 ::
172 172
173 $ awk '/W 4 / { adr=strtonum($5); if ( 173 $ awk '/W 4 / { adr=strtonum($5); if (adr >= 0xfb73ce40 &&
174 adr < 0xfb800000) print; }' 174 adr < 0xfb800000) print; }'
175 175
176 176
177 Tools for Developers 177 Tools for Developers
178 -------------------- 178 --------------------
179 179
180 The user space tools include utilities for: 180 The user space tools include utilities for:
181 - replacing numeric addresses and values wit 181 - replacing numeric addresses and values with hardware register names
182 - replaying MMIO logs, i.e., re-executing th 182 - replaying MMIO logs, i.e., re-executing the recorded writes
183 183
184 184
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.