1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ========== 3 ==========
4 Netconsole 4 Netconsole
5 ========== 5 ==========
6 6
7 7
8 started by Ingo Molnar <mingo@redhat.com>, 2001 8 started by Ingo Molnar <mingo@redhat.com>, 2001.09.17
9 9
10 2.6 port and netpoll api by Matt Mackall <mpm@s 10 2.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003
11 11
12 IPv6 support by Cong Wang <xiyou.wangcong@gmail 12 IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013
13 13
14 Extended console support by Tejun Heo <tj@kerne 14 Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015
15 15
16 Release prepend support by Breno Leitao <leitao 16 Release prepend support by Breno Leitao <leitao@debian.org>, Jul 7 2023
17 17
18 Userdata append support by Matthew Wood <thepac 18 Userdata append support by Matthew Wood <thepacketgeek@gmail.com>, Jan 22 2024
19 19
20 Please send bug reports to Matt Mackall <mpm@se 20 Please send bug reports to Matt Mackall <mpm@selenic.com>
21 Satyam Sharma <satyam.sharma@gmail.com>, and Co 21 Satyam Sharma <satyam.sharma@gmail.com>, and Cong Wang <xiyou.wangcong@gmail.com>
22 22
23 Introduction: 23 Introduction:
24 ============= 24 =============
25 25
26 This module logs kernel printk messages over U 26 This module logs kernel printk messages over UDP allowing debugging of
27 problem where disk logging fails and serial co 27 problem where disk logging fails and serial consoles are impractical.
28 28
29 It can be used either built-in or as a module. 29 It can be used either built-in or as a module. As a built-in,
30 netconsole initializes immediately after NIC c 30 netconsole initializes immediately after NIC cards and will bring up
31 the specified interface as soon as possible. W 31 the specified interface as soon as possible. While this doesn't allow
32 capture of early kernel panics, it does captur 32 capture of early kernel panics, it does capture most of the boot
33 process. 33 process.
34 34
35 Sender and receiver configuration: 35 Sender and receiver configuration:
36 ================================== 36 ==================================
37 37
38 It takes a string configuration parameter "net 38 It takes a string configuration parameter "netconsole" in the
39 following format:: 39 following format::
40 40
41 netconsole=[+][r][src-port]@[src-ip]/[<dev>],[ 41 netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
42 42
43 where 43 where
44 + if present, enable exten 44 + if present, enable extended console support
45 r if present, prepend kern 45 r if present, prepend kernel version (release) to the message
46 src-port source for UDP packets ( 46 src-port source for UDP packets (defaults to 6665)
47 src-ip source IP to use (interf 47 src-ip source IP to use (interface address)
48 dev network interface (eth0) 48 dev network interface (eth0)
49 tgt-port port for logging agent ( 49 tgt-port port for logging agent (6666)
50 tgt-ip IP address for logging a 50 tgt-ip IP address for logging agent
51 tgt-macaddr ethernet MAC address for 51 tgt-macaddr ethernet MAC address for logging agent (broadcast)
52 52
53 Examples:: 53 Examples::
54 54
55 linux netconsole=4444@10.0.0.1/eth1,9353@10.0 55 linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
56 56
57 or:: 57 or::
58 58
59 insmod netconsole netconsole=@/,@10.0.0.2/ 59 insmod netconsole netconsole=@/,@10.0.0.2/
60 60
61 or using IPv6:: 61 or using IPv6::
62 62
63 insmod netconsole netconsole=@/,@fd00:1:2:3:: 63 insmod netconsole netconsole=@/,@fd00:1:2:3::1/
64 64
65 It also supports logging to multiple remote ag 65 It also supports logging to multiple remote agents by specifying
66 parameters for the multiple agents separated b 66 parameters for the multiple agents separated by semicolons and the
67 complete string enclosed in "quotes", thusly:: 67 complete string enclosed in "quotes", thusly::
68 68
69 modprobe netconsole netconsole="@/,@10.0.0.2/ 69 modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/"
70 70
71 Built-in netconsole starts immediately after t 71 Built-in netconsole starts immediately after the TCP stack is
72 initialized and attempts to bring up the suppl 72 initialized and attempts to bring up the supplied dev at the supplied
73 address. 73 address.
74 74
75 The remote host has several options to receive 75 The remote host has several options to receive the kernel messages,
76 for example: 76 for example:
77 77
78 1) syslogd 78 1) syslogd
79 79
80 2) netcat 80 2) netcat
81 81
82 On distributions using a BSD-based netcat v 82 On distributions using a BSD-based netcat version (e.g. Fedora,
83 openSUSE and Ubuntu) the listening port mus 83 openSUSE and Ubuntu) the listening port must be specified without
84 the -p switch:: 84 the -p switch::
85 85
86 nc -u -l -p <port>' / 'nc -u -l <port> 86 nc -u -l -p <port>' / 'nc -u -l <port>
87 87
88 or:: 88 or::
89 89
90 netcat -u -l -p <port>' / 'netcat -u - 90 netcat -u -l -p <port>' / 'netcat -u -l <port>
91 91
92 3) socat 92 3) socat
93 93
94 :: 94 ::
95 95
96 socat udp-recv:<port> - 96 socat udp-recv:<port> -
97 97
98 Dynamic reconfiguration: 98 Dynamic reconfiguration:
99 ======================== 99 ========================
100 100
101 Dynamic reconfigurability is a useful addition 101 Dynamic reconfigurability is a useful addition to netconsole that enables
102 remote logging targets to be dynamically added 102 remote logging targets to be dynamically added, removed, or have their
103 parameters reconfigured at runtime from a conf 103 parameters reconfigured at runtime from a configfs-based userspace interface.
104 104
105 To include this feature, select CONFIG_NETCONS 105 To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the
106 netconsole module (or kernel, if netconsole is 106 netconsole module (or kernel, if netconsole is built-in).
107 107
108 Some examples follow (where configfs is mounte 108 Some examples follow (where configfs is mounted at the /sys/kernel/config
109 mountpoint). 109 mountpoint).
110 110
111 To add a remote logging target (target names c 111 To add a remote logging target (target names can be arbitrary)::
112 112
113 cd /sys/kernel/config/netconsole/ 113 cd /sys/kernel/config/netconsole/
114 mkdir target1 114 mkdir target1
115 115
116 Note that newly created targets have default p 116 Note that newly created targets have default parameter values (as mentioned
117 above) and are disabled by default -- they mus 117 above) and are disabled by default -- they must first be enabled by writing
118 "1" to the "enabled" attribute (usually after 118 "1" to the "enabled" attribute (usually after setting parameters accordingly)
119 as described below. 119 as described below.
120 120
121 To remove a target:: 121 To remove a target::
122 122
123 rmdir /sys/kernel/config/netconsole/othertarg 123 rmdir /sys/kernel/config/netconsole/othertarget/
124 124
125 The interface exposes these parameters of a ne 125 The interface exposes these parameters of a netconsole target to userspace:
126 126
127 ============== ====================== 127 ============== ================================= ============
128 enabled Is this target current 128 enabled Is this target currently enabled? (read-write)
129 extended Extended mode enabled 129 extended Extended mode enabled (read-write)
130 release Prepend kernel release 130 release Prepend kernel release to message (read-write)
131 dev_name Local network interfac 131 dev_name Local network interface name (read-write)
132 local_port Source UDP port to use 132 local_port Source UDP port to use (read-write)
133 remote_port Remote agent's UDP por 133 remote_port Remote agent's UDP port (read-write)
134 local_ip Source IP address to u 134 local_ip Source IP address to use (read-write)
135 remote_ip Remote agent's IP addr 135 remote_ip Remote agent's IP address (read-write)
136 local_mac Local interface's MAC 136 local_mac Local interface's MAC address (read-only)
137 remote_mac Remote agent's MAC add 137 remote_mac Remote agent's MAC address (read-write)
138 ============== ====================== 138 ============== ================================= ============
139 139
140 The "enabled" attribute is also used to contro 140 The "enabled" attribute is also used to control whether the parameters of
141 a target can be updated or not -- you can modi 141 a target can be updated or not -- you can modify the parameters of only
142 disabled targets (i.e. if "enabled" is 0). 142 disabled targets (i.e. if "enabled" is 0).
143 143
144 To update a target's parameters:: 144 To update a target's parameters::
145 145
146 cat enabled # chec 146 cat enabled # check if enabled is 1
147 echo 0 > enabled # disa 147 echo 0 > enabled # disable the target (if required)
148 echo eth2 > dev_name # set 148 echo eth2 > dev_name # set local interface
149 echo 10.0.0.4 > remote_ip # upda 149 echo 10.0.0.4 > remote_ip # update some parameter
150 echo cb:a9:87:65:43:21 > remote_mac # upda 150 echo cb:a9:87:65:43:21 > remote_mac # update more parameters
151 echo 1 > enabled # enab 151 echo 1 > enabled # enable target again
152 152
153 You can also update the local interface dynami 153 You can also update the local interface dynamically. This is especially
154 useful if you want to use interfaces that have 154 useful if you want to use interfaces that have newly come up (and may not
155 have existed when netconsole was loaded / init 155 have existed when netconsole was loaded / initialized).
156 156
157 Netconsole targets defined at boot time (or mo 157 Netconsole targets defined at boot time (or module load time) with the
158 `netconsole=` param are assigned the name `cmd 158 `netconsole=` param are assigned the name `cmdline<index>`. For example, the
159 first target in the parameter is named `cmdlin 159 first target in the parameter is named `cmdline0`. You can control and modify
160 these targets by creating configfs directories 160 these targets by creating configfs directories with the matching name.
161 161
162 Let's suppose you have two netconsole targets 162 Let's suppose you have two netconsole targets defined at boot time::
163 163
164 netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/1 164 netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc
165 165
166 You can modify these targets in runtime by cre 166 You can modify these targets in runtime by creating the following targets::
167 167
168 mkdir cmdline0 168 mkdir cmdline0
169 cat cmdline0/remote_ip 169 cat cmdline0/remote_ip
170 10.0.0.2 170 10.0.0.2
171 171
172 mkdir cmdline1 172 mkdir cmdline1
173 cat cmdline1/remote_ip 173 cat cmdline1/remote_ip
174 10.0.0.3 174 10.0.0.3
175 175
176 Append User Data 176 Append User Data
177 ---------------- 177 ----------------
178 178
179 Custom user data can be appended to the end of 179 Custom user data can be appended to the end of messages with netconsole
180 dynamic configuration enabled. User data entri 180 dynamic configuration enabled. User data entries can be modified without
181 changing the "enabled" attribute of a target. 181 changing the "enabled" attribute of a target.
182 182
183 Directories (keys) under `userdata` are limite 183 Directories (keys) under `userdata` are limited to 53 character length, and
184 data in `userdata/<key>/value` are limited to 184 data in `userdata/<key>/value` are limited to 200 bytes::
185 185
186 cd /sys/kernel/config/netconsole && mkdir cmd 186 cd /sys/kernel/config/netconsole && mkdir cmdline0
187 cd cmdline0 187 cd cmdline0
188 mkdir userdata/foo 188 mkdir userdata/foo
189 echo bar > userdata/foo/value 189 echo bar > userdata/foo/value
190 mkdir userdata/qux 190 mkdir userdata/qux
191 echo baz > userdata/qux/value 191 echo baz > userdata/qux/value
192 192
193 Messages will now include this additional user 193 Messages will now include this additional user data::
194 194
195 echo "This is a message" > /dev/kmsg 195 echo "This is a message" > /dev/kmsg
196 196
197 Sends:: 197 Sends::
198 198
199 12,607,22085407756,-;This is a message 199 12,607,22085407756,-;This is a message
200 foo=bar 200 foo=bar
201 qux=baz 201 qux=baz
202 202
203 Preview the userdata that will be appended wit 203 Preview the userdata that will be appended with::
204 204
205 cd /sys/kernel/config/netconsole/cmdline0/use 205 cd /sys/kernel/config/netconsole/cmdline0/userdata
206 for f in `ls userdata`; do echo $f=$(cat user 206 for f in `ls userdata`; do echo $f=$(cat userdata/$f/value); done
207 207
208 If a `userdata` entry is created but no data i 208 If a `userdata` entry is created but no data is written to the `value` file,
209 the entry will be omitted from netconsole mess 209 the entry will be omitted from netconsole messages::
210 210
211 cd /sys/kernel/config/netconsole && mkdir cmd 211 cd /sys/kernel/config/netconsole && mkdir cmdline0
212 cd cmdline0 212 cd cmdline0
213 mkdir userdata/foo 213 mkdir userdata/foo
214 echo bar > userdata/foo/value 214 echo bar > userdata/foo/value
215 mkdir userdata/qux 215 mkdir userdata/qux
216 216
217 The `qux` key is omitted since it has no value 217 The `qux` key is omitted since it has no value::
218 218
219 echo "This is a message" > /dev/kmsg 219 echo "This is a message" > /dev/kmsg
220 12,607,22085407756,-;This is a message 220 12,607,22085407756,-;This is a message
221 foo=bar 221 foo=bar
222 222
223 Delete `userdata` entries with `rmdir`:: 223 Delete `userdata` entries with `rmdir`::
224 224
225 rmdir /sys/kernel/config/netconsole/cmdline0/ 225 rmdir /sys/kernel/config/netconsole/cmdline0/userdata/qux
226 226
227 .. warning:: 227 .. warning::
228 When writing strings to user data values, i 228 When writing strings to user data values, input is broken up per line in
229 configfs store calls and this can cause con 229 configfs store calls and this can cause confusing behavior::
230 230
231 mkdir userdata/testing 231 mkdir userdata/testing
232 printf "val1\nval2" > userdata/testing/va 232 printf "val1\nval2" > userdata/testing/value
233 # userdata store value is called twice, f 233 # userdata store value is called twice, first with "val1\n" then "val2"
234 # so "val2" is stored, being the last val 234 # so "val2" is stored, being the last value stored
235 cat userdata/testing/value 235 cat userdata/testing/value
236 val2 236 val2
237 237
238 It is recommended to not write user data va 238 It is recommended to not write user data values with newlines.
239 239
240 Extended console: 240 Extended console:
241 ================= 241 =================
242 242
243 If '+' is prefixed to the configuration line o 243 If '+' is prefixed to the configuration line or "extended" config file
244 is set to 1, extended console support is enabl 244 is set to 1, extended console support is enabled. An example boot
245 param follows:: 245 param follows::
246 246
247 linux netconsole=+4444@10.0.0.1/eth1,9353@10. 247 linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
248 248
249 Log messages are transmitted with extended met 249 Log messages are transmitted with extended metadata header in the
250 following format which is the same as /dev/kms 250 following format which is the same as /dev/kmsg::
251 251
252 <level>,<sequnum>,<timestamp>,<contflag>;<mes 252 <level>,<sequnum>,<timestamp>,<contflag>;<message text>
253 253
254 If 'r' (release) feature is enabled, the kerne 254 If 'r' (release) feature is enabled, the kernel release version is
255 prepended to the start of the message. Example 255 prepended to the start of the message. Example::
256 256
257 6.4.0,6,444,501151268,-;netconsole: network l 257 6.4.0,6,444,501151268,-;netconsole: network logging started
258 258
259 Non printable characters in <message text> are 259 Non printable characters in <message text> are escaped using "\xff"
260 notation. If the message contains optional dic 260 notation. If the message contains optional dictionary, verbatim
261 newline is used as the delimiter. 261 newline is used as the delimiter.
262 262
263 If a message doesn't fit in certain number of 263 If a message doesn't fit in certain number of bytes (currently 1000),
264 the message is split into multiple fragments b 264 the message is split into multiple fragments by netconsole. These
265 fragments are transmitted with "ncfrag" header 265 fragments are transmitted with "ncfrag" header field added::
266 266
267 ncfrag=<byte-offset>/<total-bytes> 267 ncfrag=<byte-offset>/<total-bytes>
268 268
269 For example, assuming a lot smaller chunk size 269 For example, assuming a lot smaller chunk size, a message "the first
270 chunk, the 2nd chunk." may be split as follows 270 chunk, the 2nd chunk." may be split as follows::
271 271
272 6,416,1758426,-,ncfrag=0/31;the first chunk, 272 6,416,1758426,-,ncfrag=0/31;the first chunk,
273 6,416,1758426,-,ncfrag=16/31; the 2nd chunk. 273 6,416,1758426,-,ncfrag=16/31; the 2nd chunk.
274 274
275 Miscellaneous notes: 275 Miscellaneous notes:
276 ==================== 276 ====================
277 277
278 .. Warning:: 278 .. Warning::
279 279
280 the default target ethernet setting uses th 280 the default target ethernet setting uses the broadcast
281 ethernet address to send packets, which can 281 ethernet address to send packets, which can cause increased load on
282 other systems on the same ethernet segment. 282 other systems on the same ethernet segment.
283 283
284 .. Tip:: 284 .. Tip::
285 285
286 some LAN switches may be configured to supp 286 some LAN switches may be configured to suppress ethernet broadcasts
287 so it is advised to explicitly specify the 287 so it is advised to explicitly specify the remote agents' MAC addresses
288 from the config parameters passed to netcon 288 from the config parameters passed to netconsole.
289 289
290 .. Tip:: 290 .. Tip::
291 291
292 to find out the MAC address of, say, 10.0.0 292 to find out the MAC address of, say, 10.0.0.2, you may try using::
293 293
294 ping -c 1 10.0.0.2 ; /sbin/arp -n | gr 294 ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2
295 295
296 .. Tip:: 296 .. Tip::
297 297
298 in case the remote logging agent is on a se 298 in case the remote logging agent is on a separate LAN subnet than
299 the sender, it is suggested to try specifyi 299 the sender, it is suggested to try specifying the MAC address of the
300 default gateway (you may use /sbin/route -n 300 default gateway (you may use /sbin/route -n to find it out) as the
301 remote MAC address instead. 301 remote MAC address instead.
302 302
303 .. note:: 303 .. note::
304 304
305 the network device (eth1 in the above case) 305 the network device (eth1 in the above case) can run any kind
306 of other network traffic, netconsole is not 306 of other network traffic, netconsole is not intrusive. Netconsole
307 might cause slight delays in other traffic 307 might cause slight delays in other traffic if the volume of kernel
308 messages is high, but should have no other 308 messages is high, but should have no other impact.
309 309
310 .. note:: 310 .. note::
311 311
312 if you find that the remote logging agent i 312 if you find that the remote logging agent is not receiving or
313 printing all messages from the sender, it i 313 printing all messages from the sender, it is likely that you have set
314 the "console_loglevel" parameter (on the se 314 the "console_loglevel" parameter (on the sender) to only send high
315 priority messages to the console. You can c 315 priority messages to the console. You can change this at runtime using::
316 316
317 dmesg -n 8 317 dmesg -n 8
318 318
319 or by specifying "debug" on the kernel comm 319 or by specifying "debug" on the kernel command line at boot, to send
320 all kernel messages to the console. A speci 320 all kernel messages to the console. A specific value for this parameter
321 can also be set using the "loglevel" kernel 321 can also be set using the "loglevel" kernel boot option. See the
322 dmesg(8) man page and Documentation/admin-g 322 dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst
323 for details. 323 for details.
324 324
325 Netconsole was designed to be as instantaneous 325 Netconsole was designed to be as instantaneous as possible, to
326 enable the logging of even the most critical k 326 enable the logging of even the most critical kernel bugs. It works
327 from IRQ contexts as well, and does not enable 327 from IRQ contexts as well, and does not enable interrupts while
328 sending packets. Due to these unique needs, co 328 sending packets. Due to these unique needs, configuration cannot
329 be more automatic, and some fundamental limita 329 be more automatic, and some fundamental limitations will remain:
330 only IP networks, UDP packets and ethernet dev 330 only IP networks, UDP packets and ethernet devices are supported.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.