1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ========== 3 ==========
4 Netconsole 4 Netconsole
5 ========== 5 ==========
6 6
7 7
8 started by Ingo Molnar <mingo@redhat.com>, 2001 8 started by Ingo Molnar <mingo@redhat.com>, 2001.09.17
9 9
10 2.6 port and netpoll api by Matt Mackall <mpm@s 10 2.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003
11 11
12 IPv6 support by Cong Wang <xiyou.wangcong@gmail 12 IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013
13 13
14 Extended console support by Tejun Heo <tj@kerne 14 Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015
15 15
16 Release prepend support by Breno Leitao <leitao 16 Release prepend support by Breno Leitao <leitao@debian.org>, Jul 7 2023
17 17
18 Userdata append support by Matthew Wood <thepac <<
19 <<
20 Please send bug reports to Matt Mackall <mpm@se 18 Please send bug reports to Matt Mackall <mpm@selenic.com>
21 Satyam Sharma <satyam.sharma@gmail.com>, and Co 19 Satyam Sharma <satyam.sharma@gmail.com>, and Cong Wang <xiyou.wangcong@gmail.com>
22 20
23 Introduction: 21 Introduction:
24 ============= 22 =============
25 23
26 This module logs kernel printk messages over U 24 This module logs kernel printk messages over UDP allowing debugging of
27 problem where disk logging fails and serial co 25 problem where disk logging fails and serial consoles are impractical.
28 26
29 It can be used either built-in or as a module. 27 It can be used either built-in or as a module. As a built-in,
30 netconsole initializes immediately after NIC c 28 netconsole initializes immediately after NIC cards and will bring up
31 the specified interface as soon as possible. W 29 the specified interface as soon as possible. While this doesn't allow
32 capture of early kernel panics, it does captur 30 capture of early kernel panics, it does capture most of the boot
33 process. 31 process.
34 32
35 Sender and receiver configuration: 33 Sender and receiver configuration:
36 ================================== 34 ==================================
37 35
38 It takes a string configuration parameter "net 36 It takes a string configuration parameter "netconsole" in the
39 following format:: 37 following format::
40 38
41 netconsole=[+][r][src-port]@[src-ip]/[<dev>],[ 39 netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
42 40
43 where 41 where
44 + if present, enable exten 42 + if present, enable extended console support
45 r if present, prepend kern 43 r if present, prepend kernel version (release) to the message
46 src-port source for UDP packets ( 44 src-port source for UDP packets (defaults to 6665)
47 src-ip source IP to use (interf 45 src-ip source IP to use (interface address)
48 dev network interface (eth0) 46 dev network interface (eth0)
49 tgt-port port for logging agent ( 47 tgt-port port for logging agent (6666)
50 tgt-ip IP address for logging a 48 tgt-ip IP address for logging agent
51 tgt-macaddr ethernet MAC address for 49 tgt-macaddr ethernet MAC address for logging agent (broadcast)
52 50
53 Examples:: 51 Examples::
54 52
55 linux netconsole=4444@10.0.0.1/eth1,9353@10.0 53 linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
56 54
57 or:: 55 or::
58 56
59 insmod netconsole netconsole=@/,@10.0.0.2/ 57 insmod netconsole netconsole=@/,@10.0.0.2/
60 58
61 or using IPv6:: 59 or using IPv6::
62 60
63 insmod netconsole netconsole=@/,@fd00:1:2:3:: 61 insmod netconsole netconsole=@/,@fd00:1:2:3::1/
64 62
65 It also supports logging to multiple remote ag 63 It also supports logging to multiple remote agents by specifying
66 parameters for the multiple agents separated b 64 parameters for the multiple agents separated by semicolons and the
67 complete string enclosed in "quotes", thusly:: 65 complete string enclosed in "quotes", thusly::
68 66
69 modprobe netconsole netconsole="@/,@10.0.0.2/ 67 modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/"
70 68
71 Built-in netconsole starts immediately after t 69 Built-in netconsole starts immediately after the TCP stack is
72 initialized and attempts to bring up the suppl 70 initialized and attempts to bring up the supplied dev at the supplied
73 address. 71 address.
74 72
75 The remote host has several options to receive 73 The remote host has several options to receive the kernel messages,
76 for example: 74 for example:
77 75
78 1) syslogd 76 1) syslogd
79 77
80 2) netcat 78 2) netcat
81 79
82 On distributions using a BSD-based netcat v 80 On distributions using a BSD-based netcat version (e.g. Fedora,
83 openSUSE and Ubuntu) the listening port mus 81 openSUSE and Ubuntu) the listening port must be specified without
84 the -p switch:: 82 the -p switch::
85 83
86 nc -u -l -p <port>' / 'nc -u -l <port> 84 nc -u -l -p <port>' / 'nc -u -l <port>
87 85
88 or:: 86 or::
89 87
90 netcat -u -l -p <port>' / 'netcat -u - 88 netcat -u -l -p <port>' / 'netcat -u -l <port>
91 89
92 3) socat 90 3) socat
93 91
94 :: 92 ::
95 93
96 socat udp-recv:<port> - 94 socat udp-recv:<port> -
97 95
98 Dynamic reconfiguration: 96 Dynamic reconfiguration:
99 ======================== 97 ========================
100 98
101 Dynamic reconfigurability is a useful addition 99 Dynamic reconfigurability is a useful addition to netconsole that enables
102 remote logging targets to be dynamically added 100 remote logging targets to be dynamically added, removed, or have their
103 parameters reconfigured at runtime from a conf 101 parameters reconfigured at runtime from a configfs-based userspace interface.
104 102
105 To include this feature, select CONFIG_NETCONS 103 To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the
106 netconsole module (or kernel, if netconsole is 104 netconsole module (or kernel, if netconsole is built-in).
107 105
108 Some examples follow (where configfs is mounte 106 Some examples follow (where configfs is mounted at the /sys/kernel/config
109 mountpoint). 107 mountpoint).
110 108
111 To add a remote logging target (target names c 109 To add a remote logging target (target names can be arbitrary)::
112 110
113 cd /sys/kernel/config/netconsole/ 111 cd /sys/kernel/config/netconsole/
114 mkdir target1 112 mkdir target1
115 113
116 Note that newly created targets have default p 114 Note that newly created targets have default parameter values (as mentioned
117 above) and are disabled by default -- they mus 115 above) and are disabled by default -- they must first be enabled by writing
118 "1" to the "enabled" attribute (usually after 116 "1" to the "enabled" attribute (usually after setting parameters accordingly)
119 as described below. 117 as described below.
120 118
121 To remove a target:: 119 To remove a target::
122 120
123 rmdir /sys/kernel/config/netconsole/othertarg 121 rmdir /sys/kernel/config/netconsole/othertarget/
124 122
125 The interface exposes these parameters of a ne 123 The interface exposes these parameters of a netconsole target to userspace:
126 124
127 ============== ====================== 125 ============== ================================= ============
128 enabled Is this target current 126 enabled Is this target currently enabled? (read-write)
129 extended Extended mode enabled 127 extended Extended mode enabled (read-write)
130 release Prepend kernel release 128 release Prepend kernel release to message (read-write)
131 dev_name Local network interfac 129 dev_name Local network interface name (read-write)
132 local_port Source UDP port to use 130 local_port Source UDP port to use (read-write)
133 remote_port Remote agent's UDP por 131 remote_port Remote agent's UDP port (read-write)
134 local_ip Source IP address to u 132 local_ip Source IP address to use (read-write)
135 remote_ip Remote agent's IP addr 133 remote_ip Remote agent's IP address (read-write)
136 local_mac Local interface's MAC 134 local_mac Local interface's MAC address (read-only)
137 remote_mac Remote agent's MAC add 135 remote_mac Remote agent's MAC address (read-write)
138 ============== ====================== 136 ============== ================================= ============
139 137
140 The "enabled" attribute is also used to contro 138 The "enabled" attribute is also used to control whether the parameters of
141 a target can be updated or not -- you can modi 139 a target can be updated or not -- you can modify the parameters of only
142 disabled targets (i.e. if "enabled" is 0). 140 disabled targets (i.e. if "enabled" is 0).
143 141
144 To update a target's parameters:: 142 To update a target's parameters::
145 143
146 cat enabled # chec 144 cat enabled # check if enabled is 1
147 echo 0 > enabled # disa 145 echo 0 > enabled # disable the target (if required)
148 echo eth2 > dev_name # set 146 echo eth2 > dev_name # set local interface
149 echo 10.0.0.4 > remote_ip # upda 147 echo 10.0.0.4 > remote_ip # update some parameter
150 echo cb:a9:87:65:43:21 > remote_mac # upda 148 echo cb:a9:87:65:43:21 > remote_mac # update more parameters
151 echo 1 > enabled # enab 149 echo 1 > enabled # enable target again
152 150
153 You can also update the local interface dynami 151 You can also update the local interface dynamically. This is especially
154 useful if you want to use interfaces that have 152 useful if you want to use interfaces that have newly come up (and may not
155 have existed when netconsole was loaded / init 153 have existed when netconsole was loaded / initialized).
156 154
157 Netconsole targets defined at boot time (or mo 155 Netconsole targets defined at boot time (or module load time) with the
158 `netconsole=` param are assigned the name `cmd 156 `netconsole=` param are assigned the name `cmdline<index>`. For example, the
159 first target in the parameter is named `cmdlin 157 first target in the parameter is named `cmdline0`. You can control and modify
160 these targets by creating configfs directories 158 these targets by creating configfs directories with the matching name.
161 159
162 Let's suppose you have two netconsole targets 160 Let's suppose you have two netconsole targets defined at boot time::
163 161
164 netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/1 162 netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc
165 163
166 You can modify these targets in runtime by cre 164 You can modify these targets in runtime by creating the following targets::
167 165
168 mkdir cmdline0 166 mkdir cmdline0
169 cat cmdline0/remote_ip 167 cat cmdline0/remote_ip
170 10.0.0.2 168 10.0.0.2
171 169
172 mkdir cmdline1 170 mkdir cmdline1
173 cat cmdline1/remote_ip 171 cat cmdline1/remote_ip
174 10.0.0.3 172 10.0.0.3
175 <<
176 Append User Data <<
177 ---------------- <<
178 <<
179 Custom user data can be appended to the end of <<
180 dynamic configuration enabled. User data entri <<
181 changing the "enabled" attribute of a target. <<
182 <<
183 Directories (keys) under `userdata` are limite <<
184 data in `userdata/<key>/value` are limited to <<
185 <<
186 cd /sys/kernel/config/netconsole && mkdir cmd <<
187 cd cmdline0 <<
188 mkdir userdata/foo <<
189 echo bar > userdata/foo/value <<
190 mkdir userdata/qux <<
191 echo baz > userdata/qux/value <<
192 <<
193 Messages will now include this additional user <<
194 <<
195 echo "This is a message" > /dev/kmsg <<
196 <<
197 Sends:: <<
198 <<
199 12,607,22085407756,-;This is a message <<
200 foo=bar <<
201 qux=baz <<
202 <<
203 Preview the userdata that will be appended wit <<
204 <<
205 cd /sys/kernel/config/netconsole/cmdline0/use <<
206 for f in `ls userdata`; do echo $f=$(cat user <<
207 <<
208 If a `userdata` entry is created but no data i <<
209 the entry will be omitted from netconsole mess <<
210 <<
211 cd /sys/kernel/config/netconsole && mkdir cmd <<
212 cd cmdline0 <<
213 mkdir userdata/foo <<
214 echo bar > userdata/foo/value <<
215 mkdir userdata/qux <<
216 <<
217 The `qux` key is omitted since it has no value <<
218 <<
219 echo "This is a message" > /dev/kmsg <<
220 12,607,22085407756,-;This is a message <<
221 foo=bar <<
222 <<
223 Delete `userdata` entries with `rmdir`:: <<
224 <<
225 rmdir /sys/kernel/config/netconsole/cmdline0/ <<
226 <<
227 .. warning:: <<
228 When writing strings to user data values, i <<
229 configfs store calls and this can cause con <<
230 <<
231 mkdir userdata/testing <<
232 printf "val1\nval2" > userdata/testing/va <<
233 # userdata store value is called twice, f <<
234 # so "val2" is stored, being the last val <<
235 cat userdata/testing/value <<
236 val2 <<
237 <<
238 It is recommended to not write user data va <<
239 173
240 Extended console: 174 Extended console:
241 ================= 175 =================
242 176
243 If '+' is prefixed to the configuration line o 177 If '+' is prefixed to the configuration line or "extended" config file
244 is set to 1, extended console support is enabl 178 is set to 1, extended console support is enabled. An example boot
245 param follows:: 179 param follows::
246 180
247 linux netconsole=+4444@10.0.0.1/eth1,9353@10. 181 linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
248 182
249 Log messages are transmitted with extended met 183 Log messages are transmitted with extended metadata header in the
250 following format which is the same as /dev/kms 184 following format which is the same as /dev/kmsg::
251 185
252 <level>,<sequnum>,<timestamp>,<contflag>;<mes 186 <level>,<sequnum>,<timestamp>,<contflag>;<message text>
253 187
254 If 'r' (release) feature is enabled, the kerne 188 If 'r' (release) feature is enabled, the kernel release version is
255 prepended to the start of the message. Example 189 prepended to the start of the message. Example::
256 190
257 6.4.0,6,444,501151268,-;netconsole: network l 191 6.4.0,6,444,501151268,-;netconsole: network logging started
258 192
259 Non printable characters in <message text> are 193 Non printable characters in <message text> are escaped using "\xff"
260 notation. If the message contains optional dic 194 notation. If the message contains optional dictionary, verbatim
261 newline is used as the delimiter. 195 newline is used as the delimiter.
262 196
263 If a message doesn't fit in certain number of 197 If a message doesn't fit in certain number of bytes (currently 1000),
264 the message is split into multiple fragments b 198 the message is split into multiple fragments by netconsole. These
265 fragments are transmitted with "ncfrag" header 199 fragments are transmitted with "ncfrag" header field added::
266 200
267 ncfrag=<byte-offset>/<total-bytes> 201 ncfrag=<byte-offset>/<total-bytes>
268 202
269 For example, assuming a lot smaller chunk size 203 For example, assuming a lot smaller chunk size, a message "the first
270 chunk, the 2nd chunk." may be split as follows 204 chunk, the 2nd chunk." may be split as follows::
271 205
272 6,416,1758426,-,ncfrag=0/31;the first chunk, 206 6,416,1758426,-,ncfrag=0/31;the first chunk,
273 6,416,1758426,-,ncfrag=16/31; the 2nd chunk. 207 6,416,1758426,-,ncfrag=16/31; the 2nd chunk.
274 208
275 Miscellaneous notes: 209 Miscellaneous notes:
276 ==================== 210 ====================
277 211
278 .. Warning:: 212 .. Warning::
279 213
280 the default target ethernet setting uses th 214 the default target ethernet setting uses the broadcast
281 ethernet address to send packets, which can 215 ethernet address to send packets, which can cause increased load on
282 other systems on the same ethernet segment. 216 other systems on the same ethernet segment.
283 217
284 .. Tip:: 218 .. Tip::
285 219
286 some LAN switches may be configured to supp 220 some LAN switches may be configured to suppress ethernet broadcasts
287 so it is advised to explicitly specify the 221 so it is advised to explicitly specify the remote agents' MAC addresses
288 from the config parameters passed to netcon 222 from the config parameters passed to netconsole.
289 223
290 .. Tip:: 224 .. Tip::
291 225
292 to find out the MAC address of, say, 10.0.0 226 to find out the MAC address of, say, 10.0.0.2, you may try using::
293 227
294 ping -c 1 10.0.0.2 ; /sbin/arp -n | gr 228 ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2
295 229
296 .. Tip:: 230 .. Tip::
297 231
298 in case the remote logging agent is on a se 232 in case the remote logging agent is on a separate LAN subnet than
299 the sender, it is suggested to try specifyi 233 the sender, it is suggested to try specifying the MAC address of the
300 default gateway (you may use /sbin/route -n 234 default gateway (you may use /sbin/route -n to find it out) as the
301 remote MAC address instead. 235 remote MAC address instead.
302 236
303 .. note:: 237 .. note::
304 238
305 the network device (eth1 in the above case) 239 the network device (eth1 in the above case) can run any kind
306 of other network traffic, netconsole is not 240 of other network traffic, netconsole is not intrusive. Netconsole
307 might cause slight delays in other traffic 241 might cause slight delays in other traffic if the volume of kernel
308 messages is high, but should have no other 242 messages is high, but should have no other impact.
309 243
310 .. note:: 244 .. note::
311 245
312 if you find that the remote logging agent i 246 if you find that the remote logging agent is not receiving or
313 printing all messages from the sender, it i 247 printing all messages from the sender, it is likely that you have set
314 the "console_loglevel" parameter (on the se 248 the "console_loglevel" parameter (on the sender) to only send high
315 priority messages to the console. You can c 249 priority messages to the console. You can change this at runtime using::
316 250
317 dmesg -n 8 251 dmesg -n 8
318 252
319 or by specifying "debug" on the kernel comm 253 or by specifying "debug" on the kernel command line at boot, to send
320 all kernel messages to the console. A speci 254 all kernel messages to the console. A specific value for this parameter
321 can also be set using the "loglevel" kernel 255 can also be set using the "loglevel" kernel boot option. See the
322 dmesg(8) man page and Documentation/admin-g 256 dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst
323 for details. 257 for details.
324 258
325 Netconsole was designed to be as instantaneous 259 Netconsole was designed to be as instantaneous as possible, to
326 enable the logging of even the most critical k 260 enable the logging of even the most critical kernel bugs. It works
327 from IRQ contexts as well, and does not enable 261 from IRQ contexts as well, and does not enable interrupts while
328 sending packets. Due to these unique needs, co 262 sending packets. Due to these unique needs, configuration cannot
329 be more automatic, and some fundamental limita 263 be more automatic, and some fundamental limitations will remain:
330 only IP networks, UDP packets and ethernet dev 264 only IP networks, UDP packets and ethernet devices are supported.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.