1 ============ 1 ============
2 SNMP counter 2 SNMP counter
3 ============ 3 ============
4 4
5 This document explains the meaning of SNMP cou 5 This document explains the meaning of SNMP counters.
6 6
7 General IPv4 counters 7 General IPv4 counters
8 ===================== 8 =====================
9 All layer 4 packets and ICMP packets will chan 9 All layer 4 packets and ICMP packets will change these counters, but
10 these counters won't be changed by layer 2 pac 10 these counters won't be changed by layer 2 packets (such as STP) or
11 ARP packets. 11 ARP packets.
12 12
13 * IpInReceives 13 * IpInReceives
14 14
15 Defined in `RFC1213 ipInReceives`_ 15 Defined in `RFC1213 ipInReceives`_
16 16
17 .. _RFC1213 ipInReceives: https://tools.ietf.o 17 .. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
18 18
19 The number of packets received by the IP layer 19 The number of packets received by the IP layer. It gets increasing at the
20 beginning of ip_rcv function, always be update 20 beginning of ip_rcv function, always be updated together with
21 IpExtInOctets. It will be increased even if th 21 IpExtInOctets. It will be increased even if the packet is dropped
22 later (e.g. due to the IP header is invalid or 22 later (e.g. due to the IP header is invalid or the checksum is wrong
23 and so on). It indicates the number of aggreg 23 and so on). It indicates the number of aggregated segments after
24 GRO/LRO. 24 GRO/LRO.
25 25
26 * IpInDelivers 26 * IpInDelivers
27 27
28 Defined in `RFC1213 ipInDelivers`_ 28 Defined in `RFC1213 ipInDelivers`_
29 29
30 .. _RFC1213 ipInDelivers: https://tools.ietf.o 30 .. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
31 31
32 The number of packets delivers to the upper la 32 The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
33 ICMP and so on. If no one listens on a raw soc 33 ICMP and so on. If no one listens on a raw socket, only kernel
34 supported protocols will be delivered, if some 34 supported protocols will be delivered, if someone listens on the raw
35 socket, all valid IP packets will be delivered 35 socket, all valid IP packets will be delivered.
36 36
37 * IpOutRequests 37 * IpOutRequests
38 38
39 Defined in `RFC1213 ipOutRequests`_ 39 Defined in `RFC1213 ipOutRequests`_
40 40
41 .. _RFC1213 ipOutRequests: https://tools.ietf. 41 .. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
42 42
43 The number of packets sent via IP layer, for b 43 The number of packets sent via IP layer, for both single cast and
44 multicast packets, and would always be updated 44 multicast packets, and would always be updated together with
45 IpExtOutOctets. 45 IpExtOutOctets.
46 46
47 * IpExtInOctets and IpExtOutOctets 47 * IpExtInOctets and IpExtOutOctets
48 48
49 They are Linux kernel extensions, no RFC defin 49 They are Linux kernel extensions, no RFC definitions. Please note,
50 RFC1213 indeed defines ifInOctets and ifOutOc 50 RFC1213 indeed defines ifInOctets and ifOutOctets, but they
51 are different things. The ifInOctets and ifOut 51 are different things. The ifInOctets and ifOutOctets include the MAC
52 layer header size but IpExtInOctets and IpExtO 52 layer header size but IpExtInOctets and IpExtOutOctets don't, they
53 only include the IP layer header and the IP la 53 only include the IP layer header and the IP layer data.
54 54
55 * IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInEC 55 * IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts
56 56
57 They indicate the number of four kinds of ECN 57 They indicate the number of four kinds of ECN IP packets, please refer
58 `Explicit Congestion Notification`_ for more d 58 `Explicit Congestion Notification`_ for more details.
59 59
60 .. _Explicit Congestion Notification: https:// 60 .. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
61 61
62 These 4 counters calculate how many packets re 62 These 4 counters calculate how many packets received per ECN
63 status. They count the real frame number regar 63 status. They count the real frame number regardless the LRO/GRO. So
64 for the same packet, you might find that IpInR 64 for the same packet, you might find that IpInReceives count 1, but
65 IpExtInNoECTPkts counts 2 or more. 65 IpExtInNoECTPkts counts 2 or more.
66 66
67 * IpInHdrErrors 67 * IpInHdrErrors
68 68
69 Defined in `RFC1213 ipInHdrErrors`_. It indica 69 Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is
70 dropped due to the IP header error. It might h 70 dropped due to the IP header error. It might happen in both IP input
71 and IP forward paths. 71 and IP forward paths.
72 72
73 .. _RFC1213 ipInHdrErrors: https://tools.ietf. 73 .. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
74 74
75 * IpInAddrErrors 75 * IpInAddrErrors
76 76
77 Defined in `RFC1213 ipInAddrErrors`_. It will 77 Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two
78 scenarios: (1) The IP address is invalid. (2) 78 scenarios: (1) The IP address is invalid. (2) The destination IP
79 address is not a local address and IP forwardi 79 address is not a local address and IP forwarding is not enabled
80 80
81 .. _RFC1213 ipInAddrErrors: https://tools.ietf 81 .. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
82 82
83 * IpExtInNoRoutes 83 * IpExtInNoRoutes
84 84
85 This counter means the packet is dropped when 85 This counter means the packet is dropped when the IP stack receives a
86 packet and can't find a route for it from the 86 packet and can't find a route for it from the route table. It might
87 happen when IP forwarding is enabled and the d 87 happen when IP forwarding is enabled and the destination IP address is
88 not a local address and there is no route for 88 not a local address and there is no route for the destination IP
89 address. 89 address.
90 90
91 * IpInUnknownProtos 91 * IpInUnknownProtos
92 92
93 Defined in `RFC1213 ipInUnknownProtos`_. It wi 93 Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the
94 layer 4 protocol is unsupported by kernel. If 94 layer 4 protocol is unsupported by kernel. If an application is using
95 raw socket, kernel will always deliver the pac 95 raw socket, kernel will always deliver the packet to the raw socket
96 and this counter won't be increased. 96 and this counter won't be increased.
97 97
98 .. _RFC1213 ipInUnknownProtos: https://tools.i 98 .. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
99 99
100 * IpExtInTruncatedPkts 100 * IpExtInTruncatedPkts
101 101
102 For IPv4 packet, it means the actual data size 102 For IPv4 packet, it means the actual data size is smaller than the
103 "Total Length" field in the IPv4 header. 103 "Total Length" field in the IPv4 header.
104 104
105 * IpInDiscards 105 * IpInDiscards
106 106
107 Defined in `RFC1213 ipInDiscards`_. It indicat 107 Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped
108 in the IP receiving path and due to kernel int 108 in the IP receiving path and due to kernel internal reasons (e.g. no
109 enough memory). 109 enough memory).
110 110
111 .. _RFC1213 ipInDiscards: https://tools.ietf.o 111 .. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
112 112
113 * IpOutDiscards 113 * IpOutDiscards
114 114
115 Defined in `RFC1213 ipOutDiscards`_. It indica 115 Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is
116 dropped in the IP sending path and due to kern 116 dropped in the IP sending path and due to kernel internal reasons.
117 117
118 .. _RFC1213 ipOutDiscards: https://tools.ietf. 118 .. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
119 119
120 * IpOutNoRoutes 120 * IpOutNoRoutes
121 121
122 Defined in `RFC1213 ipOutNoRoutes`_. It indica 122 Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is
123 dropped in the IP sending path and no route is 123 dropped in the IP sending path and no route is found for it.
124 124
125 .. _RFC1213 ipOutNoRoutes: https://tools.ietf. 125 .. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
126 126
127 ICMP counters 127 ICMP counters
128 ============= 128 =============
129 * IcmpInMsgs and IcmpOutMsgs 129 * IcmpInMsgs and IcmpOutMsgs
130 130
131 Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 131 Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_
132 132
133 .. _RFC1213 icmpInMsgs: https://tools.ietf.org 133 .. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
134 .. _RFC1213 icmpOutMsgs: https://tools.ietf.or 134 .. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
135 135
136 As mentioned in the RFC1213, these two counter 136 As mentioned in the RFC1213, these two counters include errors, they
137 would be increased even if the ICMP packet has 137 would be increased even if the ICMP packet has an invalid type. The
138 ICMP output path will check the header of a ra 138 ICMP output path will check the header of a raw socket, so the
139 IcmpOutMsgs would still be updated if the IP h 139 IcmpOutMsgs would still be updated if the IP header is constructed by
140 a userspace program. 140 a userspace program.
141 141
142 * ICMP named types 142 * ICMP named types
143 143
144 | These counters include most of common ICMP t 144 | These counters include most of common ICMP types, they are:
145 | IcmpInDestUnreachs: `RFC1213 icmpInDestUnrea 145 | IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
146 | IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_ 146 | IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
147 | IcmpInParmProbs: `RFC1213 icmpInParmProbs`_ 147 | IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
148 | IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs` 148 | IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
149 | IcmpInRedirects: `RFC1213 icmpInRedirects`_ 149 | IcmpInRedirects: `RFC1213 icmpInRedirects`_
150 | IcmpInEchos: `RFC1213 icmpInEchos`_ 150 | IcmpInEchos: `RFC1213 icmpInEchos`_
151 | IcmpInEchoReps: `RFC1213 icmpInEchoReps`_ 151 | IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
152 | IcmpInTimestamps: `RFC1213 icmpInTimestamps` 152 | IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
153 | IcmpInTimestampReps: `RFC1213 icmpInTimestam 153 | IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
154 | IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_ 154 | IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
155 | IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskR 155 | IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
156 | IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnr 156 | IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
157 | IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds` 157 | IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
158 | IcmpOutParmProbs: `RFC1213 icmpOutParmProbs` 158 | IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
159 | IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuench 159 | IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
160 | IcmpOutRedirects: `RFC1213 icmpOutRedirects` 160 | IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
161 | IcmpOutEchos: `RFC1213 icmpOutEchos`_ 161 | IcmpOutEchos: `RFC1213 icmpOutEchos`_
162 | IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_ 162 | IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
163 | IcmpOutTimestamps: `RFC1213 icmpOutTimestamp 163 | IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
164 | IcmpOutTimestampReps: `RFC1213 icmpOutTimest 164 | IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
165 | IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks` 165 | IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
166 | IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMas 166 | IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_
167 167
168 .. _RFC1213 icmpInDestUnreachs: https://tools. 168 .. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
169 .. _RFC1213 icmpInTimeExcds: https://tools.iet 169 .. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
170 .. _RFC1213 icmpInParmProbs: https://tools.iet 170 .. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
171 .. _RFC1213 icmpInSrcQuenchs: https://tools.ie 171 .. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
172 .. _RFC1213 icmpInRedirects: https://tools.iet 172 .. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
173 .. _RFC1213 icmpInEchos: https://tools.ietf.or 173 .. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
174 .. _RFC1213 icmpInEchoReps: https://tools.ietf 174 .. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
175 .. _RFC1213 icmpInTimestamps: https://tools.ie 175 .. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
176 .. _RFC1213 icmpInTimestampReps: https://tools 176 .. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
177 .. _RFC1213 icmpInAddrMasks: https://tools.iet 177 .. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
178 .. _RFC1213 icmpInAddrMaskReps: https://tools. 178 .. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
179 179
180 .. _RFC1213 icmpOutDestUnreachs: https://tools 180 .. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
181 .. _RFC1213 icmpOutTimeExcds: https://tools.ie 181 .. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
182 .. _RFC1213 icmpOutParmProbs: https://tools.ie 182 .. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
183 .. _RFC1213 icmpOutSrcQuenchs: https://tools.i 183 .. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
184 .. _RFC1213 icmpOutRedirects: https://tools.ie 184 .. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
185 .. _RFC1213 icmpOutEchos: https://tools.ietf.o 185 .. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
186 .. _RFC1213 icmpOutEchoReps: https://tools.iet 186 .. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
187 .. _RFC1213 icmpOutTimestamps: https://tools.i 187 .. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
188 .. _RFC1213 icmpOutTimestampReps: https://tool 188 .. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
189 .. _RFC1213 icmpOutAddrMasks: https://tools.ie 189 .. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
190 .. _RFC1213 icmpOutAddrMaskReps: https://tools 190 .. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
191 191
192 Every ICMP type has two counters: 'In' and 'Ou 192 Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
193 Echo packet, they are IcmpInEchos and IcmpOutE 193 Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
194 straightforward. The 'In' counter means kernel 194 straightforward. The 'In' counter means kernel receives such a packet
195 and the 'Out' counter means kernel sends such 195 and the 'Out' counter means kernel sends such a packet.
196 196
197 * ICMP numeric types 197 * ICMP numeric types
198 198
199 They are IcmpMsgInType[N] and IcmpMsgOutType[N 199 They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
200 ICMP type number. These counters track all kin 200 ICMP type number. These counters track all kinds of ICMP packets. The
201 ICMP type number definition could be found in 201 ICMP type number definition could be found in the `ICMP parameters`_
202 document. 202 document.
203 203
204 .. _ICMP parameters: https://www.iana.org/assi 204 .. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
205 205
206 For example, if the Linux kernel sends an ICMP 206 For example, if the Linux kernel sends an ICMP Echo packet, the
207 IcmpMsgOutType8 would increase 1. And if kerne 207 IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
208 packet, IcmpMsgInType0 would increase 1. 208 packet, IcmpMsgInType0 would increase 1.
209 209
210 * IcmpInCsumErrors 210 * IcmpInCsumErrors
211 211
212 This counter indicates the checksum of the ICM 212 This counter indicates the checksum of the ICMP packet is
213 wrong. Kernel verifies the checksum after upda 213 wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
214 before updating IcmpMsgInType[N]. If a packet 214 before updating IcmpMsgInType[N]. If a packet has bad checksum, the
215 IcmpInMsgs would be updated but none of IcmpMs 215 IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.
216 216
217 * IcmpInErrors and IcmpOutErrors 217 * IcmpInErrors and IcmpOutErrors
218 218
219 Defined by `RFC1213 icmpInErrors`_ and `RFC121 219 Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_
220 220
221 .. _RFC1213 icmpInErrors: https://tools.ietf.o 221 .. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
222 .. _RFC1213 icmpOutErrors: https://tools.ietf. 222 .. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
223 223
224 When an error occurs in the ICMP packet handle 224 When an error occurs in the ICMP packet handler path, these two
225 counters would be updated. The receiving packe 225 counters would be updated. The receiving packet path use IcmpInErrors
226 and the sending packet path use IcmpOutErrors. 226 and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
227 is increased, IcmpInErrors would always be inc 227 is increased, IcmpInErrors would always be increased too.
228 228
229 relationship of the ICMP counters 229 relationship of the ICMP counters
230 --------------------------------- 230 ---------------------------------
231 The sum of IcmpMsgOutType[N] is always equal t 231 The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
232 are updated at the same time. The sum of IcmpM 232 are updated at the same time. The sum of IcmpMsgInType[N] plus
233 IcmpInErrors should be equal or larger than Ic 233 IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
234 receives an ICMP packet, kernel follows below 234 receives an ICMP packet, kernel follows below logic:
235 235
236 1. increase IcmpInMsgs 236 1. increase IcmpInMsgs
237 2. if has any error, update IcmpInErrors and f 237 2. if has any error, update IcmpInErrors and finish the process
238 3. update IcmpMsgOutType[N] 238 3. update IcmpMsgOutType[N]
239 4. handle the packet depending on the type, if 239 4. handle the packet depending on the type, if has any error, update
240 IcmpInErrors and finish the process 240 IcmpInErrors and finish the process
241 241
242 So if all errors occur in step (2), IcmpInMsgs 242 So if all errors occur in step (2), IcmpInMsgs should be equal to the
243 sum of IcmpMsgOutType[N] plus IcmpInErrors. If 243 sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
244 step (4), IcmpInMsgs should be equal to the su 244 step (4), IcmpInMsgs should be equal to the sum of
245 IcmpMsgOutType[N]. If the errors occur in both 245 IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
246 IcmpInMsgs should be less than the sum of Icmp 246 IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
247 IcmpInErrors. 247 IcmpInErrors.
248 248
249 General TCP counters 249 General TCP counters
250 ==================== 250 ====================
251 * TcpInSegs 251 * TcpInSegs
252 252
253 Defined in `RFC1213 tcpInSegs`_ 253 Defined in `RFC1213 tcpInSegs`_
254 254
255 .. _RFC1213 tcpInSegs: https://tools.ietf.org/ 255 .. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
256 256
257 The number of packets received by the TCP laye 257 The number of packets received by the TCP layer. As mentioned in
258 RFC1213, it includes the packets received in e 258 RFC1213, it includes the packets received in error, such as checksum
259 error, invalid TCP header and so on. Only one 259 error, invalid TCP header and so on. Only one error won't be included:
260 if the layer 2 destination address is not the 260 if the layer 2 destination address is not the NIC's layer 2
261 address. It might happen if the packet is a mu 261 address. It might happen if the packet is a multicast or broadcast
262 packet, or the NIC is in promiscuous mode. In 262 packet, or the NIC is in promiscuous mode. In these situations, the
263 packets would be delivered to the TCP layer, b 263 packets would be delivered to the TCP layer, but the TCP layer will discard
264 these packets before increasing TcpInSegs. The 264 these packets before increasing TcpInSegs. The TcpInSegs counter
265 isn't aware of GRO. So if two packets are merg 265 isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs
266 counter would only increase 1. 266 counter would only increase 1.
267 267
268 * TcpOutSegs 268 * TcpOutSegs
269 269
270 Defined in `RFC1213 tcpOutSegs`_ 270 Defined in `RFC1213 tcpOutSegs`_
271 271
272 .. _RFC1213 tcpOutSegs: https://tools.ietf.org 272 .. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
273 273
274 The number of packets sent by the TCP layer. A 274 The number of packets sent by the TCP layer. As mentioned in RFC1213,
275 it excludes the retransmitted packets. But it 275 it excludes the retransmitted packets. But it includes the SYN, ACK
276 and RST packets. Doesn't like TcpInSegs, the T 276 and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of
277 GSO, so if a packet would be split to 2 by GSO 277 GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will
278 increase 2. 278 increase 2.
279 279
280 * TcpActiveOpens 280 * TcpActiveOpens
281 281
282 Defined in `RFC1213 tcpActiveOpens`_ 282 Defined in `RFC1213 tcpActiveOpens`_
283 283
284 .. _RFC1213 tcpActiveOpens: https://tools.ietf 284 .. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
285 285
286 It means the TCP layer sends a SYN, and come i 286 It means the TCP layer sends a SYN, and come into the SYN-SENT
287 state. Every time TcpActiveOpens increases 1, 287 state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
288 increase 1. 288 increase 1.
289 289
290 * TcpPassiveOpens 290 * TcpPassiveOpens
291 291
292 Defined in `RFC1213 tcpPassiveOpens`_ 292 Defined in `RFC1213 tcpPassiveOpens`_
293 293
294 .. _RFC1213 tcpPassiveOpens: https://tools.iet 294 .. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
295 295
296 It means the TCP layer receives a SYN, replies 296 It means the TCP layer receives a SYN, replies a SYN+ACK, come into
297 the SYN-RCVD state. 297 the SYN-RCVD state.
298 298
299 * TcpExtTCPRcvCoalesce 299 * TcpExtTCPRcvCoalesce
300 300
301 When packets are received by the TCP layer and 301 When packets are received by the TCP layer and are not be read by the
302 application, the TCP layer will try to merge t 302 application, the TCP layer will try to merge them. This counter
303 indicate how many packets are merged in such s 303 indicate how many packets are merged in such situation. If GRO is
304 enabled, lots of packets would be merged by GR 304 enabled, lots of packets would be merged by GRO, these packets
305 wouldn't be counted to TcpExtTCPRcvCoalesce. 305 wouldn't be counted to TcpExtTCPRcvCoalesce.
306 306
307 * TcpExtTCPAutoCorking 307 * TcpExtTCPAutoCorking
308 308
309 When sending packets, the TCP layer will try t 309 When sending packets, the TCP layer will try to merge small packets to
310 a bigger one. This counter increase 1 for ever 310 a bigger one. This counter increase 1 for every packet merged in such
311 situation. Please refer to the LWN article for 311 situation. Please refer to the LWN article for more details:
312 https://lwn.net/Articles/576263/ 312 https://lwn.net/Articles/576263/
313 313
314 * TcpExtTCPOrigDataSent 314 * TcpExtTCPOrigDataSent
315 315
316 This counter is explained by kernel commit f19 !! 316 This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
317 explanation below:: 317 explanation below::
318 318
319 TCPOrigDataSent: number of outgoing packets 319 TCPOrigDataSent: number of outgoing packets with original data (excluding
320 retransmission but including data-in-SYN). T 320 retransmission but including data-in-SYN). This counter is different from
321 TcpOutSegs because TcpOutSegs also tracks pu 321 TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is
322 more useful to track the TCP retransmission 322 more useful to track the TCP retransmission rate.
323 323
324 * TCPSynRetrans 324 * TCPSynRetrans
325 325
326 This counter is explained by kernel commit f19 !! 326 This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
327 explanation below:: 327 explanation below::
328 328
329 TCPSynRetrans: number of SYN and SYN/ACK ret 329 TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down
330 retransmissions into SYN, fast-retransmits, 330 retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
331 331
332 * TCPFastOpenActiveFail 332 * TCPFastOpenActiveFail
333 333
334 This counter is explained by kernel commit f19 !! 334 This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
335 explanation below:: 335 explanation below::
336 336
337 TCPFastOpenActiveFail: Fast Open attempts (S 337 TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because
338 the remote does not accept it or the attempt 338 the remote does not accept it or the attempts timed out.
339 339
>> 340 .. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd
>> 341
340 * TcpExtListenOverflows and TcpExtListenDrops 342 * TcpExtListenOverflows and TcpExtListenDrops
341 343
342 When kernel receives a SYN from a client, and 344 When kernel receives a SYN from a client, and if the TCP accept queue
343 is full, kernel will drop the SYN and add 1 to 345 is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows.
344 At the same time kernel will also add 1 to Tcp 346 At the same time kernel will also add 1 to TcpExtListenDrops. When a
345 TCP socket is in LISTEN state, and kernel need 347 TCP socket is in LISTEN state, and kernel need to drop a packet,
346 kernel would always add 1 to TcpExtListenDrops 348 kernel would always add 1 to TcpExtListenDrops. So increase
347 TcpExtListenOverflows would let TcpExtListenDr 349 TcpExtListenOverflows would let TcpExtListenDrops increasing at the
348 same time, but TcpExtListenDrops would also in 350 same time, but TcpExtListenDrops would also increase without
349 TcpExtListenOverflows increasing, e.g. a memor 351 TcpExtListenOverflows increasing, e.g. a memory allocation fail would
350 also let TcpExtListenDrops increase. 352 also let TcpExtListenDrops increase.
351 353
352 Note: The above explanation is based on kernel 354 Note: The above explanation is based on kernel 4.10 or above version, on
353 an old kernel, the TCP stack has different beh 355 an old kernel, the TCP stack has different behavior when TCP accept
354 queue is full. On the old kernel, TCP stack wo 356 queue is full. On the old kernel, TCP stack won't drop the SYN, it
355 would complete the 3-way handshake. As the acc 357 would complete the 3-way handshake. As the accept queue is full, TCP
356 stack will keep the socket in the TCP half-ope 358 stack will keep the socket in the TCP half-open queue. As it is in the
357 half open queue, TCP stack will send SYN+ACK o 359 half open queue, TCP stack will send SYN+ACK on an exponential backoff
358 timer, after client replies ACK, TCP stack che 360 timer, after client replies ACK, TCP stack checks whether the accept
359 queue is still full, if it is not full, moves 361 queue is still full, if it is not full, moves the socket to the accept
360 queue, if it is full, keeps the socket in the 362 queue, if it is full, keeps the socket in the half-open queue, at next
361 time client replies ACK, this socket will get 363 time client replies ACK, this socket will get another chance to move
362 to the accept queue. 364 to the accept queue.
363 365
364 366
365 TCP Fast Open 367 TCP Fast Open
366 ============= 368 =============
367 * TcpEstabResets 369 * TcpEstabResets
368 370
369 Defined in `RFC1213 tcpEstabResets`_. 371 Defined in `RFC1213 tcpEstabResets`_.
370 372
371 .. _RFC1213 tcpEstabResets: https://tools.ietf 373 .. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
372 374
373 * TcpAttemptFails 375 * TcpAttemptFails
374 376
375 Defined in `RFC1213 tcpAttemptFails`_. 377 Defined in `RFC1213 tcpAttemptFails`_.
376 378
377 .. _RFC1213 tcpAttemptFails: https://tools.iet 379 .. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
378 380
379 * TcpOutRsts 381 * TcpOutRsts
380 382
381 Defined in `RFC1213 tcpOutRsts`_. The RFC says 383 Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates
382 the 'segments sent containing the RST flag', b 384 the 'segments sent containing the RST flag', but in linux kernel, this
383 counter indicates the segments kernel tried to 385 counter indicates the segments kernel tried to send. The sending
384 process might be failed due to some errors (e. 386 process might be failed due to some errors (e.g. memory alloc failed).
385 387
386 .. _RFC1213 tcpOutRsts: https://tools.ietf.org 388 .. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
387 389
388 * TcpExtTCPSpuriousRtxHostQueues 390 * TcpExtTCPSpuriousRtxHostQueues
389 391
390 When the TCP stack wants to retransmit a packe 392 When the TCP stack wants to retransmit a packet, and finds that packet
391 is not lost in the network, but the packet is 393 is not lost in the network, but the packet is not sent yet, the TCP
392 stack would give up the retransmission and upd 394 stack would give up the retransmission and update this counter. It
393 might happen if a packet stays too long time i 395 might happen if a packet stays too long time in a qdisc or driver
394 queue. 396 queue.
395 397
396 * TcpEstabResets 398 * TcpEstabResets
397 399
398 The socket receives a RST packet in Establish 400 The socket receives a RST packet in Establish or CloseWait state.
399 401
400 * TcpExtTCPKeepAlive 402 * TcpExtTCPKeepAlive
401 403
402 This counter indicates many keepalive packets 404 This counter indicates many keepalive packets were sent. The keepalive
403 won't be enabled by default. A userspace progr 405 won't be enabled by default. A userspace program could enable it by
404 setting the SO_KEEPALIVE socket option. 406 setting the SO_KEEPALIVE socket option.
405 407
406 * TcpExtTCPSpuriousRTOs 408 * TcpExtTCPSpuriousRTOs
407 409
408 The spurious retransmission timeout detected b 410 The spurious retransmission timeout detected by the `F-RTO`_
409 algorithm. 411 algorithm.
410 412
411 .. _F-RTO: https://tools.ietf.org/html/rfc5682 413 .. _F-RTO: https://tools.ietf.org/html/rfc5682
412 414
413 TCP Fast Path 415 TCP Fast Path
414 ============= 416 =============
415 When kernel receives a TCP packet, it has two 417 When kernel receives a TCP packet, it has two paths to handler the
416 packet, one is fast path, another is slow path 418 packet, one is fast path, another is slow path. The comment in kernel
417 code provides a good explanation of them, I pa 419 code provides a good explanation of them, I pasted them below::
418 420
419 It is split into a fast path and a slow path 421 It is split into a fast path and a slow path. The fast path is
420 disabled when: 422 disabled when:
421 423
422 - A zero window was announced from us 424 - A zero window was announced from us
423 - zero window probing 425 - zero window probing
424 is only handled properly on the slow path. 426 is only handled properly on the slow path.
425 - Out of order segments arrived. 427 - Out of order segments arrived.
426 - Urgent data is expected. 428 - Urgent data is expected.
427 - There is no buffer space left 429 - There is no buffer space left
428 - Unexpected TCP flags/window values/header 430 - Unexpected TCP flags/window values/header lengths are received
429 (detected by checking the TCP header again 431 (detected by checking the TCP header against pred_flags)
430 - Data is sent in both directions. The fast 432 - Data is sent in both directions. The fast path only supports pure senders
431 or pure receivers (this means either the s 433 or pure receivers (this means either the sequence number or the ack
432 value must stay constant) 434 value must stay constant)
433 - Unexpected TCP option. 435 - Unexpected TCP option.
434 436
435 Kernel will try to use fast path unless any of 437 Kernel will try to use fast path unless any of the above conditions
436 are satisfied. If the packets are out of order 438 are satisfied. If the packets are out of order, kernel will handle
437 them in slow path, which means the performance 439 them in slow path, which means the performance might be not very
438 good. Kernel would also come into slow path if 440 good. Kernel would also come into slow path if the "Delayed ack" is
439 used, because when using "Delayed ack", the da 441 used, because when using "Delayed ack", the data is sent in both
440 directions. When the TCP window scale option i 442 directions. When the TCP window scale option is not used, kernel will
441 try to enable fast path immediately when the c 443 try to enable fast path immediately when the connection comes into the
442 established state, but if the TCP window scale 444 established state, but if the TCP window scale option is used, kernel
443 will disable the fast path at first, and try t 445 will disable the fast path at first, and try to enable it after kernel
444 receives packets. 446 receives packets.
445 447
446 * TcpExtTCPPureAcks and TcpExtTCPHPAcks 448 * TcpExtTCPPureAcks and TcpExtTCPHPAcks
447 449
448 If a packet set ACK flag and has no data, it i 450 If a packet set ACK flag and has no data, it is a pure ACK packet, if
449 kernel handles it in the fast path, TcpExtTCPH 451 kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,
450 if kernel handles it in the slow path, TcpExtT 452 if kernel handles it in the slow path, TcpExtTCPPureAcks will
451 increase 1. 453 increase 1.
452 454
453 * TcpExtTCPHPHits 455 * TcpExtTCPHPHits
454 456
455 If a TCP packet has data (which means it is no 457 If a TCP packet has data (which means it is not a pure ACK packet),
456 and this packet is handled in the fast path, T 458 and this packet is handled in the fast path, TcpExtTCPHPHits will
457 increase 1. 459 increase 1.
458 460
459 461
460 TCP abort 462 TCP abort
461 ========= 463 =========
462 * TcpExtTCPAbortOnData 464 * TcpExtTCPAbortOnData
463 465
464 It means TCP layer has data in flight, but nee 466 It means TCP layer has data in flight, but need to close the
465 connection. So TCP layer sends a RST to the ot 467 connection. So TCP layer sends a RST to the other side, indicate the
466 connection is not closed very graceful. An eas 468 connection is not closed very graceful. An easy way to increase this
467 counter is using the SO_LINGER option. Please 469 counter is using the SO_LINGER option. Please refer to the SO_LINGER
468 section of the `socket man page`_: 470 section of the `socket man page`_:
469 471
470 .. _socket man page: http://man7.org/linux/man 472 .. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
471 473
472 By default, when an application closes a conne 474 By default, when an application closes a connection, the close function
473 will return immediately and kernel will try to 475 will return immediately and kernel will try to send the in-flight data
474 async. If you use the SO_LINGER option, set l_ 476 async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
475 to a positive number, the close function won't 477 to a positive number, the close function won't return immediately, but
476 wait for the in-flight data are acked by the o 478 wait for the in-flight data are acked by the other side, the max wait
477 time is l_linger seconds. If set l_onoff to 1 479 time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
478 when the application closes a connection, kern 480 when the application closes a connection, kernel will send a RST
479 immediately and increase the TcpExtTCPAbortOnD 481 immediately and increase the TcpExtTCPAbortOnData counter.
480 482
481 * TcpExtTCPAbortOnClose 483 * TcpExtTCPAbortOnClose
482 484
483 This counter means the application has unread 485 This counter means the application has unread data in the TCP layer when
484 the application wants to close the TCP connect 486 the application wants to close the TCP connection. In such a situation,
485 kernel will send a RST to the other side of th 487 kernel will send a RST to the other side of the TCP connection.
486 488
487 * TcpExtTCPAbortOnMemory 489 * TcpExtTCPAbortOnMemory
488 490
489 When an application closes a TCP connection, k 491 When an application closes a TCP connection, kernel still need to track
490 the connection, let it complete the TCP discon 492 the connection, let it complete the TCP disconnect process. E.g. an
491 app calls the close method of a socket, kernel 493 app calls the close method of a socket, kernel sends fin to the other
492 side of the connection, then the app has no re 494 side of the connection, then the app has no relationship with the
493 socket any more, but kernel need to keep the s 495 socket any more, but kernel need to keep the socket, this socket
494 becomes an orphan socket, kernel waits for the 496 becomes an orphan socket, kernel waits for the reply of the other side,
495 and would come to the TIME_WAIT state finally. 497 and would come to the TIME_WAIT state finally. When kernel has no
496 enough memory to keep the orphan socket, kerne 498 enough memory to keep the orphan socket, kernel would send an RST to
497 the other side, and delete the socket, in such 499 the other side, and delete the socket, in such situation, kernel will
498 increase 1 to the TcpExtTCPAbortOnMemory. Two 500 increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
499 TcpExtTCPAbortOnMemory: 501 TcpExtTCPAbortOnMemory:
500 502
501 1. the memory used by the TCP protocol is high 503 1. the memory used by the TCP protocol is higher than the third value of
502 the tcp_mem. Please refer the tcp_mem section 504 the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:
503 505
504 .. _TCP man page: http://man7.org/linux/man-pa 506 .. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
505 507
506 2. the orphan socket count is higher than net. 508 2. the orphan socket count is higher than net.ipv4.tcp_max_orphans
507 509
508 510
509 * TcpExtTCPAbortOnTimeout 511 * TcpExtTCPAbortOnTimeout
510 512
511 This counter will increase when any of the TCP 513 This counter will increase when any of the TCP timers expire. In such
512 situation, kernel won't send RST, just give up 514 situation, kernel won't send RST, just give up the connection.
513 515
514 * TcpExtTCPAbortOnLinger 516 * TcpExtTCPAbortOnLinger
515 517
516 When a TCP connection comes into FIN_WAIT_2 st 518 When a TCP connection comes into FIN_WAIT_2 state, instead of waiting
517 for the fin packet from the other side, kernel 519 for the fin packet from the other side, kernel could send a RST and
518 delete the socket immediately. This is not the 520 delete the socket immediately. This is not the default behavior of
519 Linux kernel TCP stack. By configuring the TCP 521 Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,
520 you could let kernel follow this behavior. 522 you could let kernel follow this behavior.
521 523
522 * TcpExtTCPAbortFailed 524 * TcpExtTCPAbortFailed
523 525
524 The kernel TCP layer will send RST if the `RFC 526 The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is
525 satisfied. If an internal error occurs during 527 satisfied. If an internal error occurs during this process,
526 TcpExtTCPAbortFailed will be increased. 528 TcpExtTCPAbortFailed will be increased.
527 529
528 .. _RFC2525 2.17 section: https://tools.ietf.o 530 .. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
529 531
530 TCP Hybrid Slow Start 532 TCP Hybrid Slow Start
531 ===================== 533 =====================
532 The Hybrid Slow Start algorithm is an enhancem 534 The Hybrid Slow Start algorithm is an enhancement of the traditional
533 TCP congestion window Slow Start algorithm. It 535 TCP congestion window Slow Start algorithm. It uses two pieces of
534 information to detect whether the max bandwidt 536 information to detect whether the max bandwidth of the TCP path is
535 approached. The two pieces of information are 537 approached. The two pieces of information are ACK train length and
536 increase in packet delay. For detail informati 538 increase in packet delay. For detail information, please refer the
537 `Hybrid Slow Start paper`_. Either ACK train l 539 `Hybrid Slow Start paper`_. Either ACK train length or packet delay
538 hits a specific threshold, the congestion cont 540 hits a specific threshold, the congestion control algorithm will come
539 into the Congestion Avoidance state. Until v4. 541 into the Congestion Avoidance state. Until v4.20, two congestion
540 control algorithms are using Hybrid Slow Start 542 control algorithms are using Hybrid Slow Start, they are cubic (the
541 default congestion control algorithm) and cdg. 543 default congestion control algorithm) and cdg. Four snmp counters
542 relate with the Hybrid Slow Start algorithm. 544 relate with the Hybrid Slow Start algorithm.
543 545
544 .. _Hybrid Slow Start paper: https://pdfs.sema 546 .. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf
545 547
546 * TcpExtTCPHystartTrainDetect 548 * TcpExtTCPHystartTrainDetect
547 549
548 How many times the ACK train length threshold 550 How many times the ACK train length threshold is detected
549 551
550 * TcpExtTCPHystartTrainCwnd 552 * TcpExtTCPHystartTrainCwnd
551 553
552 The sum of CWND detected by ACK train length. 554 The sum of CWND detected by ACK train length. Dividing this value by
553 TcpExtTCPHystartTrainDetect is the average CWN 555 TcpExtTCPHystartTrainDetect is the average CWND which detected by the
554 ACK train length. 556 ACK train length.
555 557
556 * TcpExtTCPHystartDelayDetect 558 * TcpExtTCPHystartDelayDetect
557 559
558 How many times the packet delay threshold is d 560 How many times the packet delay threshold is detected.
559 561
560 * TcpExtTCPHystartDelayCwnd 562 * TcpExtTCPHystartDelayCwnd
561 563
562 The sum of CWND detected by packet delay. Divi 564 The sum of CWND detected by packet delay. Dividing this value by
563 TcpExtTCPHystartDelayDetect is the average CWN 565 TcpExtTCPHystartDelayDetect is the average CWND which detected by the
564 packet delay. 566 packet delay.
565 567
566 TCP retransmission and congestion control 568 TCP retransmission and congestion control
567 ========================================= 569 =========================================
568 The TCP protocol has two retransmission mechan 570 The TCP protocol has two retransmission mechanisms: SACK and fast
569 recovery. They are exclusive with each other. 571 recovery. They are exclusive with each other. When SACK is enabled,
570 the kernel TCP stack would use SACK, or kernel 572 the kernel TCP stack would use SACK, or kernel would use fast
571 recovery. The SACK is a TCP option, which is d 573 recovery. The SACK is a TCP option, which is defined in `RFC2018`_,
572 the fast recovery is defined in `RFC6582`_, wh 574 the fast recovery is defined in `RFC6582`_, which is also called
573 'Reno'. 575 'Reno'.
574 576
575 The TCP congestion control is a big and comple 577 The TCP congestion control is a big and complex topic. To understand
576 the related snmp counter, we need to know the 578 the related snmp counter, we need to know the states of the congestion
577 control state machine. There are 5 states: Ope 579 control state machine. There are 5 states: Open, Disorder, CWR,
578 Recovery and Loss. For details about these sta 580 Recovery and Loss. For details about these states, please refer page 5
579 and page 6 of this document: 581 and page 6 of this document:
580 https://pdfs.semanticscholar.org/0e9c/968d09ab 582 https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf
581 583
582 .. _RFC2018: https://tools.ietf.org/html/rfc20 584 .. _RFC2018: https://tools.ietf.org/html/rfc2018
583 .. _RFC6582: https://tools.ietf.org/html/rfc65 585 .. _RFC6582: https://tools.ietf.org/html/rfc6582
584 586
585 * TcpExtTCPRenoRecovery and TcpExtTCPSackRecov 587 * TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery
586 588
587 When the congestion control comes into Recover 589 When the congestion control comes into Recovery state, if sack is
588 used, TcpExtTCPSackRecovery increases 1, if sa 590 used, TcpExtTCPSackRecovery increases 1, if sack is not used,
589 TcpExtTCPRenoRecovery increases 1. These two c 591 TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP
590 stack begins to retransmit the lost packets. 592 stack begins to retransmit the lost packets.
591 593
592 * TcpExtTCPSACKReneging 594 * TcpExtTCPSACKReneging
593 595
594 A packet was acknowledged by SACK, but the rec 596 A packet was acknowledged by SACK, but the receiver has dropped this
595 packet, so the sender needs to retransmit this 597 packet, so the sender needs to retransmit this packet. In this
596 situation, the sender adds 1 to TcpExtTCPSACKR 598 situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver
597 could drop a packet which has been acknowledge 599 could drop a packet which has been acknowledged by SACK, although it is
598 unusual, it is allowed by the TCP protocol. Th 600 unusual, it is allowed by the TCP protocol. The sender doesn't really
599 know what happened on the receiver side. The s 601 know what happened on the receiver side. The sender just waits until
600 the RTO expires for this packet, then the send 602 the RTO expires for this packet, then the sender assumes this packet
601 has been dropped by the receiver. 603 has been dropped by the receiver.
602 604
603 * TcpExtTCPRenoReorder 605 * TcpExtTCPRenoReorder
604 606
605 The reorder packet is detected by fast recover 607 The reorder packet is detected by fast recovery. It would only be used
606 if SACK is disabled. The fast recovery algorit 608 if SACK is disabled. The fast recovery algorithm detects recorder by
607 the duplicate ACK number. E.g., if retransmiss 609 the duplicate ACK number. E.g., if retransmission is triggered, and
608 the original retransmitted packet is not lost, 610 the original retransmitted packet is not lost, it is just out of
609 order, the receiver would acknowledge multiple 611 order, the receiver would acknowledge multiple times, one for the
610 retransmitted packet, another for the arriving 612 retransmitted packet, another for the arriving of the original out of
611 order packet. Thus the sender would find more 613 order packet. Thus the sender would find more ACks than its
612 expectation, and the sender knows out of order 614 expectation, and the sender knows out of order occurs.
613 615
614 * TcpExtTCPTSReorder 616 * TcpExtTCPTSReorder
615 617
616 The reorder packet is detected when a hole is 618 The reorder packet is detected when a hole is filled. E.g., assume the
617 sender sends packet 1,2,3,4,5, and the receivi 619 sender sends packet 1,2,3,4,5, and the receiving order is
618 1,2,4,5,3. When the sender receives the ACK of 620 1,2,4,5,3. When the sender receives the ACK of packet 3 (which will
619 fill the hole), two conditions will let TcpExt 621 fill the hole), two conditions will let TcpExtTCPTSReorder increase
620 1: (1) if the packet 3 is not re-retransmitted 622 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
621 3 is retransmitted but the timestamp of the pa 623 3 is retransmitted but the timestamp of the packet 3's ACK is earlier
622 than the retransmission timestamp. 624 than the retransmission timestamp.
623 625
624 * TcpExtTCPSACKReorder 626 * TcpExtTCPSACKReorder
625 627
626 The reorder packet detected by SACK. The SACK 628 The reorder packet detected by SACK. The SACK has two methods to
627 detect reorder: (1) DSACK is received by the s 629 detect reorder: (1) DSACK is received by the sender. It means the
628 sender sends the same packet more than one tim 630 sender sends the same packet more than one times. And the only reason
629 is the sender believes an out of order packet 631 is the sender believes an out of order packet is lost so it sends the
630 packet again. (2) Assume packet 1,2,3,4,5 are 632 packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and
631 the sender has received SACKs for packet 2 and 633 the sender has received SACKs for packet 2 and 5, now the sender
632 receives SACK for packet 4 and the sender does 634 receives SACK for packet 4 and the sender doesn't retransmit the
633 packet yet, the sender would know packet 4 is 635 packet yet, the sender would know packet 4 is out of order. The TCP
634 stack of kernel will increase TcpExtTCPSACKReo 636 stack of kernel will increase TcpExtTCPSACKReorder for both of the
635 above scenarios. 637 above scenarios.
636 638
637 * TcpExtTCPSlowStartRetrans 639 * TcpExtTCPSlowStartRetrans
638 640
639 The TCP stack wants to retransmit a packet and 641 The TCP stack wants to retransmit a packet and the congestion control
640 state is 'Loss'. 642 state is 'Loss'.
641 643
642 * TcpExtTCPFastRetrans 644 * TcpExtTCPFastRetrans
643 645
644 The TCP stack wants to retransmit a packet and 646 The TCP stack wants to retransmit a packet and the congestion control
645 state is not 'Loss'. 647 state is not 'Loss'.
646 648
647 * TcpExtTCPLostRetransmit 649 * TcpExtTCPLostRetransmit
648 650
649 A SACK points out that a retransmission packet 651 A SACK points out that a retransmission packet is lost again.
650 652
651 * TcpExtTCPRetransFail 653 * TcpExtTCPRetransFail
652 654
653 The TCP stack tries to deliver a retransmissio 655 The TCP stack tries to deliver a retransmission packet to lower layers
654 but the lower layers return an error. 656 but the lower layers return an error.
655 657
656 * TcpExtTCPSynRetrans 658 * TcpExtTCPSynRetrans
657 659
658 The TCP stack retransmits a SYN packet. 660 The TCP stack retransmits a SYN packet.
659 661
660 DSACK 662 DSACK
661 ===== 663 =====
662 The DSACK is defined in `RFC2883`_. The receiv 664 The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report
663 duplicate packets to the sender. There are two 665 duplicate packets to the sender. There are two kinds of
664 duplications: (1) a packet which has been ackn 666 duplications: (1) a packet which has been acknowledged is
665 duplicate. (2) an out of order packet is dupli 667 duplicate. (2) an out of order packet is duplicate. The TCP stack
666 counts these two kinds of duplications on both 668 counts these two kinds of duplications on both receiver side and
667 sender side. 669 sender side.
668 670
669 .. _RFC2883 : https://tools.ietf.org/html/rfc2 671 .. _RFC2883 : https://tools.ietf.org/html/rfc2883
670 672
671 * TcpExtTCPDSACKOldSent 673 * TcpExtTCPDSACKOldSent
672 674
673 The TCP stack receives a duplicate packet whic 675 The TCP stack receives a duplicate packet which has been acked, so it
674 sends a DSACK to the sender. 676 sends a DSACK to the sender.
675 677
676 * TcpExtTCPDSACKOfoSent 678 * TcpExtTCPDSACKOfoSent
677 679
678 The TCP stack receives an out of order duplica 680 The TCP stack receives an out of order duplicate packet, so it sends a
679 DSACK to the sender. 681 DSACK to the sender.
680 682
681 * TcpExtTCPDSACKRecv 683 * TcpExtTCPDSACKRecv
682 684
683 The TCP stack receives a DSACK, which indicate 685 The TCP stack receives a DSACK, which indicates an acknowledged
684 duplicate packet is received. 686 duplicate packet is received.
685 687
686 * TcpExtTCPDSACKOfoRecv 688 * TcpExtTCPDSACKOfoRecv
687 689
688 The TCP stack receives a DSACK, which indicate 690 The TCP stack receives a DSACK, which indicate an out of order
689 duplicate packet is received. 691 duplicate packet is received.
690 692
691 invalid SACK and DSACK 693 invalid SACK and DSACK
692 ====================== 694 ======================
693 When a SACK (or DSACK) block is invalid, a cor 695 When a SACK (or DSACK) block is invalid, a corresponding counter would
694 be updated. The validation method is base on t 696 be updated. The validation method is base on the start/end sequence
695 number of the SACK block. For more details, pl 697 number of the SACK block. For more details, please refer the comment
696 of the function tcp_is_sackblock_valid in the 698 of the function tcp_is_sackblock_valid in the kernel source code. A
697 SACK option could have up to 4 blocks, they ar 699 SACK option could have up to 4 blocks, they are checked
698 individually. E.g., if 3 blocks of a SACk is i 700 individually. E.g., if 3 blocks of a SACk is invalid, the
699 corresponding counter would be updated 3 times !! 701 corresponding counter would be updated 3 times. The comment of the
700 18f02545a9a1 ("[TCP] MIB: Add counters for dis !! 702 `Add counters for discarded SACK blocks`_ patch has additional
701 has additional explanation: !! 703 explanation:
>> 704
>> 705 .. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18f02545a9a16c9a89778b91a162ad16d510bb32
702 706
703 * TcpExtTCPSACKDiscard 707 * TcpExtTCPSACKDiscard
704 708
705 This counter indicates how many SACK blocks ar 709 This counter indicates how many SACK blocks are invalid. If the invalid
706 SACK block is caused by ACK recording, the TCP 710 SACK block is caused by ACK recording, the TCP stack will only ignore
707 it and won't update this counter. 711 it and won't update this counter.
708 712
709 * TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKI 713 * TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo
710 714
711 When a DSACK block is invalid, one of these tw 715 When a DSACK block is invalid, one of these two counters would be
712 updated. Which counter will be updated depends 716 updated. Which counter will be updated depends on the undo_marker flag
713 of the TCP socket. If the undo_marker is not s 717 of the TCP socket. If the undo_marker is not set, the TCP stack isn't
714 likely to re-transmit any packets, and we stil 718 likely to re-transmit any packets, and we still receive an invalid
715 DSACK block, the reason might be that the pack 719 DSACK block, the reason might be that the packet is duplicated in the
716 middle of the network. In such scenario, TcpEx 720 middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo
717 will be updated. If the undo_marker is set, Tc 721 will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld
718 will be updated. As implied in its name, it mi 722 will be updated. As implied in its name, it might be an old packet.
719 723
720 SACK shift 724 SACK shift
721 ========== 725 ==========
722 The linux networking stack stores data in sk_b 726 The linux networking stack stores data in sk_buff struct (skb for
723 short). If a SACK block acrosses multiple skb, 727 short). If a SACK block acrosses multiple skb, the TCP stack will try
724 to re-arrange data in these skb. E.g. if a SAC 728 to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
725 10 to 15, skb1 has seq 10 to 13, skb2 has seq 729 10 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and
726 15 in skb2 would be moved to skb1. This operat 730 15 in skb2 would be moved to skb1. This operation is 'shift'. If a
727 SACK block acknowledges seq 10 to 20, skb1 has 731 SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has
728 seq 14 to 20. All data in skb2 will be moved t 732 seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be
729 discard, this operation is 'merge'. 733 discard, this operation is 'merge'.
730 734
731 * TcpExtTCPSackShifted 735 * TcpExtTCPSackShifted
732 736
733 A skb is shifted 737 A skb is shifted
734 738
735 * TcpExtTCPSackMerged 739 * TcpExtTCPSackMerged
736 740
737 A skb is merged 741 A skb is merged
738 742
739 * TcpExtTCPSackShiftFallback 743 * TcpExtTCPSackShiftFallback
740 744
741 A skb should be shifted or merged, but the TCP 745 A skb should be shifted or merged, but the TCP stack doesn't do it for
742 some reasons. 746 some reasons.
743 747
744 TCP out of order 748 TCP out of order
745 ================ 749 ================
746 * TcpExtTCPOFOQueue 750 * TcpExtTCPOFOQueue
747 751
748 The TCP layer receives an out of order packet 752 The TCP layer receives an out of order packet and has enough memory
749 to queue it. 753 to queue it.
750 754
751 * TcpExtTCPOFODrop 755 * TcpExtTCPOFODrop
752 756
753 The TCP layer receives an out of order packet 757 The TCP layer receives an out of order packet but doesn't have enough
754 memory, so drops it. Such packets won't be cou 758 memory, so drops it. Such packets won't be counted into
755 TcpExtTCPOFOQueue. 759 TcpExtTCPOFOQueue.
756 760
757 * TcpExtTCPOFOMerge 761 * TcpExtTCPOFOMerge
758 762
759 The received out of order packet has an overla 763 The received out of order packet has an overlay with the previous
760 packet. the overlay part will be dropped. All 764 packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge
761 packets will also be counted into TcpExtTCPOFO 765 packets will also be counted into TcpExtTCPOFOQueue.
762 766
763 TCP PAWS 767 TCP PAWS
764 ======== 768 ========
765 PAWS (Protection Against Wrapped Sequence numb 769 PAWS (Protection Against Wrapped Sequence numbers) is an algorithm
766 which is used to drop old packets. It depends 770 which is used to drop old packets. It depends on the TCP
767 timestamps. For detail information, please ref 771 timestamps. For detail information, please refer the `timestamp wiki`_
768 and the `RFC of PAWS`_. 772 and the `RFC of PAWS`_.
769 773
770 .. _RFC of PAWS: https://tools.ietf.org/html/r 774 .. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
771 .. _timestamp wiki: https://en.wikipedia.org/w 775 .. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps
772 776
773 * TcpExtPAWSActive 777 * TcpExtPAWSActive
774 778
775 Packets are dropped by PAWS in Syn-Sent status 779 Packets are dropped by PAWS in Syn-Sent status.
776 780
777 * TcpExtPAWSEstab 781 * TcpExtPAWSEstab
778 782
779 Packets are dropped by PAWS in any status othe 783 Packets are dropped by PAWS in any status other than Syn-Sent.
780 784
781 TCP ACK skip 785 TCP ACK skip
782 ============ 786 ============
783 In some scenarios, kernel would avoid sending 787 In some scenarios, kernel would avoid sending duplicate ACKs too
784 frequently. Please find more details in the tc 788 frequently. Please find more details in the tcp_invalid_ratelimit
785 section of the `sysctl document`_. When kernel 789 section of the `sysctl document`_. When kernel decides to skip an ACK
786 due to tcp_invalid_ratelimit, kernel would upd 790 due to tcp_invalid_ratelimit, kernel would update one of below
787 counters to indicate the ACK is skipped in whi 791 counters to indicate the ACK is skipped in which scenario. The ACK
788 would only be skipped if the received packet i 792 would only be skipped if the received packet is either a SYN packet or
789 it has no data. 793 it has no data.
790 794
791 .. _sysctl document: https://www.kernel.org/do 795 .. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst
792 796
793 * TcpExtTCPACKSkippedSynRecv 797 * TcpExtTCPACKSkippedSynRecv
794 798
795 The ACK is skipped in Syn-Recv status. The Syn 799 The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
796 TCP stack receives a SYN and replies SYN+ACK. 800 TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is
797 waiting for an ACK. Generally, the TCP stack d 801 waiting for an ACK. Generally, the TCP stack doesn't need to send ACK
798 in the Syn-Recv status. But in several scenari 802 in the Syn-Recv status. But in several scenarios, the TCP stack need
799 to send an ACK. E.g., the TCP stack receives t 803 to send an ACK. E.g., the TCP stack receives the same SYN packet
800 repeately, the received packet does not pass t 804 repeately, the received packet does not pass the PAWS check, or the
801 received packet sequence number is out of wind 805 received packet sequence number is out of window. In these scenarios,
802 the TCP stack needs to send ACK. If the ACk se 806 the TCP stack needs to send ACK. If the ACk sending frequency is higher than
803 tcp_invalid_ratelimit allows, the TCP stack wi 807 tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and
804 increase TcpExtTCPACKSkippedSynRecv. 808 increase TcpExtTCPACKSkippedSynRecv.
805 809
806 810
807 * TcpExtTCPACKSkippedPAWS 811 * TcpExtTCPACKSkippedPAWS
808 812
809 The ACK is skipped due to PAWS (Protect Agains 813 The ACK is skipped due to PAWS (Protect Against Wrapped Sequence
810 numbers) check fails. If the PAWS check fails 814 numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
811 or Time-Wait statuses, the skipped ACK would b 815 or Time-Wait statuses, the skipped ACK would be counted to
812 TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippe 816 TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or
813 TcpExtTCPACKSkippedTimeWait. In all other stat 817 TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK
814 would be counted to TcpExtTCPACKSkippedPAWS. 818 would be counted to TcpExtTCPACKSkippedPAWS.
815 819
816 * TcpExtTCPACKSkippedSeq 820 * TcpExtTCPACKSkippedSeq
817 821
818 The sequence number is out of window and the t 822 The sequence number is out of window and the timestamp passes the PAWS
819 check and the TCP status is not Syn-Recv, Fin- 823 check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
820 824
821 * TcpExtTCPACKSkippedFinWait2 825 * TcpExtTCPACKSkippedFinWait2
822 826
823 The ACK is skipped in Fin-Wait-2 status, the r 827 The ACK is skipped in Fin-Wait-2 status, the reason would be either
824 PAWS check fails or the received sequence numb 828 PAWS check fails or the received sequence number is out of window.
825 829
826 * TcpExtTCPACKSkippedTimeWait 830 * TcpExtTCPACKSkippedTimeWait
827 831
828 The ACK is skipped in Time-Wait status, the re 832 The ACK is skipped in Time-Wait status, the reason would be either
829 PAWS check failed or the received sequence num 833 PAWS check failed or the received sequence number is out of window.
830 834
831 * TcpExtTCPACKSkippedChallenge 835 * TcpExtTCPACKSkippedChallenge
832 836
833 The ACK is skipped if the ACK is a challenge A 837 The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines
834 3 kind of challenge ACK, please refer `RFC 596 838 3 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,
835 `RFC 5961 section 4.2`_ and `RFC 5961 section 839 `RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these
836 three scenarios, In some TCP status, the linux 840 three scenarios, In some TCP status, the linux TCP stack would also
837 send challenge ACKs if the ACK number is befor 841 send challenge ACKs if the ACK number is before the first
838 unacknowledged number (more strict than `RFC 5 842 unacknowledged number (more strict than `RFC 5961 section 5.2`_).
839 843
840 .. _RFC 5961 section 3.2: https://tools.ietf.o 844 .. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
841 .. _RFC 5961 section 4.2: https://tools.ietf.o 845 .. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
842 .. _RFC 5961 section 5.2: https://tools.ietf.o 846 .. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
843 847
844 TCP receive window 848 TCP receive window
845 ================== 849 ==================
846 * TcpExtTCPWantZeroWindowAdv 850 * TcpExtTCPWantZeroWindowAdv
847 851
848 Depending on current memory usage, the TCP sta 852 Depending on current memory usage, the TCP stack tries to set receive
849 window to zero. But the receive window might s 853 window to zero. But the receive window might still be a no-zero
850 value. For example, if the previous window siz 854 value. For example, if the previous window size is 10, and the TCP
851 stack receives 3 bytes, the current window siz 855 stack receives 3 bytes, the current window size would be 7 even if the
852 window size calculated by the memory usage is 856 window size calculated by the memory usage is zero.
853 857
854 * TcpExtTCPToZeroWindowAdv 858 * TcpExtTCPToZeroWindowAdv
855 859
856 The TCP receive window is set to zero from a n 860 The TCP receive window is set to zero from a no-zero value.
857 861
858 * TcpExtTCPFromZeroWindowAdv 862 * TcpExtTCPFromZeroWindowAdv
859 863
860 The TCP receive window is set to no-zero value 864 The TCP receive window is set to no-zero value from zero.
861 865
862 866
863 Delayed ACK 867 Delayed ACK
864 =========== 868 ===========
865 The TCP Delayed ACK is a technique which is us 869 The TCP Delayed ACK is a technique which is used for reducing the
866 packet count in the network. For more details, 870 packet count in the network. For more details, please refer the
867 `Delayed ACK wiki`_ 871 `Delayed ACK wiki`_
868 872
869 .. _Delayed ACK wiki: https://en.wikipedia.org 873 .. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
870 874
871 * TcpExtDelayedACKs 875 * TcpExtDelayedACKs
872 876
873 A delayed ACK timer expires. The TCP stack wil 877 A delayed ACK timer expires. The TCP stack will send a pure ACK packet
874 and exit the delayed ACK mode. 878 and exit the delayed ACK mode.
875 879
876 * TcpExtDelayedACKLocked 880 * TcpExtDelayedACKLocked
877 881
878 A delayed ACK timer expires, but the TCP stack 882 A delayed ACK timer expires, but the TCP stack can't send an ACK
879 immediately due to the socket is locked by a u 883 immediately due to the socket is locked by a userspace program. The
880 TCP stack will send a pure ACK later (after th 884 TCP stack will send a pure ACK later (after the userspace program
881 unlock the socket). When the TCP stack sends t 885 unlock the socket). When the TCP stack sends the pure ACK later, the
882 TCP stack will also update TcpExtDelayedACKs a 886 TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK
883 mode. 887 mode.
884 888
885 * TcpExtDelayedACKLost 889 * TcpExtDelayedACKLost
886 890
887 It will be updated when the TCP stack receives 891 It will be updated when the TCP stack receives a packet which has been
888 ACKed. A Delayed ACK loss might cause this iss 892 ACKed. A Delayed ACK loss might cause this issue, but it would also be
889 triggered by other reasons, such as a packet i 893 triggered by other reasons, such as a packet is duplicated in the
890 network. 894 network.
891 895
892 Tail Loss Probe (TLP) 896 Tail Loss Probe (TLP)
893 ===================== 897 =====================
894 TLP is an algorithm which is used to detect TC 898 TLP is an algorithm which is used to detect TCP packet loss. For more
895 details, please refer the `TLP paper`_. 899 details, please refer the `TLP paper`_.
896 900
897 .. _TLP paper: https://tools.ietf.org/html/dra 901 .. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
898 902
899 * TcpExtTCPLossProbes 903 * TcpExtTCPLossProbes
900 904
901 A TLP probe packet is sent. 905 A TLP probe packet is sent.
902 906
903 * TcpExtTCPLossProbeRecovery 907 * TcpExtTCPLossProbeRecovery
904 908
905 A packet loss is detected and recovered by TLP 909 A packet loss is detected and recovered by TLP.
906 910
907 TCP Fast Open description 911 TCP Fast Open description
908 ========================= 912 =========================
909 TCP Fast Open is a technology which allows dat 913 TCP Fast Open is a technology which allows data transfer before the
910 3-way handshake complete. Please refer the `TC 914 3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
911 general description. 915 general description.
912 916
913 .. _TCP Fast Open wiki: https://en.wikipedia.o 917 .. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open
914 918
915 * TcpExtTCPFastOpenActive 919 * TcpExtTCPFastOpenActive
916 920
917 When the TCP stack receives an ACK packet in t 921 When the TCP stack receives an ACK packet in the SYN-SENT status, and
918 the ACK packet acknowledges the data in the SY 922 the ACK packet acknowledges the data in the SYN packet, the TCP stack
919 understand the TFO cookie is accepted by the o 923 understand the TFO cookie is accepted by the other side, then it
920 updates this counter. 924 updates this counter.
921 925
922 * TcpExtTCPFastOpenActiveFail 926 * TcpExtTCPFastOpenActiveFail
923 927
924 This counter indicates that the TCP stack init 928 This counter indicates that the TCP stack initiated a TCP Fast Open,
925 but it failed. This counter would be updated i 929 but it failed. This counter would be updated in three scenarios: (1)
926 the other side doesn't acknowledge the data in 930 the other side doesn't acknowledge the data in the SYN packet. (2) The
927 SYN packet which has the TFO cookie is timeout 931 SYN packet which has the TFO cookie is timeout at least once. (3)
928 after the 3-way handshake, the retransmission 932 after the 3-way handshake, the retransmission timeout happens
929 net.ipv4.tcp_retries1 times, because some midd 933 net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
930 fast open after the handshake. 934 fast open after the handshake.
931 935
932 * TcpExtTCPFastOpenPassive 936 * TcpExtTCPFastOpenPassive
933 937
934 This counter indicates how many times the TCP 938 This counter indicates how many times the TCP stack accepts the fast
935 open request. 939 open request.
936 940
937 * TcpExtTCPFastOpenPassiveFail 941 * TcpExtTCPFastOpenPassiveFail
938 942
939 This counter indicates how many times the TCP 943 This counter indicates how many times the TCP stack rejects the fast
940 open request. It is caused by either the TFO c 944 open request. It is caused by either the TFO cookie is invalid or the
941 TCP stack finds an error during the socket cre 945 TCP stack finds an error during the socket creating process.
942 946
943 * TcpExtTCPFastOpenListenOverflow 947 * TcpExtTCPFastOpenListenOverflow
944 948
945 When the pending fast open request number is l 949 When the pending fast open request number is larger than
946 fastopenq->max_qlen, the TCP stack will reject 950 fastopenq->max_qlen, the TCP stack will reject the fast open request
947 and update this counter. When this counter is 951 and update this counter. When this counter is updated, the TCP stack
948 won't update TcpExtTCPFastOpenPassive or 952 won't update TcpExtTCPFastOpenPassive or
949 TcpExtTCPFastOpenPassiveFail. The fastopenq->m 953 TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
950 TCP_FASTOPEN socket operation and it could not 954 TCP_FASTOPEN socket operation and it could not be larger than
951 net.core.somaxconn. For example: 955 net.core.somaxconn. For example:
952 956
953 setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, 957 setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));
954 958
955 * TcpExtTCPFastOpenCookieReqd 959 * TcpExtTCPFastOpenCookieReqd
956 960
957 This counter indicates how many times a client 961 This counter indicates how many times a client wants to request a TFO
958 cookie. 962 cookie.
959 963
960 SYN cookies 964 SYN cookies
961 =========== 965 ===========
962 SYN cookies are used to mitigate SYN flood, fo 966 SYN cookies are used to mitigate SYN flood, for details, please refer
963 the `SYN cookies wiki`_. 967 the `SYN cookies wiki`_.
964 968
965 .. _SYN cookies wiki: https://en.wikipedia.org 969 .. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies
966 970
967 * TcpExtSyncookiesSent 971 * TcpExtSyncookiesSent
968 972
969 It indicates how many SYN cookies are sent. 973 It indicates how many SYN cookies are sent.
970 974
971 * TcpExtSyncookiesRecv 975 * TcpExtSyncookiesRecv
972 976
973 How many reply packets of the SYN cookies the 977 How many reply packets of the SYN cookies the TCP stack receives.
974 978
975 * TcpExtSyncookiesFailed 979 * TcpExtSyncookiesFailed
976 980
977 The MSS decoded from the SYN cookie is invalid 981 The MSS decoded from the SYN cookie is invalid. When this counter is
978 updated, the received packet won't be treated 982 updated, the received packet won't be treated as a SYN cookie and the
979 TcpExtSyncookiesRecv counter won't be updated. !! 983 TcpExtSyncookiesRecv counter wont be updated.
980 984
981 Challenge ACK 985 Challenge ACK
982 ============= 986 =============
983 For details of challenge ACK, please refer the 987 For details of challenge ACK, please refer the explanation of
984 TcpExtTCPACKSkippedChallenge. 988 TcpExtTCPACKSkippedChallenge.
985 989
986 * TcpExtTCPChallengeACK 990 * TcpExtTCPChallengeACK
987 991
988 The number of challenge acks sent. 992 The number of challenge acks sent.
989 993
990 * TcpExtTCPSYNChallenge 994 * TcpExtTCPSYNChallenge
991 995
992 The number of challenge acks sent in response 996 The number of challenge acks sent in response to SYN packets. After
993 updates this counter, the TCP stack might send 997 updates this counter, the TCP stack might send a challenge ACK and
994 update the TcpExtTCPChallengeACK counter, or i 998 update the TcpExtTCPChallengeACK counter, or it might also skip to
995 send the challenge and update the TcpExtTCPACK 999 send the challenge and update the TcpExtTCPACKSkippedChallenge.
996 1000
997 prune 1001 prune
998 ===== 1002 =====
999 When a socket is under memory pressure, the TC 1003 When a socket is under memory pressure, the TCP stack will try to
1000 reclaim memory from the receiving queue and o 1004 reclaim memory from the receiving queue and out of order queue. One of
1001 the reclaiming method is 'collapse', which me 1005 the reclaiming method is 'collapse', which means allocate a big skb,
1002 copy the contiguous skbs to the single big sk 1006 copy the contiguous skbs to the single big skb, and free these
1003 contiguous skbs. 1007 contiguous skbs.
1004 1008
1005 * TcpExtPruneCalled 1009 * TcpExtPruneCalled
1006 1010
1007 The TCP stack tries to reclaim memory for a s 1011 The TCP stack tries to reclaim memory for a socket. After updates this
1008 counter, the TCP stack will try to collapse t 1012 counter, the TCP stack will try to collapse the out of order queue and
1009 the receiving queue. If the memory is still n 1013 the receiving queue. If the memory is still not enough, the TCP stack
1010 will try to discard packets from the out of o 1014 will try to discard packets from the out of order queue (and update the
1011 TcpExtOfoPruned counter) 1015 TcpExtOfoPruned counter)
1012 1016
1013 * TcpExtOfoPruned 1017 * TcpExtOfoPruned
1014 1018
1015 The TCP stack tries to discard packet on the 1019 The TCP stack tries to discard packet on the out of order queue.
1016 1020
1017 * TcpExtRcvPruned 1021 * TcpExtRcvPruned
1018 1022
1019 After 'collapse' and discard packets from the 1023 After 'collapse' and discard packets from the out of order queue, if
1020 the actually used memory is still larger than 1024 the actually used memory is still larger than the max allowed memory,
1021 this counter will be updated. It means the 'p 1025 this counter will be updated. It means the 'prune' fails.
1022 1026
1023 * TcpExtTCPRcvCollapsed 1027 * TcpExtTCPRcvCollapsed
1024 1028
1025 This counter indicates how many skbs are free 1029 This counter indicates how many skbs are freed during 'collapse'.
1026 1030
1027 examples 1031 examples
1028 ======== 1032 ========
1029 1033
1030 ping test 1034 ping test
1031 --------- 1035 ---------
1032 Run the ping command against the public dns s 1036 Run the ping command against the public dns server 8.8.8.8::
1033 1037
1034 nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1 1038 nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
1035 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data 1039 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
1036 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 t 1040 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
1037 1041
1038 --- 8.8.8.8 ping statistics --- 1042 --- 8.8.8.8 ping statistics ---
1039 1 packets transmitted, 1 received, 0% packe 1043 1 packets transmitted, 1 received, 0% packet loss, time 0ms
1040 rtt min/avg/max/mdev = 17.875/17.875/17.875 1044 rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
1041 1045
1042 The nstayt result:: 1046 The nstayt result::
1043 1047
1044 nstatuser@nstat-a:~$ nstat 1048 nstatuser@nstat-a:~$ nstat
1045 #kernel 1049 #kernel
1046 IpInReceives 1 1050 IpInReceives 1 0.0
1047 IpInDelivers 1 1051 IpInDelivers 1 0.0
1048 IpOutRequests 1 1052 IpOutRequests 1 0.0
1049 IcmpInMsgs 1 1053 IcmpInMsgs 1 0.0
1050 IcmpInEchoReps 1 1054 IcmpInEchoReps 1 0.0
1051 IcmpOutMsgs 1 1055 IcmpOutMsgs 1 0.0
1052 IcmpOutEchos 1 1056 IcmpOutEchos 1 0.0
1053 IcmpMsgInType0 1 1057 IcmpMsgInType0 1 0.0
1054 IcmpMsgOutType8 1 1058 IcmpMsgOutType8 1 0.0
1055 IpExtInOctets 84 1059 IpExtInOctets 84 0.0
1056 IpExtOutOctets 84 1060 IpExtOutOctets 84 0.0
1057 IpExtInNoECTPkts 1 1061 IpExtInNoECTPkts 1 0.0
1058 1062
1059 The Linux server sent an ICMP Echo packet, so 1063 The Linux server sent an ICMP Echo packet, so IpOutRequests,
1060 IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 1064 IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
1061 server got ICMP Echo Reply from 8.8.8.8, so I 1065 server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
1062 IcmpInEchoReps and IcmpMsgInType0 were increa 1066 IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
1063 was passed to the ICMP layer via IP layer, so 1067 was passed to the ICMP layer via IP layer, so IpInDelivers was
1064 increased 1. The default ping data size is 48 1068 increased 1. The default ping data size is 48, so an ICMP Echo packet
1065 and its corresponding Echo Reply packet are c 1069 and its corresponding Echo Reply packet are constructed by:
1066 1070
1067 * 14 bytes MAC header 1071 * 14 bytes MAC header
1068 * 20 bytes IP header 1072 * 20 bytes IP header
1069 * 16 bytes ICMP header 1073 * 16 bytes ICMP header
1070 * 48 bytes data (default value of the ping co 1074 * 48 bytes data (default value of the ping command)
1071 1075
1072 So the IpExtInOctets and IpExtOutOctets are 2 1076 So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
1073 1077
1074 tcp 3-way handshake 1078 tcp 3-way handshake
1075 ------------------- 1079 -------------------
1076 On server side, we run:: 1080 On server side, we run::
1077 1081
1078 nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000 1082 nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
1079 Listening on [0.0.0.0] (family 0, port 9000 1083 Listening on [0.0.0.0] (family 0, port 9000)
1080 1084
1081 On client side, we run:: 1085 On client side, we run::
1082 1086
1083 nstatuser@nstat-a:~$ nc -nv 192.168.122.251 1087 nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
1084 Connection to 192.168.122.251 9000 port [tc 1088 Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
1085 1089
1086 The server listened on tcp 9000 port, the cli 1090 The server listened on tcp 9000 port, the client connected to it, they
1087 completed the 3-way handshake. 1091 completed the 3-way handshake.
1088 1092
1089 On server side, we can find below nstat outpu 1093 On server side, we can find below nstat output::
1090 1094
1091 nstatuser@nstat-b:~$ nstat | grep -i tcp 1095 nstatuser@nstat-b:~$ nstat | grep -i tcp
1092 TcpPassiveOpens 1 1096 TcpPassiveOpens 1 0.0
1093 TcpInSegs 2 1097 TcpInSegs 2 0.0
1094 TcpOutSegs 1 1098 TcpOutSegs 1 0.0
1095 TcpExtTCPPureAcks 1 1099 TcpExtTCPPureAcks 1 0.0
1096 1100
1097 On client side, we can find below nstat outpu 1101 On client side, we can find below nstat output::
1098 1102
1099 nstatuser@nstat-a:~$ nstat | grep -i tcp 1103 nstatuser@nstat-a:~$ nstat | grep -i tcp
1100 TcpActiveOpens 1 1104 TcpActiveOpens 1 0.0
1101 TcpInSegs 1 1105 TcpInSegs 1 0.0
1102 TcpOutSegs 2 1106 TcpOutSegs 2 0.0
1103 1107
1104 When the server received the first SYN, it re 1108 When the server received the first SYN, it replied a SYN+ACK, and came into
1105 SYN-RCVD state, so TcpPassiveOpens increased 1109 SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
1106 SYN, sent SYN+ACK, received ACK, so server se 1110 SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
1107 packets, TcpInSegs increased 2, TcpOutSegs in 1111 packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
1108 of the 3-way handshake is a pure ACK without 1112 of the 3-way handshake is a pure ACK without data, so
1109 TcpExtTCPPureAcks increased 1. 1113 TcpExtTCPPureAcks increased 1.
1110 1114
1111 When the client sent SYN, the client came int 1115 When the client sent SYN, the client came into the SYN-SENT state, so
1112 TcpActiveOpens increased 1, the client sent S 1116 TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
1113 ACK, so client sent 2 packets, received 1 pac 1117 ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
1114 1, TcpOutSegs increased 2. 1118 1, TcpOutSegs increased 2.
1115 1119
1116 TCP normal traffic 1120 TCP normal traffic
1117 ------------------ 1121 ------------------
1118 Run nc on server:: 1122 Run nc on server::
1119 1123
1120 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 1124 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1121 Listening on [0.0.0.0] (family 0, port 9000 1125 Listening on [0.0.0.0] (family 0, port 9000)
1122 1126
1123 Run nc on client:: 1127 Run nc on client::
1124 1128
1125 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1129 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1126 Connection to nstat-b 9000 port [tcp/*] suc 1130 Connection to nstat-b 9000 port [tcp/*] succeeded!
1127 1131
1128 Input a string in the nc client ('hello' in o 1132 Input a string in the nc client ('hello' in our example)::
1129 1133
1130 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1134 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1131 Connection to nstat-b 9000 port [tcp/*] suc 1135 Connection to nstat-b 9000 port [tcp/*] succeeded!
1132 hello 1136 hello
1133 1137
1134 The client side nstat output:: 1138 The client side nstat output::
1135 1139
1136 nstatuser@nstat-a:~$ nstat 1140 nstatuser@nstat-a:~$ nstat
1137 #kernel 1141 #kernel
1138 IpInReceives 1 1142 IpInReceives 1 0.0
1139 IpInDelivers 1 1143 IpInDelivers 1 0.0
1140 IpOutRequests 1 1144 IpOutRequests 1 0.0
1141 TcpInSegs 1 1145 TcpInSegs 1 0.0
1142 TcpOutSegs 1 1146 TcpOutSegs 1 0.0
1143 TcpExtTCPPureAcks 1 1147 TcpExtTCPPureAcks 1 0.0
1144 TcpExtTCPOrigDataSent 1 1148 TcpExtTCPOrigDataSent 1 0.0
1145 IpExtInOctets 52 1149 IpExtInOctets 52 0.0
1146 IpExtOutOctets 58 1150 IpExtOutOctets 58 0.0
1147 IpExtInNoECTPkts 1 1151 IpExtInNoECTPkts 1 0.0
1148 1152
1149 The server side nstat output:: 1153 The server side nstat output::
1150 1154
1151 nstatuser@nstat-b:~$ nstat 1155 nstatuser@nstat-b:~$ nstat
1152 #kernel 1156 #kernel
1153 IpInReceives 1 1157 IpInReceives 1 0.0
1154 IpInDelivers 1 1158 IpInDelivers 1 0.0
1155 IpOutRequests 1 1159 IpOutRequests 1 0.0
1156 TcpInSegs 1 1160 TcpInSegs 1 0.0
1157 TcpOutSegs 1 1161 TcpOutSegs 1 0.0
1158 IpExtInOctets 58 1162 IpExtInOctets 58 0.0
1159 IpExtOutOctets 52 1163 IpExtOutOctets 52 0.0
1160 IpExtInNoECTPkts 1 1164 IpExtInNoECTPkts 1 0.0
1161 1165
1162 Input a string in nc client side again ('worl 1166 Input a string in nc client side again ('world' in our example)::
1163 1167
1164 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1168 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1165 Connection to nstat-b 9000 port [tcp/*] suc 1169 Connection to nstat-b 9000 port [tcp/*] succeeded!
1166 hello 1170 hello
1167 world 1171 world
1168 1172
1169 Client side nstat output:: 1173 Client side nstat output::
1170 1174
1171 nstatuser@nstat-a:~$ nstat 1175 nstatuser@nstat-a:~$ nstat
1172 #kernel 1176 #kernel
1173 IpInReceives 1 1177 IpInReceives 1 0.0
1174 IpInDelivers 1 1178 IpInDelivers 1 0.0
1175 IpOutRequests 1 1179 IpOutRequests 1 0.0
1176 TcpInSegs 1 1180 TcpInSegs 1 0.0
1177 TcpOutSegs 1 1181 TcpOutSegs 1 0.0
1178 TcpExtTCPHPAcks 1 1182 TcpExtTCPHPAcks 1 0.0
1179 TcpExtTCPOrigDataSent 1 1183 TcpExtTCPOrigDataSent 1 0.0
1180 IpExtInOctets 52 1184 IpExtInOctets 52 0.0
1181 IpExtOutOctets 58 1185 IpExtOutOctets 58 0.0
1182 IpExtInNoECTPkts 1 1186 IpExtInNoECTPkts 1 0.0
1183 1187
1184 1188
1185 Server side nstat output:: 1189 Server side nstat output::
1186 1190
1187 nstatuser@nstat-b:~$ nstat 1191 nstatuser@nstat-b:~$ nstat
1188 #kernel 1192 #kernel
1189 IpInReceives 1 1193 IpInReceives 1 0.0
1190 IpInDelivers 1 1194 IpInDelivers 1 0.0
1191 IpOutRequests 1 1195 IpOutRequests 1 0.0
1192 TcpInSegs 1 1196 TcpInSegs 1 0.0
1193 TcpOutSegs 1 1197 TcpOutSegs 1 0.0
1194 TcpExtTCPHPHits 1 1198 TcpExtTCPHPHits 1 0.0
1195 IpExtInOctets 58 1199 IpExtInOctets 58 0.0
1196 IpExtOutOctets 52 1200 IpExtOutOctets 52 0.0
1197 IpExtInNoECTPkts 1 1201 IpExtInNoECTPkts 1 0.0
1198 1202
1199 Compare the first client-side nstat and the s 1203 Compare the first client-side nstat and the second client-side nstat,
1200 we could find one difference: the first one h 1204 we could find one difference: the first one had a 'TcpExtTCPPureAcks',
1201 but the second one had a 'TcpExtTCPHPAcks'. T 1205 but the second one had a 'TcpExtTCPHPAcks'. The first server-side
1202 nstat and the second server-side nstat had a 1206 nstat and the second server-side nstat had a difference too: the
1203 second server-side nstat had a TcpExtTCPHPHit 1207 second server-side nstat had a TcpExtTCPHPHits, but the first
1204 server-side nstat didn't have it. The network 1208 server-side nstat didn't have it. The network traffic patterns were
1205 exactly the same: the client sent a packet to 1209 exactly the same: the client sent a packet to the server, the server
1206 replied an ACK. But kernel handled them in di 1210 replied an ACK. But kernel handled them in different ways. When the
1207 TCP window scale option is not used, kernel w 1211 TCP window scale option is not used, kernel will try to enable fast
1208 path immediately when the connection comes in 1212 path immediately when the connection comes into the established state,
1209 but if the TCP window scale option is used, k 1213 but if the TCP window scale option is used, kernel will disable the
1210 fast path at first, and try to enable it afte 1214 fast path at first, and try to enable it after kernel receives
1211 packets. We could use the 'ss' command to ver 1215 packets. We could use the 'ss' command to verify whether the window
1212 scale option is used. e.g. run below command 1216 scale option is used. e.g. run below command on either server or
1213 client:: 1217 client::
1214 1218
1215 nstatuser@nstat-a:~$ ss -o state establishe 1219 nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
1216 Netid Recv-Q Send-Q Local 1220 Netid Recv-Q Send-Q Local Address:Port Peer Address:Port
1217 tcp 0 0 192.168 1221 tcp 0 0 192.168.122.250:40654 192.168.122.251:9000
1218 ts sack cubic wscale:7,7 rto:204 1222 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98
1219 1223
1220 The 'wscale:7,7' means both server and client 1224 The 'wscale:7,7' means both server and client set the window scale
1221 option to 7. Now we could explain the nstat o 1225 option to 7. Now we could explain the nstat output in our test:
1222 1226
1223 In the first nstat output of client side, the 1227 In the first nstat output of client side, the client sent a packet, server
1224 reply an ACK, when kernel handled this ACK, t 1228 reply an ACK, when kernel handled this ACK, the fast path was not
1225 enabled, so the ACK was counted into 'TcpExtT 1229 enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.
1226 1230
1227 In the second nstat output of client side, th 1231 In the second nstat output of client side, the client sent a packet again,
1228 and received another ACK from the server, in 1232 and received another ACK from the server, in this time, the fast path is
1229 enabled, and the ACK was qualified for fast p 1233 enabled, and the ACK was qualified for fast path, so it was handled by
1230 the fast path, so this ACK was counted into T 1234 the fast path, so this ACK was counted into TcpExtTCPHPAcks.
1231 1235
1232 In the first nstat output of server side, fas 1236 In the first nstat output of server side, fast path was not enabled,
1233 so there was no 'TcpExtTCPHPHits'. 1237 so there was no 'TcpExtTCPHPHits'.
1234 1238
1235 In the second nstat output of server side, th 1239 In the second nstat output of server side, the fast path was enabled,
1236 and the packet received from client qualified 1240 and the packet received from client qualified for fast path, so it
1237 was counted into 'TcpExtTCPHPHits'. 1241 was counted into 'TcpExtTCPHPHits'.
1238 1242
1239 TcpExtTCPAbortOnClose 1243 TcpExtTCPAbortOnClose
1240 --------------------- 1244 ---------------------
1241 On the server side, we run below python scrip 1245 On the server side, we run below python script::
1242 1246
1243 import socket 1247 import socket
1244 import time 1248 import time
1245 1249
1246 port = 9000 1250 port = 9000
1247 1251
1248 s = socket.socket(socket.AF_INET, socket.SO 1252 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1249 s.bind(('0.0.0.0', port)) 1253 s.bind(('0.0.0.0', port))
1250 s.listen(1) 1254 s.listen(1)
1251 sock, addr = s.accept() 1255 sock, addr = s.accept()
1252 while True: 1256 while True:
1253 time.sleep(9999999) 1257 time.sleep(9999999)
1254 1258
1255 This python script listen on 9000 port, but d 1259 This python script listen on 9000 port, but doesn't read anything from
1256 the connection. 1260 the connection.
1257 1261
1258 On the client side, we send the string "hello 1262 On the client side, we send the string "hello" by nc::
1259 1263
1260 nstatuser@nstat-a:~$ echo "hello" | nc nsta 1264 nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
1261 1265
1262 Then, we come back to the server side, the se 1266 Then, we come back to the server side, the server has received the "hello"
1263 packet, and the TCP layer has acked this pack 1267 packet, and the TCP layer has acked this packet, but the application didn't
1264 read it yet. We type Ctrl-C to terminate the 1268 read it yet. We type Ctrl-C to terminate the server script. Then we
1265 could find TcpExtTCPAbortOnClose increased 1 1269 could find TcpExtTCPAbortOnClose increased 1 on the server side::
1266 1270
1267 nstatuser@nstat-b:~$ nstat | grep -i abort 1271 nstatuser@nstat-b:~$ nstat | grep -i abort
1268 TcpExtTCPAbortOnClose 1 1272 TcpExtTCPAbortOnClose 1 0.0
1269 1273
1270 If we run tcpdump on the server side, we coul 1274 If we run tcpdump on the server side, we could find the server sent a
1271 RST after we type Ctrl-C. 1275 RST after we type Ctrl-C.
1272 1276
1273 TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTi 1277 TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout
1274 --------------------------------------------- 1278 ---------------------------------------------------
1275 Below is an example which let the orphan sock 1279 Below is an example which let the orphan socket count be higher than
1276 net.ipv4.tcp_max_orphans. 1280 net.ipv4.tcp_max_orphans.
1277 Change tcp_max_orphans to a smaller value on 1281 Change tcp_max_orphans to a smaller value on client::
1278 1282
1279 sudo bash -c "echo 10 > /proc/sys/net/ipv4/ 1283 sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
1280 1284
1281 Client code (create 64 connection to server): 1285 Client code (create 64 connection to server)::
1282 1286
1283 nstatuser@nstat-a:~$ cat client_orphan.py 1287 nstatuser@nstat-a:~$ cat client_orphan.py
1284 import socket 1288 import socket
1285 import time 1289 import time
1286 1290
1287 server = 'nstat-b' # server address 1291 server = 'nstat-b' # server address
1288 port = 9000 1292 port = 9000
1289 1293
1290 count = 64 1294 count = 64
1291 1295
1292 connection_list = [] 1296 connection_list = []
1293 1297
1294 for i in range(64): 1298 for i in range(64):
1295 s = socket.socket(socket.AF_INET, socke 1299 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1296 s.connect((server, port)) 1300 s.connect((server, port))
1297 connection_list.append(s) 1301 connection_list.append(s)
1298 print("connection_count: %d" % len(conn 1302 print("connection_count: %d" % len(connection_list))
1299 1303
1300 while True: 1304 while True:
1301 time.sleep(99999) 1305 time.sleep(99999)
1302 1306
1303 Server code (accept 64 connection from client 1307 Server code (accept 64 connection from client)::
1304 1308
1305 nstatuser@nstat-b:~$ cat server_orphan.py 1309 nstatuser@nstat-b:~$ cat server_orphan.py
1306 import socket 1310 import socket
1307 import time 1311 import time
1308 1312
1309 port = 9000 1313 port = 9000
1310 count = 64 1314 count = 64
1311 1315
1312 s = socket.socket(socket.AF_INET, socket.SO 1316 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1313 s.bind(('0.0.0.0', port)) 1317 s.bind(('0.0.0.0', port))
1314 s.listen(count) 1318 s.listen(count)
1315 connection_list = [] 1319 connection_list = []
1316 while True: 1320 while True:
1317 sock, addr = s.accept() 1321 sock, addr = s.accept()
1318 connection_list.append((sock, addr)) 1322 connection_list.append((sock, addr))
1319 print("connection_count: %d" % len(conn 1323 print("connection_count: %d" % len(connection_list))
1320 1324
1321 Run the python scripts on server and client. 1325 Run the python scripts on server and client.
1322 1326
1323 On server:: 1327 On server::
1324 1328
1325 python3 server_orphan.py 1329 python3 server_orphan.py
1326 1330
1327 On client:: 1331 On client::
1328 1332
1329 python3 client_orphan.py 1333 python3 client_orphan.py
1330 1334
1331 Run iptables on server:: 1335 Run iptables on server::
1332 1336
1333 sudo iptables -A INPUT -i ens3 -p tcp --des 1337 sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
1334 1338
1335 Type Ctrl-C on client, stop client_orphan.py. 1339 Type Ctrl-C on client, stop client_orphan.py.
1336 1340
1337 Check TcpExtTCPAbortOnMemory on client:: 1341 Check TcpExtTCPAbortOnMemory on client::
1338 1342
1339 nstatuser@nstat-a:~$ nstat | grep -i abort 1343 nstatuser@nstat-a:~$ nstat | grep -i abort
1340 TcpExtTCPAbortOnMemory 54 1344 TcpExtTCPAbortOnMemory 54 0.0
1341 1345
1342 Check orphaned socket count on client:: 1346 Check orphaned socket count on client::
1343 1347
1344 nstatuser@nstat-a:~$ ss -s 1348 nstatuser@nstat-a:~$ ss -s
1345 Total: 131 (kernel 0) 1349 Total: 131 (kernel 0)
1346 TCP: 14 (estab 1, closed 0, orphaned 10, 1350 TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0
1347 1351
1348 Transport Total IP IPv6 1352 Transport Total IP IPv6
1349 * 0 - - 1353 * 0 - -
1350 RAW 1 0 1 1354 RAW 1 0 1
1351 UDP 1 1 0 1355 UDP 1 1 0
1352 TCP 14 13 1 1356 TCP 14 13 1
1353 INET 16 14 2 1357 INET 16 14 2
1354 FRAG 0 0 0 1358 FRAG 0 0 0
1355 1359
1356 The explanation of the test: after run server 1360 The explanation of the test: after run server_orphan.py and
1357 client_orphan.py, we set up 64 connections be 1361 client_orphan.py, we set up 64 connections between server and
1358 client. Run the iptables command, the server 1362 client. Run the iptables command, the server will drop all packets from
1359 the client, type Ctrl-C on client_orphan.py, 1363 the client, type Ctrl-C on client_orphan.py, the system of the client
1360 would try to close these connections, and bef 1364 would try to close these connections, and before they are closed
1361 gracefully, these connections became orphan s 1365 gracefully, these connections became orphan sockets. As the iptables
1362 of the server blocked packets from the client 1366 of the server blocked packets from the client, the server won't receive fin
1363 from the client, so all connection on clients 1367 from the client, so all connection on clients would be stuck on FIN_WAIT_1
1364 stage, so they will keep as orphan sockets un 1368 stage, so they will keep as orphan sockets until timeout. We have echo
1365 10 to /proc/sys/net/ipv4/tcp_max_orphans, so 1369 10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
1366 only keep 10 orphan sockets, for all other or 1370 only keep 10 orphan sockets, for all other orphan sockets, the client
1367 system sent RST for them and delete them. We 1371 system sent RST for them and delete them. We have 64 connections, so
1368 the 'ss -s' command shows the system has 10 o 1372 the 'ss -s' command shows the system has 10 orphan sockets, and the
1369 value of TcpExtTCPAbortOnMemory was 54. 1373 value of TcpExtTCPAbortOnMemory was 54.
1370 1374
1371 An additional explanation about orphan socket 1375 An additional explanation about orphan socket count: You could find the
1372 exactly orphan socket count by the 'ss -s' co 1376 exactly orphan socket count by the 'ss -s' command, but when kernel
1373 decide whither increases TcpExtTCPAbortOnMemo 1377 decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel
1374 doesn't always check the exactly orphan socke 1378 doesn't always check the exactly orphan socket count. For increasing
1375 performance, kernel checks an approximate cou 1379 performance, kernel checks an approximate count firstly, if the
1376 approximate count is more than tcp_max_orphan 1380 approximate count is more than tcp_max_orphans, kernel checks the
1377 exact count again. So if the approximate coun 1381 exact count again. So if the approximate count is less than
1378 tcp_max_orphans, but exactly count is more th 1382 tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
1379 would find TcpExtTCPAbortOnMemory is not incr 1383 would find TcpExtTCPAbortOnMemory is not increased at all. If
1380 tcp_max_orphans is large enough, it won't occ 1384 tcp_max_orphans is large enough, it won't occur, but if you decrease
1381 tcp_max_orphans to a small value like our tes 1385 tcp_max_orphans to a small value like our test, you might find this
1382 issue. So in our test, the client set up 64 c 1386 issue. So in our test, the client set up 64 connections although the
1383 tcp_max_orphans is 10. If the client only set 1387 tcp_max_orphans is 10. If the client only set up 11 connections, we
1384 can't find the change of TcpExtTCPAbortOnMemo 1388 can't find the change of TcpExtTCPAbortOnMemory.
1385 1389
1386 Continue the previous test, we wait for sever 1390 Continue the previous test, we wait for several minutes. Because of the
1387 iptables on the server blocked the traffic, t 1391 iptables on the server blocked the traffic, the server wouldn't receive
1388 fin, and all the client's orphan sockets woul 1392 fin, and all the client's orphan sockets would timeout on the
1389 FIN_WAIT_1 state finally. So we wait for a fe 1393 FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
1390 10 timeout on the client:: 1394 10 timeout on the client::
1391 1395
1392 nstatuser@nstat-a:~$ nstat | grep -i abort 1396 nstatuser@nstat-a:~$ nstat | grep -i abort
1393 TcpExtTCPAbortOnTimeout 10 1397 TcpExtTCPAbortOnTimeout 10 0.0
1394 1398
1395 TcpExtTCPAbortOnLinger 1399 TcpExtTCPAbortOnLinger
1396 ---------------------- 1400 ----------------------
1397 The server side code:: 1401 The server side code::
1398 1402
1399 nstatuser@nstat-b:~$ cat server_linger.py 1403 nstatuser@nstat-b:~$ cat server_linger.py
1400 import socket 1404 import socket
1401 import time 1405 import time
1402 1406
1403 port = 9000 1407 port = 9000
1404 1408
1405 s = socket.socket(socket.AF_INET, socket.SO 1409 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1406 s.bind(('0.0.0.0', port)) 1410 s.bind(('0.0.0.0', port))
1407 s.listen(1) 1411 s.listen(1)
1408 sock, addr = s.accept() 1412 sock, addr = s.accept()
1409 while True: 1413 while True:
1410 time.sleep(9999999) 1414 time.sleep(9999999)
1411 1415
1412 The client side code:: 1416 The client side code::
1413 1417
1414 nstatuser@nstat-a:~$ cat client_linger.py 1418 nstatuser@nstat-a:~$ cat client_linger.py
1415 import socket 1419 import socket
1416 import struct 1420 import struct
1417 1421
1418 server = 'nstat-b' # server address 1422 server = 'nstat-b' # server address
1419 port = 9000 1423 port = 9000
1420 1424
1421 s = socket.socket(socket.AF_INET, socket.SO 1425 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1422 s.setsockopt(socket.SOL_SOCKET, socket.SO_L 1426 s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
1423 s.setsockopt(socket.SOL_TCP, socket.TCP_LIN 1427 s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
1424 s.connect((server, port)) 1428 s.connect((server, port))
1425 s.close() 1429 s.close()
1426 1430
1427 Run server_linger.py on server:: 1431 Run server_linger.py on server::
1428 1432
1429 nstatuser@nstat-b:~$ python3 server_linger. 1433 nstatuser@nstat-b:~$ python3 server_linger.py
1430 1434
1431 Run client_linger.py on client:: 1435 Run client_linger.py on client::
1432 1436
1433 nstatuser@nstat-a:~$ python3 client_linger. 1437 nstatuser@nstat-a:~$ python3 client_linger.py
1434 1438
1435 After run client_linger.py, check the output 1439 After run client_linger.py, check the output of nstat::
1436 1440
1437 nstatuser@nstat-a:~$ nstat | grep -i abort 1441 nstatuser@nstat-a:~$ nstat | grep -i abort
1438 TcpExtTCPAbortOnLinger 1 1442 TcpExtTCPAbortOnLinger 1 0.0
1439 1443
1440 TcpExtTCPRcvCoalesce 1444 TcpExtTCPRcvCoalesce
1441 -------------------- 1445 --------------------
1442 On the server, we run a program which listen 1446 On the server, we run a program which listen on TCP port 9000, but
1443 doesn't read any data:: 1447 doesn't read any data::
1444 1448
1445 import socket 1449 import socket
1446 import time 1450 import time
1447 port = 9000 1451 port = 9000
1448 s = socket.socket(socket.AF_INET, socket.SO 1452 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1449 s.bind(('0.0.0.0', port)) 1453 s.bind(('0.0.0.0', port))
1450 s.listen(1) 1454 s.listen(1)
1451 sock, addr = s.accept() 1455 sock, addr = s.accept()
1452 while True: 1456 while True:
1453 time.sleep(9999999) 1457 time.sleep(9999999)
1454 1458
1455 Save the above code as server_coalesce.py, an 1459 Save the above code as server_coalesce.py, and run::
1456 1460
1457 python3 server_coalesce.py 1461 python3 server_coalesce.py
1458 1462
1459 On the client, save below code as client_coal 1463 On the client, save below code as client_coalesce.py::
1460 1464
1461 import socket 1465 import socket
1462 server = 'nstat-b' 1466 server = 'nstat-b'
1463 port = 9000 1467 port = 9000
1464 s = socket.socket(socket.AF_INET, socket.SO 1468 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1465 s.connect((server, port)) 1469 s.connect((server, port))
1466 1470
1467 Run:: 1471 Run::
1468 1472
1469 nstatuser@nstat-a:~$ python3 -i client_coal 1473 nstatuser@nstat-a:~$ python3 -i client_coalesce.py
1470 1474
1471 We use '-i' to come into the interactive mode 1475 We use '-i' to come into the interactive mode, then a packet::
1472 1476
1473 >>> s.send(b'foo') 1477 >>> s.send(b'foo')
1474 3 1478 3
1475 1479
1476 Send a packet again:: 1480 Send a packet again::
1477 1481
1478 >>> s.send(b'bar') 1482 >>> s.send(b'bar')
1479 3 1483 3
1480 1484
1481 On the server, run nstat:: 1485 On the server, run nstat::
1482 1486
1483 ubuntu@nstat-b:~$ nstat 1487 ubuntu@nstat-b:~$ nstat
1484 #kernel 1488 #kernel
1485 IpInReceives 2 1489 IpInReceives 2 0.0
1486 IpInDelivers 2 1490 IpInDelivers 2 0.0
1487 IpOutRequests 2 1491 IpOutRequests 2 0.0
1488 TcpInSegs 2 1492 TcpInSegs 2 0.0
1489 TcpOutSegs 2 1493 TcpOutSegs 2 0.0
1490 TcpExtTCPRcvCoalesce 1 1494 TcpExtTCPRcvCoalesce 1 0.0
1491 IpExtInOctets 110 1495 IpExtInOctets 110 0.0
1492 IpExtOutOctets 104 1496 IpExtOutOctets 104 0.0
1493 IpExtInNoECTPkts 2 1497 IpExtInNoECTPkts 2 0.0
1494 1498
1495 The client sent two packets, server didn't re 1499 The client sent two packets, server didn't read any data. When
1496 the second packet arrived at server, the firs 1500 the second packet arrived at server, the first packet was still in
1497 the receiving queue. So the TCP layer merged 1501 the receiving queue. So the TCP layer merged the two packets, and we
1498 could find the TcpExtTCPRcvCoalesce increased 1502 could find the TcpExtTCPRcvCoalesce increased 1.
1499 1503
1500 TcpExtListenOverflows and TcpExtListenDrops 1504 TcpExtListenOverflows and TcpExtListenDrops
1501 ------------------------------------------- 1505 -------------------------------------------
1502 On server, run the nc command, listen on port 1506 On server, run the nc command, listen on port 9000::
1503 1507
1504 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 1508 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1505 Listening on [0.0.0.0] (family 0, port 9000 1509 Listening on [0.0.0.0] (family 0, port 9000)
1506 1510
1507 On client, run 3 nc commands in different ter 1511 On client, run 3 nc commands in different terminals::
1508 1512
1509 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1513 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1510 Connection to nstat-b 9000 port [tcp/*] suc 1514 Connection to nstat-b 9000 port [tcp/*] succeeded!
1511 1515
1512 The nc command only accepts 1 connection, and 1516 The nc command only accepts 1 connection, and the accept queue length
1513 is 1. On current linux implementation, set qu 1517 is 1. On current linux implementation, set queue length to n means the
1514 actual queue length is n+1. Now we create 3 c 1518 actual queue length is n+1. Now we create 3 connections, 1 is accepted
1515 by nc, 2 in accepted queue, so the accept que 1519 by nc, 2 in accepted queue, so the accept queue is full.
1516 1520
1517 Before running the 4th nc, we clean the nstat 1521 Before running the 4th nc, we clean the nstat history on the server::
1518 1522
1519 nstatuser@nstat-b:~$ nstat -n 1523 nstatuser@nstat-b:~$ nstat -n
1520 1524
1521 Run the 4th nc on the client:: 1525 Run the 4th nc on the client::
1522 1526
1523 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1527 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1524 1528
1525 If the nc server is running on kernel 4.10 or 1529 If the nc server is running on kernel 4.10 or higher version, you
1526 won't see the "Connection to ... succeeded!" 1530 won't see the "Connection to ... succeeded!" string, because kernel
1527 will drop the SYN if the accept queue is full 1531 will drop the SYN if the accept queue is full. If the nc client is running
1528 on an old kernel, you would see that the conn 1532 on an old kernel, you would see that the connection is succeeded,
1529 because kernel would complete the 3 way hands 1533 because kernel would complete the 3 way handshake and keep the socket
1530 on half open queue. I did the test on kernel 1534 on half open queue. I did the test on kernel 4.15. Below is the nstat
1531 on the server:: 1535 on the server::
1532 1536
1533 nstatuser@nstat-b:~$ nstat 1537 nstatuser@nstat-b:~$ nstat
1534 #kernel 1538 #kernel
1535 IpInReceives 4 1539 IpInReceives 4 0.0
1536 IpInDelivers 4 1540 IpInDelivers 4 0.0
1537 TcpInSegs 4 1541 TcpInSegs 4 0.0
1538 TcpExtListenOverflows 4 1542 TcpExtListenOverflows 4 0.0
1539 TcpExtListenDrops 4 1543 TcpExtListenDrops 4 0.0
1540 IpExtInOctets 240 1544 IpExtInOctets 240 0.0
1541 IpExtInNoECTPkts 4 1545 IpExtInNoECTPkts 4 0.0
1542 1546
1543 Both TcpExtListenOverflows and TcpExtListenDr 1547 Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time
1544 between the 4th nc and the nstat was longer, 1548 between the 4th nc and the nstat was longer, the value of
1545 TcpExtListenOverflows and TcpExtListenDrops w 1549 TcpExtListenOverflows and TcpExtListenDrops would be larger, because
1546 the SYN of the 4th nc was dropped, the client 1550 the SYN of the 4th nc was dropped, the client was retrying.
1547 1551
1548 IpInAddrErrors, IpExtInNoRoutes and IpOutNoRo 1552 IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes
1549 --------------------------------------------- 1553 -------------------------------------------------
1550 server A IP address: 192.168.122.250 1554 server A IP address: 192.168.122.250
1551 server B IP address: 192.168.122.251 1555 server B IP address: 192.168.122.251
1552 Prepare on server A, add a route to server B: 1556 Prepare on server A, add a route to server B::
1553 1557
1554 $ sudo ip route add 8.8.8.8/32 via 192.168. 1558 $ sudo ip route add 8.8.8.8/32 via 192.168.122.251
1555 1559
1556 Prepare on server B, disable send_redirects f 1560 Prepare on server B, disable send_redirects for all interfaces::
1557 1561
1558 $ sudo sysctl -w net.ipv4.conf.all.send_red 1562 $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
1559 $ sudo sysctl -w net.ipv4.conf.ens3.send_re 1563 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
1560 $ sudo sysctl -w net.ipv4.conf.lo.send_redi 1564 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
1561 $ sudo sysctl -w net.ipv4.conf.default.send 1565 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
1562 1566
1563 We want to let sever A send a packet to 8.8.8 1567 We want to let sever A send a packet to 8.8.8.8, and route the packet
1564 to server B. When server B receives such pack 1568 to server B. When server B receives such packet, it might send a ICMP
1565 Redirect message to server A, set send_redire 1569 Redirect message to server A, set send_redirects to 0 will disable
1566 this behavior. 1570 this behavior.
1567 1571
1568 First, generate InAddrErrors. On server B, we 1572 First, generate InAddrErrors. On server B, we disable IP forwarding::
1569 1573
1570 $ sudo sysctl -w net.ipv4.conf.all.forwardi 1574 $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
1571 1575
1572 On server A, we send packets to 8.8.8.8:: 1576 On server A, we send packets to 8.8.8.8::
1573 1577
1574 $ nc -v 8.8.8.8 53 1578 $ nc -v 8.8.8.8 53
1575 1579
1576 On server B, we check the output of nstat:: 1580 On server B, we check the output of nstat::
1577 1581
1578 $ nstat 1582 $ nstat
1579 #kernel 1583 #kernel
1580 IpInReceives 3 1584 IpInReceives 3 0.0
1581 IpInAddrErrors 3 1585 IpInAddrErrors 3 0.0
1582 IpExtInOctets 180 1586 IpExtInOctets 180 0.0
1583 IpExtInNoECTPkts 3 1587 IpExtInNoECTPkts 3 0.0
1584 1588
1585 As we have let server A route 8.8.8.8 to serv 1589 As we have let server A route 8.8.8.8 to server B, and we disabled IP
1586 forwarding on server B, Server A sent packets 1590 forwarding on server B, Server A sent packets to server B, then server B
1587 dropped packets and increased IpInAddrErrors. 1591 dropped packets and increased IpInAddrErrors. As the nc command would
1588 re-send the SYN packet if it didn't receive a 1592 re-send the SYN packet if it didn't receive a SYN+ACK, we could find
1589 multiple IpInAddrErrors. 1593 multiple IpInAddrErrors.
1590 1594
1591 Second, generate IpExtInNoRoutes. On server B 1595 Second, generate IpExtInNoRoutes. On server B, we enable IP
1592 forwarding:: 1596 forwarding::
1593 1597
1594 $ sudo sysctl -w net.ipv4.conf.all.forwardi 1598 $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
1595 1599
1596 Check the route table of server B and remove 1600 Check the route table of server B and remove the default route::
1597 1601
1598 $ ip route show 1602 $ ip route show
1599 default via 192.168.122.1 dev ens3 proto st 1603 default via 192.168.122.1 dev ens3 proto static
1600 192.168.122.0/24 dev ens3 proto kernel scop 1604 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251
1601 $ sudo ip route delete default via 192.168. 1605 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static
1602 1606
1603 On server A, we contact 8.8.8.8 again:: 1607 On server A, we contact 8.8.8.8 again::
1604 1608
1605 $ nc -v 8.8.8.8 53 1609 $ nc -v 8.8.8.8 53
1606 nc: connect to 8.8.8.8 port 53 (tcp) failed 1610 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable
1607 1611
1608 On server B, run nstat:: 1612 On server B, run nstat::
1609 1613
1610 $ nstat 1614 $ nstat
1611 #kernel 1615 #kernel
1612 IpInReceives 1 1616 IpInReceives 1 0.0
1613 IpOutRequests 1 1617 IpOutRequests 1 0.0
1614 IcmpOutMsgs 1 1618 IcmpOutMsgs 1 0.0
1615 IcmpOutDestUnreachs 1 1619 IcmpOutDestUnreachs 1 0.0
1616 IcmpMsgOutType3 1 1620 IcmpMsgOutType3 1 0.0
1617 IpExtInNoRoutes 1 1621 IpExtInNoRoutes 1 0.0
1618 IpExtInOctets 60 1622 IpExtInOctets 60 0.0
1619 IpExtOutOctets 88 1623 IpExtOutOctets 88 0.0
1620 IpExtInNoECTPkts 1 1624 IpExtInNoECTPkts 1 0.0
1621 1625
1622 We enabled IP forwarding on server B, when se 1626 We enabled IP forwarding on server B, when server B received a packet
1623 which destination IP address is 8.8.8.8, serv 1627 which destination IP address is 8.8.8.8, server B will try to forward
1624 this packet. We have deleted the default rout 1628 this packet. We have deleted the default route, there was no route for
1625 8.8.8.8, so server B increase IpExtInNoRoutes 1629 8.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP
1626 Destination Unreachable" message to server A. 1630 Destination Unreachable" message to server A.
1627 1631
1628 Third, generate IpOutNoRoutes. Run ping comma 1632 Third, generate IpOutNoRoutes. Run ping command on server B::
1629 1633
1630 $ ping -c 1 8.8.8.8 1634 $ ping -c 1 8.8.8.8
1631 connect: Network is unreachable 1635 connect: Network is unreachable
1632 1636
1633 Run nstat on server B:: 1637 Run nstat on server B::
1634 1638
1635 $ nstat 1639 $ nstat
1636 #kernel 1640 #kernel
1637 IpOutNoRoutes 1 1641 IpOutNoRoutes 1 0.0
1638 1642
1639 We have deleted the default route on server B 1643 We have deleted the default route on server B. Server B couldn't find
1640 a route for the 8.8.8.8 IP address, so server 1644 a route for the 8.8.8.8 IP address, so server B increased
1641 IpOutNoRoutes. 1645 IpOutNoRoutes.
1642 1646
1643 TcpExtTCPACKSkippedSynRecv 1647 TcpExtTCPACKSkippedSynRecv
1644 -------------------------- 1648 --------------------------
1645 In this test, we send 3 same SYN packets from 1649 In this test, we send 3 same SYN packets from client to server. The
1646 first SYN will let server create a socket, se 1650 first SYN will let server create a socket, set it to Syn-Recv status,
1647 and reply a SYN/ACK. The second SYN will let 1651 and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK
1648 again, and record the reply time (the duplica 1652 again, and record the reply time (the duplicate ACK reply time). The
1649 third SYN will let server check the previous 1653 third SYN will let server check the previous duplicate ACK reply time,
1650 and decide to skip the duplicate ACK, then in 1654 and decide to skip the duplicate ACK, then increase the
1651 TcpExtTCPACKSkippedSynRecv counter. 1655 TcpExtTCPACKSkippedSynRecv counter.
1652 1656
1653 Run tcpdump to capture a SYN packet:: 1657 Run tcpdump to capture a SYN packet::
1654 1658
1655 nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w / 1659 nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
1656 tcpdump: listening on ens3, link-type EN10M 1660 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1657 1661
1658 Open another terminal, run nc command:: 1662 Open another terminal, run nc command::
1659 1663
1660 nstatuser@nstat-a:~$ nc nstat-b 9000 1664 nstatuser@nstat-a:~$ nc nstat-b 9000
1661 1665
1662 As the nstat-b didn't listen on port 9000, it 1666 As the nstat-b didn't listen on port 9000, it should reply a RST, and
1663 the nc command exited immediately. It was eno 1667 the nc command exited immediately. It was enough for the tcpdump
1664 command to capture a SYN packet. A linux serv 1668 command to capture a SYN packet. A linux server might use hardware
1665 offload for the TCP checksum, so the checksum 1669 offload for the TCP checksum, so the checksum in the /tmp/syn.pcap
1666 might be not correct. We call tcprewrite to f 1670 might be not correct. We call tcprewrite to fix it::
1667 1671
1668 nstatuser@nstat-a:~$ tcprewrite --infile=/t 1672 nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
1669 1673
1670 On nstat-b, we run nc to listen on port 9000: 1674 On nstat-b, we run nc to listen on port 9000::
1671 1675
1672 nstatuser@nstat-b:~$ nc -lkv 9000 1676 nstatuser@nstat-b:~$ nc -lkv 9000
1673 Listening on [0.0.0.0] (family 0, port 9000 1677 Listening on [0.0.0.0] (family 0, port 9000)
1674 1678
1675 On nstat-a, we blocked the packet from port 9 1679 On nstat-a, we blocked the packet from port 9000, or nstat-a would send
1676 RST to nstat-b:: 1680 RST to nstat-b::
1677 1681
1678 nstatuser@nstat-a:~$ sudo iptables -A INPUT 1682 nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
1679 1683
1680 Send 3 SYN repeatedly to nstat-b:: !! 1684 Send 3 SYN repeatly to nstat-b::
1681 1685
1682 nstatuser@nstat-a:~$ for i in {1..3}; do su 1686 nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
1683 1687
1684 Check snmp counter on nstat-b:: 1688 Check snmp counter on nstat-b::
1685 1689
1686 nstatuser@nstat-b:~$ nstat | grep -i skip 1690 nstatuser@nstat-b:~$ nstat | grep -i skip
1687 TcpExtTCPACKSkippedSynRecv 1 1691 TcpExtTCPACKSkippedSynRecv 1 0.0
1688 1692
1689 As we expected, TcpExtTCPACKSkippedSynRecv is 1693 As we expected, TcpExtTCPACKSkippedSynRecv is 1.
1690 1694
1691 TcpExtTCPACKSkippedPAWS 1695 TcpExtTCPACKSkippedPAWS
1692 ----------------------- 1696 -----------------------
1693 To trigger PAWS, we could send an old SYN. 1697 To trigger PAWS, we could send an old SYN.
1694 1698
1695 On nstat-b, let nc listen on port 9000:: 1699 On nstat-b, let nc listen on port 9000::
1696 1700
1697 nstatuser@nstat-b:~$ nc -lkv 9000 1701 nstatuser@nstat-b:~$ nc -lkv 9000
1698 Listening on [0.0.0.0] (family 0, port 9000 1702 Listening on [0.0.0.0] (family 0, port 9000)
1699 1703
1700 On nstat-a, run tcpdump to capture a SYN:: 1704 On nstat-a, run tcpdump to capture a SYN::
1701 1705
1702 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/p 1706 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
1703 tcpdump: listening on ens3, link-type EN10M 1707 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1704 1708
1705 On nstat-a, run nc as a client to connect nst 1709 On nstat-a, run nc as a client to connect nstat-b::
1706 1710
1707 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1711 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1708 Connection to nstat-b 9000 port [tcp/*] suc 1712 Connection to nstat-b 9000 port [tcp/*] succeeded!
1709 1713
1710 Now the tcpdump has captured the SYN and exit 1714 Now the tcpdump has captured the SYN and exit. We should fix the
1711 checksum:: 1715 checksum::
1712 1716
1713 nstatuser@nstat-a:~$ tcprewrite --infile /t 1717 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
1714 1718
1715 Send the SYN packet twice:: 1719 Send the SYN packet twice::
1716 1720
1717 nstatuser@nstat-a:~$ for i in {1..2}; do su 1721 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
1718 1722
1719 On nstat-b, check the snmp counter:: 1723 On nstat-b, check the snmp counter::
1720 1724
1721 nstatuser@nstat-b:~$ nstat | grep -i skip 1725 nstatuser@nstat-b:~$ nstat | grep -i skip
1722 TcpExtTCPACKSkippedPAWS 1 1726 TcpExtTCPACKSkippedPAWS 1 0.0
1723 1727
1724 We sent two SYN via tcpreplay, both of them w 1728 We sent two SYN via tcpreplay, both of them would let PAWS check
1725 failed, the nstat-b replied an ACK for the fi 1729 failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
1726 for the second SYN, and updated TcpExtTCPACKS 1730 for the second SYN, and updated TcpExtTCPACKSkippedPAWS.
1727 1731
1728 TcpExtTCPACKSkippedSeq 1732 TcpExtTCPACKSkippedSeq
1729 ---------------------- 1733 ----------------------
1730 To trigger TcpExtTCPACKSkippedSeq, we send pa 1734 To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid
1731 timestamp (to pass PAWS check) but the sequen 1735 timestamp (to pass PAWS check) but the sequence number is out of
1732 window. The linux TCP stack would avoid to sk 1736 window. The linux TCP stack would avoid to skip if the packet has
1733 data, so we need a pure ACK packet. To genera 1737 data, so we need a pure ACK packet. To generate such a packet, we
1734 could create two sockets: one on port 9000, a 1738 could create two sockets: one on port 9000, another on port 9001. Then
1735 we capture an ACK on port 9001, change the so 1739 we capture an ACK on port 9001, change the source/destination port
1736 numbers to match the port 9000 socket. Then w 1740 numbers to match the port 9000 socket. Then we could trigger
1737 TcpExtTCPACKSkippedSeq via this packet. 1741 TcpExtTCPACKSkippedSeq via this packet.
1738 1742
1739 On nstat-b, open two terminals, run two nc co 1743 On nstat-b, open two terminals, run two nc commands to listen on both
1740 port 9000 and port 9001:: 1744 port 9000 and port 9001::
1741 1745
1742 nstatuser@nstat-b:~$ nc -lkv 9000 1746 nstatuser@nstat-b:~$ nc -lkv 9000
1743 Listening on [0.0.0.0] (family 0, port 9000 1747 Listening on [0.0.0.0] (family 0, port 9000)
1744 1748
1745 nstatuser@nstat-b:~$ nc -lkv 9001 1749 nstatuser@nstat-b:~$ nc -lkv 9001
1746 Listening on [0.0.0.0] (family 0, port 9001 1750 Listening on [0.0.0.0] (family 0, port 9001)
1747 1751
1748 On nstat-a, run two nc clients:: 1752 On nstat-a, run two nc clients::
1749 1753
1750 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1754 nstatuser@nstat-a:~$ nc -v nstat-b 9000
1751 Connection to nstat-b 9000 port [tcp/*] suc 1755 Connection to nstat-b 9000 port [tcp/*] succeeded!
1752 1756
1753 nstatuser@nstat-a:~$ nc -v nstat-b 9001 1757 nstatuser@nstat-a:~$ nc -v nstat-b 9001
1754 Connection to nstat-b 9001 port [tcp/*] suc 1758 Connection to nstat-b 9001 port [tcp/*] succeeded!
1755 1759
1756 On nstat-a, run tcpdump to capture an ACK:: 1760 On nstat-a, run tcpdump to capture an ACK::
1757 1761
1758 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/s 1762 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
1759 tcpdump: listening on ens3, link-type EN10M 1763 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1760 1764
1761 On nstat-b, send a packet via the port 9001 s 1765 On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
1762 string 'foo' in our example:: 1766 string 'foo' in our example::
1763 1767
1764 nstatuser@nstat-b:~$ nc -lkv 9001 1768 nstatuser@nstat-b:~$ nc -lkv 9001
1765 Listening on [0.0.0.0] (family 0, port 9001 1769 Listening on [0.0.0.0] (family 0, port 9001)
1766 Connection from nstat-a 42132 received! 1770 Connection from nstat-a 42132 received!
1767 foo 1771 foo
1768 1772
1769 On nstat-a, the tcpdump should have captured 1773 On nstat-a, the tcpdump should have captured the ACK. We should check
1770 the source port numbers of the two nc clients 1774 the source port numbers of the two nc clients::
1771 1775
1772 nstatuser@nstat-a:~$ ss -ta '( dport = :900 1776 nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
1773 State Recv-Q Send-Q Local Addres 1777 State Recv-Q Send-Q Local Address:Port Peer Address:Port
1774 ESTAB 0 0 192.168.122.25 1778 ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000
1775 ESTAB 0 0 192.168.122.25 1779 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001
1776 1780
1777 Run tcprewrite, change port 9001 to port 9000 1781 Run tcprewrite, change port 9001 to port 9000, change port 42132 to
1778 port 50208:: 1782 port 50208::
1779 1783
1780 nstatuser@nstat-a:~$ tcprewrite --infile /t 1784 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum
1781 1785
1782 Now the /tmp/seq.pcap is the packet we need. 1786 Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
1783 1787
1784 nstatuser@nstat-a:~$ for i in {1..2}; do su 1788 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
1785 1789
1786 Check TcpExtTCPACKSkippedSeq on nstat-b:: 1790 Check TcpExtTCPACKSkippedSeq on nstat-b::
1787 1791
1788 nstatuser@nstat-b:~$ nstat | grep -i skip 1792 nstatuser@nstat-b:~$ nstat | grep -i skip
1789 TcpExtTCPACKSkippedSeq 1 1793 TcpExtTCPACKSkippedSeq 1 0.0
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.