1 ==================
2 IP over InfiniBand
3 ==================
4
5 The ib_ipoib driver is an implementation of
6 protocol as specified by RFC 4391 and 4392,
7 working group. It is a "native" implementat
8 setting the interface type to ARPHRD_INFINIB
9 address length to 20 (earlier proprietary im
10 masqueraded to the kernel as ethernet interf
11
12 Partitions and P_Keys
13 =====================
14
15 When the IPoIB driver is loaded, it creates
16 port using the P_Key at index 0. To create
17 different P_Key, write the desired P_Key int
18 /sys/class/net/<intf name>/create_child file
19
20 echo 0x8001 > /sys/class/net/ib0/create_ch
21
22 This will create an interface named ib0.8001
23 remove a subinterface, use the "delete_child
24
25 echo 0x8001 > /sys/class/net/ib0/delete_ch
26
27 The P_Key for any interface is given by the
28 main interface for a subinterface is in "par
29
30 Child interface create/delete can also be do
31 rtnl_link_ops, where children created using
32
33 Datagram vs Connected modes
34 ===========================
35
36 The IPoIB driver supports two modes of opera
37 connected. The mode is set and read through
38 /sys/class/net/<intf name>/mode file.
39
40 In datagram mode, the IB UD (Unreliable Data
41 and so the interface MTU has is equal to the
42 IPoIB encapsulation header (4 bytes). For e
43 fabric with a 2K MTU, the IPoIB MTU will be
44
45 In connected mode, the IB RC (Reliable Conne
46 Connected mode takes advantage of the connec
47 transport and allows an MTU up to the maxima
48 which reduces the number of IP packets neede
49 datagrams, TCP segments, etc and increases t
50 messages.
51
52 In connected mode, the interface's UD QP is
53 and communication with peers that don't supp
54 this case, RX emulation of ICMP PMTU packets
55 networking stack to use the smaller UD MTU f
56
57 Stateless offloads
58 ==================
59
60 If the IB HW supports IPoIB stateless offloa
61 TCP/IP checksum and/or Large Send (LSO) offl
62 network stack.
63
64 Large Receive (LRO) offloading is also imple
65 on/off using ethtool calls. Currently LRO i
66 checksum offload capable devices.
67
68 Stateless offloads are supported only in dat
69
70 Interrupt moderation
71 ====================
72
73 If the underlying IB device supports CQ even
74 use ethtool to set interrupt mitigation para
75 the overhead incurred by handling interrupts
76 IPoIB doesn't use events for TX completion s
77 moderation is supported.
78
79 Debugging Information
80 =====================
81
82 By compiling the IPoIB driver with CONFIG_IN
83 to 'y', tracing messages are compiled into t
84 turned on by setting the module parameters d
85 mcast_debug_level to 1. These parameters ca
86 runtime through files in /sys/module/ib_ipoi
87
88 CONFIG_INFINIBAND_IPOIB_DEBUG also enables f
89 virtual filesystem. By mounting this filesy
90
91 mount -t debugfs none /sys/kernel/debug
92
93 it is possible to get statistics about multi
94 files /sys/kernel/debug/ipoib/ib0_mcg and so
95
96 The performance impact of this option is neg
97 is safe to enable this option with debug_lev
98 operation.
99
100 CONFIG_INFINIBAND_IPOIB_DEBUG_DATA enables e
101 the data path when data_debug_level is set t
102 the output disabled, enabling this configura
103 performance, because it adds tests to the fa
104
105 References
106 ==========
107
108 Transmission of IP over InfiniBand (IPoIB) (
109 http://ietf.org/rfc/rfc4391.txt
110
111 IP over InfiniBand (IPoIB) Architecture (RFC
112 http://ietf.org/rfc/rfc4392.txt
113
114 IP over InfiniBand: Connected Mode (RFC 4755
115 http://ietf.org/rfc/rfc4755.txt
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.