1 What: /sys/class/rtrs-client 1 What: /sys/class/rtrs-client
2 Date: Feb 2020 2 Date: Feb 2020
3 KernelVersion: 5.7 3 KernelVersion: 5.7
4 Contact: Jack Wang <jinpu.wang@cloud.ion 4 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
5 Description: When a user of RTRS API create 5 Description: When a user of RTRS API creates a new session, a directory entry with
6 the name of that session is cr 6 the name of that session is created under /sys/class/rtrs-client/<session-name>/
7 7
8 What: /sys/class/rtrs-client/<sessio 8 What: /sys/class/rtrs-client/<session-name>/add_path
9 Date: Feb 2020 9 Date: Feb 2020
10 KernelVersion: 5.7 10 KernelVersion: 5.7
11 Contact: Jack Wang <jinpu.wang@cloud.ion 11 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
12 Description: RW, adds a new path (connectio 12 Description: RW, adds a new path (connection) to an existing session. Expected format is the
13 following:: 13 following::
14 14
15 <[source addr,]destination 15 <[source addr,]destination addr>
16 *addr ::= [ ip:<ipv4|ipv6> 16 *addr ::= [ ip:<ipv4|ipv6> | gid:<gid> ]
17 17
18 What: /sys/class/rtrs-client/<sessio 18 What: /sys/class/rtrs-client/<session-name>/max_reconnect_attempts
19 Date: Feb 2020 19 Date: Feb 2020
20 KernelVersion: 5.7 20 KernelVersion: 5.7
21 Contact: Jack Wang <jinpu.wang@cloud.ion 21 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
22 Description: Maximum number reconnect attem 22 Description: Maximum number reconnect attempts the client should make before giving up
23 after connection breaks unexpe 23 after connection breaks unexpectedly.
24 24
25 What: /sys/class/rtrs-client/<sessio 25 What: /sys/class/rtrs-client/<session-name>/mp_policy
26 Date: Feb 2020 26 Date: Feb 2020
27 KernelVersion: 5.7 27 KernelVersion: 5.7
28 Contact: Jack Wang <jinpu.wang@cloud.ion 28 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
29 Description: Multipath policy specifies whi 29 Description: Multipath policy specifies which path should be selected on each IO:
30 30
31 round-robin (0): 31 round-robin (0):
32 select path in per CPU rou 32 select path in per CPU round-robin manner.
33 33
34 min-inflight (1): 34 min-inflight (1):
35 select path with minimum i 35 select path with minimum inflights.
36 36
37 min-latency (2): 37 min-latency (2):
38 select path with minimum l 38 select path with minimum latency.
39 39
40 What: /sys/class/rtrs-client/<sessio 40 What: /sys/class/rtrs-client/<session-name>/paths/
41 Date: Feb 2020 41 Date: Feb 2020
42 KernelVersion: 5.7 42 KernelVersion: 5.7
43 Contact: Jack Wang <jinpu.wang@cloud.ion 43 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
44 Description: Each path belonging to a given 44 Description: Each path belonging to a given session is listed here by its source and
45 destination address. When a ne 45 destination address. When a new path is added to a session by writing to
46 the "add_path" entry, a direct< 46 the "add_path" entry, a directory <src@dst> is created.
47 47
48 What: /sys/class/rtrs-client/<session 48 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/state
49 Date: Feb 2020 49 Date: Feb 2020
50 KernelVersion: 5.7 50 KernelVersion: 5.7
51 Contact: Jack Wang <jinpu.wang@cloud.ion 51 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
52 Description: RO, Contains "connected" if th 52 Description: RO, Contains "connected" if the session is connected to the peer and fully
53 functional. Otherwise the fil 53 functional. Otherwise the file contains "disconnected"
54 54
55 What: /sys/class/rtrs-client/<session 55 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/reconnect
56 Date: Feb 2020 56 Date: Feb 2020
57 KernelVersion: 5.7 57 KernelVersion: 5.7
58 Contact: Jack Wang <jinpu.wang@cloud.ion 58 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
59 Description: Write "1" to the file in order 59 Description: Write "1" to the file in order to reconnect the path.
60 Operation is blocking and retu 60 Operation is blocking and returns 0 if reconnect was successful.
61 61
62 What: /sys/class/rtrs-client/<session 62 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/disconnect
63 Date: Feb 2020 63 Date: Feb 2020
64 KernelVersion: 5.7 64 KernelVersion: 5.7
65 Contact: Jack Wang <jinpu.wang@cloud.ion 65 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
66 Description: Write "1" to the file in order 66 Description: Write "1" to the file in order to disconnect the path.
67 Operation blocks until RTRS pa 67 Operation blocks until RTRS path is disconnected.
68 68
69 What: /sys/class/rtrs-client/<session 69 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/remove_path
70 Date: Feb 2020 70 Date: Feb 2020
71 KernelVersion: 5.7 71 KernelVersion: 5.7
72 Contact: Jack Wang <jinpu.wang@cloud.ion 72 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
73 Description: Write "1" to the file in order 73 Description: Write "1" to the file in order to disconnected and remove the path
74 from the session. Operation b 74 from the session. Operation blocks until the path is disconnected
75 and removed from the session. 75 and removed from the session.
76 76
77 What: /sys/class/rtrs-client/<session 77 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/hca_name
78 Date: Feb 2020 78 Date: Feb 2020
79 KernelVersion: 5.7 79 KernelVersion: 5.7
80 Contact: Jack Wang <jinpu.wang@cloud.ion 80 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
81 Description: RO, Contains the name of HCA t !! 81 Description: RO, Contains the the name of HCA the connection established on.
82 82
83 What: /sys/class/rtrs-client/<session 83 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/hca_port
84 Date: Feb 2020 84 Date: Feb 2020
85 KernelVersion: 5.7 85 KernelVersion: 5.7
86 Contact: Jack Wang <jinpu.wang@cloud.ion 86 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
87 Description: RO, Contains the port number o 87 Description: RO, Contains the port number of active port traffic is going through.
88 88
89 What: /sys/class/rtrs-client/<session 89 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/src_addr
90 Date: Feb 2020 90 Date: Feb 2020
91 KernelVersion: 5.7 91 KernelVersion: 5.7
92 Contact: Jack Wang <jinpu.wang@cloud.ion 92 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
93 Description: RO, Contains the source addres 93 Description: RO, Contains the source address of the path
94 94
95 What: /sys/class/rtrs-client/<session 95 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/dst_addr
96 Date: Feb 2020 96 Date: Feb 2020
97 KernelVersion: 5.7 97 KernelVersion: 5.7
98 Contact: Jack Wang <jinpu.wang@cloud.ion 98 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
99 Description: RO, Contains the destination a 99 Description: RO, Contains the destination address of the path
100 100
101 What: /sys/class/rtrs-client/<session 101 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/cur_latency
102 Date: Feb 2020 102 Date: Feb 2020
103 KernelVersion: 5.7 103 KernelVersion: 5.7
104 Contact: Jack Wang <jinpu.wang@cloud.ion 104 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
105 Description: RO, Contains the latency time 105 Description: RO, Contains the latency time calculated by the heart-beat messages.
106 Whenever the client sends hear 106 Whenever the client sends heart-beat message, it checks the time gap
107 between sending the heart-beat 107 between sending the heart-beat message and receiving the ACK.
108 This value can be changed regu 108 This value can be changed regularly.
109 109
110 What: /sys/class/rtrs-client/<session 110 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/stats/reset_all
111 Date: Feb 2020 111 Date: Feb 2020
112 KernelVersion: 5.7 112 KernelVersion: 5.7
113 Contact: Jack Wang <jinpu.wang@cloud.ion 113 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
114 Description: RW, Read will return usage hel 114 Description: RW, Read will return usage help, write 0 will clear all the statistics.
115 115
116 What: /sys/class/rtrs-client/<session 116 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/stats/cpu_migration
117 Date: Feb 2020 117 Date: Feb 2020
118 KernelVersion: 5.7 118 KernelVersion: 5.7
119 Contact: Jack Wang <jinpu.wang@cloud.ion 119 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
120 Description: RTRS expects that each HCA IRQ 120 Description: RTRS expects that each HCA IRQ is pinned to a separate CPU. If it's
121 not the case, the processing o 121 not the case, the processing of an I/O response could be processed on a
122 different CPU than where it wa 122 different CPU than where it was originally submitted. This file shows
123 how many interrupts where gene 123 how many interrupts where generated on a non expected CPU.
124 124
125 "from:" 125 "from:"
126 is the CPU on which the IR 126 is the CPU on which the IRQ was expected, but not generated.
127 "to:" 127 "to:"
128 is the CPU on which the IR 128 is the CPU on which the IRQ was generated, but not expected.
129 129
130 What: /sys/class/rtrs-client/<session 130 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/stats/reconnects
131 Date: Feb 2020 131 Date: Feb 2020
132 KernelVersion: 5.7 132 KernelVersion: 5.7
133 Contact: Jack Wang <jinpu.wang@cloud.ion 133 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
134 Description: Contains 2 unsigned int values 134 Description: Contains 2 unsigned int values, the first one records number of successful
135 reconnects in the path lifetim 135 reconnects in the path lifetime, the second one records number of failed
136 reconnects in the path lifetim 136 reconnects in the path lifetime.
137 137
138 What: /sys/class/rtrs-client/<session 138 What: /sys/class/rtrs-client/<session-name>/paths/<src@dst>/stats/rdma
139 Date: Feb 2020 139 Date: Feb 2020
140 KernelVersion: 5.7 140 KernelVersion: 5.7
141 Contact: Jack Wang <jinpu.wang@cloud.ion 141 Contact: Jack Wang <jinpu.wang@cloud.ionos.com> Danil Kipnis <danil.kipnis@cloud.ionos.com>
142 Description: Contains statistics regarding 142 Description: Contains statistics regarding rdma operations and inflight operations.
143 The output consists of 6 value 143 The output consists of 6 values::
144 144
145 <read-count> <read-total-s 145 <read-count> <read-total-size> <write-count> \
146 <write-total-size> <inflig 146 <write-total-size> <inflights> <failovered>
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.