1 .. SPDX-License-Identifier: GPL-2.0
2
3 ============================
4 Glock internal locking rules
5 ============================
6
7 This documents the basic principles of the glo
8 internals. Each glock (struct gfs2_glock in fs
9 has two main (internal) locks:
10
11 1. A spinlock (gl_lockref.lock) which protect
12 as gl_state, gl_target and the list of hol
13 2. A non-blocking bit lock, GLF_LOCK, which i
14 threads from making calls to the DLM, etc.
15 thread takes this lock, it must then call
16 workqueue) when it releases it in order to
17 are completed.
18
19 The gl_holders list contains all the queued lo
20 just the holders) associated with the glock. I
21 held locks, then they will be contiguous entri
22 of the list. Locks are granted in strictly the
23 are queued.
24
25 There are three lock states that users of the
26 namely shared (SH), deferred (DF) and exclusiv
27 to the following DLM lock modes:
28
29 ========== ====== =======================
30 Glock mode DLM lock mode
31 ========== ====== =======================
32 UN IV/NL Unlocked (no DLM lock a
33 SH PR (Protected read)
34 DF CW (Concurrent write)
35 EX EX (Exclusive)
36 ========== ====== =======================
37
38 Thus DF is basically a shared mode which is in
39 shared lock mode, SH. In GFS2 the DF mode is u
40 operations. The glocks are basically a lock pl
41 with cache management. The following rules app
42
43 ========== ============== ==========
44 Glock mode Cache Metadata Cache data
45 ========== ============== ==========
46 UN No No
47 DF Yes No
48 SH Yes Yes
49 EX Yes Yes
50 ========== ============== ==========
51
52 These rules are implemented using the various
53 are defined for each type of glock. Not all ty
54 all the modes. Only inode glocks use the DF mo
55
56 Table of glock operations and per type constan
57
58 ============== ===========================
59 Field Purpose
60 ============== ===========================
61 go_sync Called before remote state
62 go_xmote_bh Called after remote state c
63 go_inval Called if remote state chan
64 go_instantiate Called when a glock has bee
65 go_held Called every time a glock h
66 go_dump Called to print content of
67 error to dump glock to the
68 go_callback Called if the DLM sends a c
69 go_unlocked Called when a glock is unlo
70 go_type The type of the glock, ``LM
71 go_flags GLOF_ASPACE is set, if the
72 associated with it
73 ============== ===========================
74
75 The minimum hold time for each lock is the tim
76 grant for which we ignore remote demote reques
77 prevent a situation where locks are being boun
78 from node to node with none of the nodes makin
79 tends to show up most with shared mmapped file
80 to by multiple nodes. By delaying the demotion
81 remote callback, that gives the userspace prog
82 some progress before the pages are unmapped.
83
84 Eventually, we hope to make the glock "EX" mod
85 local locking will be done with the i_mutex as
86 glock.
87
88 Locking rules for glock operations:
89
90 ============== ====================== ===
91 Operation GLF_LOCK bit lock held gl_
92 ============== ====================== ===
93 go_sync Yes
94 go_xmote_bh Yes
95 go_inval Yes
96 go_instantiate No
97 go_held No
98 go_dump Sometimes
99 go_callback Sometimes (N/A)
100 go_unlocked Yes
101 ============== ====================== ===
102
103 .. Note::
104
105 Operations must not drop either the bit loc
106 if its held on entry. go_dump and do_demote
107 Note that go_dump will only be called if th
108 indicates that it is caching uptodate data.
109
110 Glock locking order within GFS2:
111
112 1. i_rwsem (if required)
113 2. Rename glock (for rename only)
114 3. Inode glock(s)
115 (Parents before children, inodes at "same
116 lock number order)
117 4. Rgrp glock(s) (for (de)allocation operatio
118 5. Transaction glock (via gfs2_trans_begin) f
119 6. i_rw_mutex (if required)
120 7. Page lock (always last, very important!)
121
122 There are two glocks per inode. One deals with
123 itself (locking order as above), and the other
124 glock is used in conjunction with the i_nlink
125 determine the lifetime of the inode in questio
126 is on a per-inode basis. Locking of rgrps is o
127 In general we prefer to lock local locks prior
128
129 Glock Statistics
130 ----------------
131
132 The stats are divided into two sets: those rel
133 super block and those relating to an individua
134 super block stats are done on a per cpu basis
135 try and reduce the overhead of gathering them.
136 further divided by glock type. All timings are
137
138 In the case of both the super block and glock
139 the same information is gathered in each case.
140 block timing statistics are used to provide de
141 the glock timing statistics, so that newly cre
142 should have, as far as possible, a sensible st
143 The per-glock counters are initialised to zero
144 glock is created. The per-glock statistics are
145 the glock is ejected from memory.
146
147 The statistics are divided into three pairs of
148 variance, plus two counters. The mean/variance
149 smoothed exponential estimates and the algorit
150 one which will be very familiar to those used
151 of round trip times in network code. See "TCP/
152 Volume 1", W. Richard Stevens, sect 21.3, "Rou
153 p. 299 and onwards. Also, Volume 2, Sect. 25.1
154 Unlike the TCP/IP Illustrated case, the mean a
155 not scaled, but are in units of integer nanose
156
157 The three pairs of mean/variance measure the f
158 things:
159
160 1. DLM lock time (non-blocking requests)
161 2. DLM lock time (blocking requests)
162 3. Inter-request time (again to the DLM)
163
164 A non-blocking request is one which will compl
165 away, whatever the state of the DLM lock in qu
166 currently means any requests when (a) the curr
167 the lock is exclusive, i.e. a lock demotion (b
168 state is either null or unlocked (again, a dem
169 "try lock" flag is set. A blocking request cov
170 lock requests.
171
172 There are two counters. The first is there pri
173 how many lock requests have been made, and thu
174 has gone into the mean/variance calculations.
175 is counting queuing of holders at the top laye
176 code. Hopefully that number will be a lot larg
177 of dlm lock requests issued.
178
179 So why gather these statistics? There are seve
180 we'd like to get a better idea of these timing
181
182 1. To be able to better set the glock "min hol
183 2. To spot performance issues more easily
184 3. To improve the algorithm for selecting reso
185 allocation (to base it on lock wait time, r
186 using a "try lock")
187
188 Due to the smoothing action of the updates, a
189 some input quantity being sampled will only fu
190 into account after 8 samples (or 4 for the var
191 needs to be carefully considered when interpre
192 results.
193
194 Knowing both the time it takes a lock request
195 the average time between lock requests for a g
196 can compute the total percentage of the time f
197 node is able to use a glock vs. time that the
198 cluster has its share. That will be very usefu
199 the lock min hold time.
200
201 Great care has been taken to ensure that we
202 measure exactly the quantities that we want, a
203 as possible. There are always inaccuracies in
204 measuring system, but I hope this is as accura
205 can reasonably make it.
206
207 Per sb stats can be found here::
208
209 /sys/kernel/debug/gfs2/<fsname>/sbstats
210
211 Per glock stats can be found here::
212
213 /sys/kernel/debug/gfs2/<fsname>/glstats
214
215 Assuming that debugfs is mounted on /sys/kerne
216 that <fsname> is replaced with the name of the
217 in question.
218
219 The abbreviations used in the output as are fo
220
221 ========= ===================================
222 srtt Smoothed round trip time for non bl
223 srttvar Variance estimate for srtt
224 srttb Smoothed round trip time for (poten
225 srttvarb Variance estimate for srttb
226 sirt Smoothed inter request time (for dl
227 sirtvar Variance estimate for sirt
228 dlm Number of dlm requests made (dcnt i
229 queue Number of glock requests queued (qc
230 ========= ===================================
231
232 The sbstats file contains a set of these stats
233 for each type) and for each cpu (one column pe
234 a set of these stats for each glock in a simil
235 using the format mean/variance for each of the
236
237 The gfs2_glock_lock_time tracepoint prints out
238 for the glock in question, along with some add
239 reply that is received:
240
241 ====== =====================================
242 status The status of the dlm request
243 flags The dlm request flags
244 tdiff The time taken by this specific reque
245 ====== =====================================
246
247 (remaining fields as per above list)
248
249
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.