1 .. _rcu_dereference_doc: 1 .. _rcu_dereference_doc:
2 2
3 PROPER CARE AND FEEDING OF RETURN VALUES FROM 3 PROPER CARE AND FEEDING OF RETURN VALUES FROM rcu_dereference()
4 ============================================== 4 ===============================================================
5 5
6 Proper care and feeding of address and data de !! 6 Most of the time, you can use values from rcu_dereference() or one of
7 important to correct use of things like RCU. !! 7 the similar primitives without worries. Dereferencing (prefix "*"),
8 returned from the rcu_dereference() family of !! 8 field selection ("->"), assignment ("="), address-of ("&"), addition and
9 data dependencies. These dependencies extend !! 9 subtraction of constants, and casts all work quite naturally and safely.
10 macro's load of the pointer to the later use o !! 10
11 either the address of a later memory access (r !! 11 It is nevertheless possible to get into trouble with other operations.
12 dependency) or the value written by a later me !! 12 Follow these rules to keep your RCU code working properly:
13 a data dependency). <<
14 <<
15 Most of the time, these dependencies are prese <<
16 freely use values from rcu_dereference(). For <<
17 (prefix "*"), field selection ("->"), assignme <<
18 ("&"), casts, and addition or subtraction of c <<
19 naturally and safely. However, because curren <<
20 either address or data dependencies into accou <<
21 to get into trouble. <<
22 <<
23 Follow these rules to preserve the address and <<
24 from your calls to rcu_dereference() and frien <<
25 readers working properly: <<
26 13
27 - You must use one of the rcu_dereferenc 14 - You must use one of the rcu_dereference() family of primitives
28 to load an RCU-protected pointer, othe 15 to load an RCU-protected pointer, otherwise CONFIG_PROVE_RCU
29 will complain. Worse yet, your code c 16 will complain. Worse yet, your code can see random memory-corruption
30 bugs due to games that compilers and D 17 bugs due to games that compilers and DEC Alpha can play.
31 Without one of the rcu_dereference() p 18 Without one of the rcu_dereference() primitives, compilers
32 can reload the value, and won't your c 19 can reload the value, and won't your code have fun with two
33 different values for a single pointer! 20 different values for a single pointer! Without rcu_dereference(),
34 DEC Alpha can load a pointer, derefere 21 DEC Alpha can load a pointer, dereference that pointer, and
35 return data preceding initialization t !! 22 return data preceding initialization that preceded the store of
36 of the pointer. (As noted later, in r !! 23 the pointer.
37 also prevents DEC Alpha from playing t <<
38 24
39 In addition, the volatile cast in rcu_ 25 In addition, the volatile cast in rcu_dereference() prevents the
40 compiler from deducing the resulting p 26 compiler from deducing the resulting pointer value. Please see
41 the section entitled "EXAMPLE WHERE TH 27 the section entitled "EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH"
42 for an example where the compiler can 28 for an example where the compiler can in fact deduce the exact
43 value of the pointer, and thus cause m 29 value of the pointer, and thus cause misordering.
44 30
45 - In the special case where data is adde 31 - In the special case where data is added but is never removed
46 while readers are accessing the struct 32 while readers are accessing the structure, READ_ONCE() may be used
47 instead of rcu_dereference(). In this 33 instead of rcu_dereference(). In this case, use of READ_ONCE()
48 takes on the role of the lockless_dere 34 takes on the role of the lockless_dereference() primitive that
49 was removed in v4.15. 35 was removed in v4.15.
50 36
51 - You are only permitted to use rcu_dere !! 37 - You are only permitted to use rcu_dereference on pointer values.
52 The compiler simply knows too much abo 38 The compiler simply knows too much about integral values to
53 trust it to carry dependencies through 39 trust it to carry dependencies through integer operations.
54 There are a very few exceptions, namel 40 There are a very few exceptions, namely that you can temporarily
55 cast the pointer to uintptr_t in order 41 cast the pointer to uintptr_t in order to:
56 42
57 - Set bits and clear bits down i 43 - Set bits and clear bits down in the must-be-zero low-order
58 bits of that pointer. This cl 44 bits of that pointer. This clearly means that the pointer
59 must have alignment constraint 45 must have alignment constraints, for example, this does
60 *not* work in general for char 46 *not* work in general for char* pointers.
61 47
62 - XOR bits to translate pointers 48 - XOR bits to translate pointers, as is done in some
63 classic buddy-allocator algori 49 classic buddy-allocator algorithms.
64 50
65 It is important to cast the value back 51 It is important to cast the value back to pointer before
66 doing much of anything else with it. 52 doing much of anything else with it.
67 53
68 - Avoid cancellation when using the "+" 54 - Avoid cancellation when using the "+" and "-" infix arithmetic
69 operators. For example, for a given v 55 operators. For example, for a given variable "x", avoid
70 "(x-(uintptr_t)x)" for char* pointers. 56 "(x-(uintptr_t)x)" for char* pointers. The compiler is within its
71 rights to substitute zero for this sor 57 rights to substitute zero for this sort of expression, so that
72 subsequent accesses no longer depend o 58 subsequent accesses no longer depend on the rcu_dereference(),
73 again possibly resulting in bugs due t 59 again possibly resulting in bugs due to misordering.
74 60
75 Of course, if "p" is a pointer from rc 61 Of course, if "p" is a pointer from rcu_dereference(), and "a"
76 and "b" are integers that happen to be 62 and "b" are integers that happen to be equal, the expression
77 "p+a-b" is safe because its value stil 63 "p+a-b" is safe because its value still necessarily depends on
78 the rcu_dereference(), thus maintainin 64 the rcu_dereference(), thus maintaining proper ordering.
79 65
80 - If you are using RCU to protect JITed 66 - If you are using RCU to protect JITed functions, so that the
81 "()" function-invocation operator is a 67 "()" function-invocation operator is applied to a value obtained
82 (directly or indirectly) from rcu_dere 68 (directly or indirectly) from rcu_dereference(), you may need to
83 interact directly with the hardware to 69 interact directly with the hardware to flush instruction caches.
84 This issue arises on some systems when 70 This issue arises on some systems when a newly JITed function is
85 using the same memory that was used by 71 using the same memory that was used by an earlier JITed function.
86 72
87 - Do not use the results from relational 73 - Do not use the results from relational operators ("==", "!=",
88 ">", ">=", "<", or "<=") when derefere 74 ">", ">=", "<", or "<=") when dereferencing. For example,
89 the following (quite strange) code is 75 the following (quite strange) code is buggy::
90 76
91 int *p; 77 int *p;
92 int *q; 78 int *q;
93 79
94 ... 80 ...
95 81
96 p = rcu_dereference(gp) 82 p = rcu_dereference(gp)
97 q = &global_q; 83 q = &global_q;
98 q += p > &oom_p; 84 q += p > &oom_p;
99 r1 = *q; /* BUGGY!!! */ 85 r1 = *q; /* BUGGY!!! */
100 86
101 As before, the reason this is buggy is 87 As before, the reason this is buggy is that relational operators
102 are often compiled using branches. An 88 are often compiled using branches. And as before, although
103 weak-memory machines such as ARM or Po 89 weak-memory machines such as ARM or PowerPC do order stores
104 after such branches, but can speculate 90 after such branches, but can speculate loads, which can again
105 result in misordering bugs. 91 result in misordering bugs.
106 92
107 - Be very careful about comparing pointe 93 - Be very careful about comparing pointers obtained from
108 rcu_dereference() against non-NULL val 94 rcu_dereference() against non-NULL values. As Linus Torvalds
109 explained, if the two pointers are equ 95 explained, if the two pointers are equal, the compiler could
110 substitute the pointer you are compari 96 substitute the pointer you are comparing against for the pointer
111 obtained from rcu_dereference(). For 97 obtained from rcu_dereference(). For example::
112 98
113 p = rcu_dereference(gp); 99 p = rcu_dereference(gp);
114 if (p == &default_struct) 100 if (p == &default_struct)
115 do_default(p->a); 101 do_default(p->a);
116 102
117 Because the compiler now knows that th 103 Because the compiler now knows that the value of "p" is exactly
118 the address of the variable "default_s 104 the address of the variable "default_struct", it is free to
119 transform this code into the following 105 transform this code into the following::
120 106
121 p = rcu_dereference(gp); 107 p = rcu_dereference(gp);
122 if (p == &default_struct) 108 if (p == &default_struct)
123 do_default(default_str 109 do_default(default_struct.a);
124 110
125 On ARM and Power hardware, the load fr 111 On ARM and Power hardware, the load from "default_struct.a"
126 can now be speculated, such that it mi 112 can now be speculated, such that it might happen before the
127 rcu_dereference(). This could result 113 rcu_dereference(). This could result in bugs due to misordering.
128 114
129 However, comparisons are OK in the fol 115 However, comparisons are OK in the following cases:
130 116
131 - The comparison was against the 117 - The comparison was against the NULL pointer. If the
132 compiler knows that the pointe 118 compiler knows that the pointer is NULL, you had better
133 not be dereferencing it anyway 119 not be dereferencing it anyway. If the comparison is
134 non-equal, the compiler is non 120 non-equal, the compiler is none the wiser. Therefore,
135 it is safe to compare pointers 121 it is safe to compare pointers from rcu_dereference()
136 against NULL pointers. 122 against NULL pointers.
137 123
138 - The pointer is never dereferen 124 - The pointer is never dereferenced after being compared.
139 Since there are no subsequent 125 Since there are no subsequent dereferences, the compiler
140 cannot use anything it learned 126 cannot use anything it learned from the comparison
141 to reorder the non-existent su 127 to reorder the non-existent subsequent dereferences.
142 This sort of comparison occurs 128 This sort of comparison occurs frequently when scanning
143 RCU-protected circular linked 129 RCU-protected circular linked lists.
144 130
145 Note that if the pointer compa !! 131 Note that if checks for being within an RCU read-side
146 of an RCU read-side critical s !! 132 critical section are not required and the pointer is never
147 is never dereferenced, rcu_acc !! 133 dereferenced, rcu_access_pointer() should be used in place
148 used in place of rcu_dereferen !! 134 of rcu_dereference().
149 it is best to avoid accidental <<
150 the rcu_access_pointer() retur <<
151 assigning it to a variable. <<
152 <<
153 Within an RCU read-side critic <<
154 reason to use rcu_access_point <<
155 135
156 - The comparison is against a po 136 - The comparison is against a pointer that references memory
157 that was initialized "a long t 137 that was initialized "a long time ago." The reason
158 this is safe is that even if m 138 this is safe is that even if misordering occurs, the
159 misordering will not affect th 139 misordering will not affect the accesses that follow
160 the comparison. So exactly ho 140 the comparison. So exactly how long ago is "a long
161 time ago"? Here are some poss 141 time ago"? Here are some possibilities:
162 142
163 - Compile time. 143 - Compile time.
164 144
165 - Boot time. 145 - Boot time.
166 146
167 - Module-init time for m 147 - Module-init time for module code.
168 148
169 - Prior to kthread creat 149 - Prior to kthread creation for kthread code.
170 150
171 - During some prior acqu 151 - During some prior acquisition of the lock that
172 we now hold. 152 we now hold.
173 153
174 - Before mod_timer() tim 154 - Before mod_timer() time for a timer handler.
175 155
176 There are many other possibili 156 There are many other possibilities involving the Linux
177 kernel's wide array of primiti 157 kernel's wide array of primitives that cause code to
178 be invoked at a later time. 158 be invoked at a later time.
179 159
180 - The pointer being compared aga 160 - The pointer being compared against also came from
181 rcu_dereference(). In this ca 161 rcu_dereference(). In this case, both pointers depend
182 on one rcu_dereference() or an 162 on one rcu_dereference() or another, so you get proper
183 ordering either way. 163 ordering either way.
184 164
185 That said, this situation can 165 That said, this situation can make certain RCU usage
186 bugs more likely to happen. W 166 bugs more likely to happen. Which can be a good thing,
187 at least if they happen during 167 at least if they happen during testing. An example
188 of such an RCU usage bug is sh 168 of such an RCU usage bug is shown in the section titled
189 "EXAMPLE OF AMPLIFIED RCU-USAG 169 "EXAMPLE OF AMPLIFIED RCU-USAGE BUG".
190 170
191 - All of the accesses following 171 - All of the accesses following the comparison are stores,
192 so that a control dependency p 172 so that a control dependency preserves the needed ordering.
193 That said, it is easy to get c 173 That said, it is easy to get control dependencies wrong.
194 Please see the "CONTROL DEPEND 174 Please see the "CONTROL DEPENDENCIES" section of
195 Documentation/memory-barriers. 175 Documentation/memory-barriers.txt for more details.
196 176
197 - The pointers are not equal *an 177 - The pointers are not equal *and* the compiler does
198 not have enough information to 178 not have enough information to deduce the value of the
199 pointer. Note that the volati 179 pointer. Note that the volatile cast in rcu_dereference()
200 will normally prevent the comp 180 will normally prevent the compiler from knowing too much.
201 181
202 However, please note that if t 182 However, please note that if the compiler knows that the
203 pointer takes on only one of t 183 pointer takes on only one of two values, a not-equal
204 comparison will provide exactl 184 comparison will provide exactly the information that the
205 compiler needs to deduce the v 185 compiler needs to deduce the value of the pointer.
206 186
207 - Disable any value-speculation optimiza 187 - Disable any value-speculation optimizations that your compiler
208 might provide, especially if you are m 188 might provide, especially if you are making use of feedback-based
209 optimizations that take data collected 189 optimizations that take data collected from prior runs. Such
210 value-speculation optimizations reorde 190 value-speculation optimizations reorder operations by design.
211 191
212 There is one exception to this rule: 192 There is one exception to this rule: Value-speculation
213 optimizations that leverage the branch 193 optimizations that leverage the branch-prediction hardware are
214 safe on strongly ordered systems (such 194 safe on strongly ordered systems (such as x86), but not on weakly
215 ordered systems (such as ARM or Power) 195 ordered systems (such as ARM or Power). Choose your compiler
216 command-line options wisely! 196 command-line options wisely!
217 197
218 198
219 EXAMPLE OF AMPLIFIED RCU-USAGE BUG 199 EXAMPLE OF AMPLIFIED RCU-USAGE BUG
220 ---------------------------------- 200 ----------------------------------
221 201
222 Because updaters can run concurrently with RCU 202 Because updaters can run concurrently with RCU readers, RCU readers can
223 see stale and/or inconsistent values. If RCU 203 see stale and/or inconsistent values. If RCU readers need fresh or
224 consistent values, which they sometimes do, th 204 consistent values, which they sometimes do, they need to take proper
225 precautions. To see this, consider the follow 205 precautions. To see this, consider the following code fragment::
226 206
227 struct foo { 207 struct foo {
228 int a; 208 int a;
229 int b; 209 int b;
230 int c; 210 int c;
231 }; 211 };
232 struct foo *gp1; 212 struct foo *gp1;
233 struct foo *gp2; 213 struct foo *gp2;
234 214
235 void updater(void) 215 void updater(void)
236 { 216 {
237 struct foo *p; 217 struct foo *p;
238 218
239 p = kmalloc(...); 219 p = kmalloc(...);
240 if (p == NULL) 220 if (p == NULL)
241 deal_with_it(); 221 deal_with_it();
242 p->a = 42; /* Each field in i 222 p->a = 42; /* Each field in its own cache line. */
243 p->b = 43; 223 p->b = 43;
244 p->c = 44; 224 p->c = 44;
245 rcu_assign_pointer(gp1, p); 225 rcu_assign_pointer(gp1, p);
246 p->b = 143; 226 p->b = 143;
247 p->c = 144; 227 p->c = 144;
248 rcu_assign_pointer(gp2, p); 228 rcu_assign_pointer(gp2, p);
249 } 229 }
250 230
251 void reader(void) 231 void reader(void)
252 { 232 {
253 struct foo *p; 233 struct foo *p;
254 struct foo *q; 234 struct foo *q;
255 int r1, r2; 235 int r1, r2;
256 236
257 rcu_read_lock(); <<
258 p = rcu_dereference(gp2); 237 p = rcu_dereference(gp2);
259 if (p == NULL) 238 if (p == NULL)
260 return; 239 return;
261 r1 = p->b; /* Guaranteed to g 240 r1 = p->b; /* Guaranteed to get 143. */
262 q = rcu_dereference(gp1); /* 241 q = rcu_dereference(gp1); /* Guaranteed non-NULL. */
263 if (p == q) { 242 if (p == q) {
264 /* The compiler decide 243 /* The compiler decides that q->c is same as p->c. */
265 r2 = p->c; /* Could ge 244 r2 = p->c; /* Could get 44 on weakly order system. */
266 } else { <<
267 r2 = p->c - r1; /* Unc <<
268 } 245 }
269 rcu_read_unlock(); <<
270 do_something_with(r1, r2); 246 do_something_with(r1, r2);
271 } 247 }
272 248
273 You might be surprised that the outcome (r1 == 249 You might be surprised that the outcome (r1 == 143 && r2 == 44) is possible,
274 but you should not be. After all, the updater 250 but you should not be. After all, the updater might have been invoked
275 a second time between the time reader() loaded 251 a second time between the time reader() loaded into "r1" and the time
276 that it loaded into "r2". The fact that this 252 that it loaded into "r2". The fact that this same result can occur due
277 to some reordering from the compiler and CPUs 253 to some reordering from the compiler and CPUs is beside the point.
278 254
279 But suppose that the reader needs a consistent 255 But suppose that the reader needs a consistent view?
280 256
281 Then one approach is to use locking, for examp 257 Then one approach is to use locking, for example, as follows::
282 258
283 struct foo { 259 struct foo {
284 int a; 260 int a;
285 int b; 261 int b;
286 int c; 262 int c;
287 spinlock_t lock; 263 spinlock_t lock;
288 }; 264 };
289 struct foo *gp1; 265 struct foo *gp1;
290 struct foo *gp2; 266 struct foo *gp2;
291 267
292 void updater(void) 268 void updater(void)
293 { 269 {
294 struct foo *p; 270 struct foo *p;
295 271
296 p = kmalloc(...); 272 p = kmalloc(...);
297 if (p == NULL) 273 if (p == NULL)
298 deal_with_it(); 274 deal_with_it();
299 spin_lock(&p->lock); 275 spin_lock(&p->lock);
300 p->a = 42; /* Each field in i 276 p->a = 42; /* Each field in its own cache line. */
301 p->b = 43; 277 p->b = 43;
302 p->c = 44; 278 p->c = 44;
303 spin_unlock(&p->lock); 279 spin_unlock(&p->lock);
304 rcu_assign_pointer(gp1, p); 280 rcu_assign_pointer(gp1, p);
305 spin_lock(&p->lock); 281 spin_lock(&p->lock);
306 p->b = 143; 282 p->b = 143;
307 p->c = 144; 283 p->c = 144;
308 spin_unlock(&p->lock); 284 spin_unlock(&p->lock);
309 rcu_assign_pointer(gp2, p); 285 rcu_assign_pointer(gp2, p);
310 } 286 }
311 287
312 void reader(void) 288 void reader(void)
313 { 289 {
314 struct foo *p; 290 struct foo *p;
315 struct foo *q; 291 struct foo *q;
316 int r1, r2; 292 int r1, r2;
317 293
318 rcu_read_lock(); <<
319 p = rcu_dereference(gp2); 294 p = rcu_dereference(gp2);
320 if (p == NULL) 295 if (p == NULL)
321 return; 296 return;
322 spin_lock(&p->lock); 297 spin_lock(&p->lock);
323 r1 = p->b; /* Guaranteed to g 298 r1 = p->b; /* Guaranteed to get 143. */
324 q = rcu_dereference(gp1); /* 299 q = rcu_dereference(gp1); /* Guaranteed non-NULL. */
325 if (p == q) { 300 if (p == q) {
326 /* The compiler decide 301 /* The compiler decides that q->c is same as p->c. */
327 r2 = p->c; /* Locking 302 r2 = p->c; /* Locking guarantees r2 == 144. */
328 } else { <<
329 spin_lock(&q->lock); <<
330 r2 = q->c - r1; <<
331 spin_unlock(&q->lock); <<
332 } 303 }
333 rcu_read_unlock(); <<
334 spin_unlock(&p->lock); 304 spin_unlock(&p->lock);
335 do_something_with(r1, r2); 305 do_something_with(r1, r2);
336 } 306 }
337 307
338 As always, use the right tool for the job! 308 As always, use the right tool for the job!
339 309
340 310
341 EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH 311 EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH
342 ----------------------------------------- 312 -----------------------------------------
343 313
344 If a pointer obtained from rcu_dereference() c 314 If a pointer obtained from rcu_dereference() compares not-equal to some
345 other pointer, the compiler normally has no cl 315 other pointer, the compiler normally has no clue what the value of the
346 first pointer might be. This lack of knowledg 316 first pointer might be. This lack of knowledge prevents the compiler
347 from carrying out optimizations that otherwise 317 from carrying out optimizations that otherwise might destroy the ordering
348 guarantees that RCU depends on. And the volat 318 guarantees that RCU depends on. And the volatile cast in rcu_dereference()
349 should prevent the compiler from guessing the 319 should prevent the compiler from guessing the value.
350 320
351 But without rcu_dereference(), the compiler kn 321 But without rcu_dereference(), the compiler knows more than you might
352 expect. Consider the following code fragment: 322 expect. Consider the following code fragment::
353 323
354 struct foo { 324 struct foo {
355 int a; 325 int a;
356 int b; 326 int b;
357 }; 327 };
358 static struct foo variable1; 328 static struct foo variable1;
359 static struct foo variable2; 329 static struct foo variable2;
360 static struct foo *gp = &variable1; 330 static struct foo *gp = &variable1;
361 331
362 void updater(void) 332 void updater(void)
363 { 333 {
364 initialize_foo(&variable2); 334 initialize_foo(&variable2);
365 rcu_assign_pointer(gp, &variab 335 rcu_assign_pointer(gp, &variable2);
366 /* 336 /*
367 * The above is the only store 337 * The above is the only store to gp in this translation unit,
368 * and the address of gp is no 338 * and the address of gp is not exported in any way.
369 */ 339 */
370 } 340 }
371 341
372 int reader(void) 342 int reader(void)
373 { 343 {
374 struct foo *p; 344 struct foo *p;
375 345
376 p = gp; 346 p = gp;
377 barrier(); 347 barrier();
378 if (p == &variable1) 348 if (p == &variable1)
379 return p->a; /* Must b 349 return p->a; /* Must be variable1.a. */
380 else 350 else
381 return p->b; /* Must b 351 return p->b; /* Must be variable2.b. */
382 } 352 }
383 353
384 Because the compiler can see all stores to "gp 354 Because the compiler can see all stores to "gp", it knows that the only
385 possible values of "gp" are "variable1" on the 355 possible values of "gp" are "variable1" on the one hand and "variable2"
386 on the other. The comparison in reader() ther 356 on the other. The comparison in reader() therefore tells the compiler
387 the exact value of "p" even in the not-equals 357 the exact value of "p" even in the not-equals case. This allows the
388 compiler to make the return values independent 358 compiler to make the return values independent of the load from "gp",
389 in turn destroying the ordering between this l 359 in turn destroying the ordering between this load and the loads of the
390 return values. This can result in "p->b" retu 360 return values. This can result in "p->b" returning pre-initialization
391 garbage values on weakly ordered systems. !! 361 garbage values.
392 362
393 In short, rcu_dereference() is *not* optional 363 In short, rcu_dereference() is *not* optional when you are going to
394 dereference the resulting pointer. 364 dereference the resulting pointer.
395 365
396 366
397 WHICH MEMBER OF THE rcu_dereference() FAMILY S 367 WHICH MEMBER OF THE rcu_dereference() FAMILY SHOULD YOU USE?
398 ---------------------------------------------- 368 ------------------------------------------------------------
399 369
400 First, please avoid using rcu_dereference_raw( 370 First, please avoid using rcu_dereference_raw() and also please avoid
401 using rcu_dereference_check() and rcu_derefere 371 using rcu_dereference_check() and rcu_dereference_protected() with a
402 second argument with a constant value of 1 (or 372 second argument with a constant value of 1 (or true, for that matter).
403 With that caution out of the way, here is some 373 With that caution out of the way, here is some guidance for which
404 member of the rcu_dereference() to use in vari 374 member of the rcu_dereference() to use in various situations:
405 375
406 1. If the access needs to be within an RC 376 1. If the access needs to be within an RCU read-side critical
407 section, use rcu_dereference(). With 377 section, use rcu_dereference(). With the new consolidated
408 RCU flavors, an RCU read-side critical 378 RCU flavors, an RCU read-side critical section is entered
409 using rcu_read_lock(), anything that d 379 using rcu_read_lock(), anything that disables bottom halves,
410 anything that disables interrupts, or 380 anything that disables interrupts, or anything that disables
411 preemption. Please note that spinlock !! 381 preemption.
412 are also implied RCU read-side critica <<
413 they are preemptible, as they are in k <<
414 CONFIG_PREEMPT_RT=y. <<
415 382
416 2. If the access might be within an RCU r 383 2. If the access might be within an RCU read-side critical section
417 on the one hand, or protected by (say) 384 on the one hand, or protected by (say) my_lock on the other,
418 use rcu_dereference_check(), for examp 385 use rcu_dereference_check(), for example::
419 386
420 p1 = rcu_dereference_check(p-> 387 p1 = rcu_dereference_check(p->rcu_protected_pointer,
421 loc 388 lockdep_is_held(&my_lock));
422 389
423 390
424 3. If the access might be within an RCU r 391 3. If the access might be within an RCU read-side critical section
425 on the one hand, or protected by eithe 392 on the one hand, or protected by either my_lock or your_lock on
426 the other, again use rcu_dereference_c 393 the other, again use rcu_dereference_check(), for example::
427 394
428 p1 = rcu_dereference_check(p-> 395 p1 = rcu_dereference_check(p->rcu_protected_pointer,
429 loc 396 lockdep_is_held(&my_lock) ||
430 loc 397 lockdep_is_held(&your_lock));
431 398
432 4. If the access is on the update side, s 399 4. If the access is on the update side, so that it is always protected
433 by my_lock, use rcu_dereference_protec 400 by my_lock, use rcu_dereference_protected()::
434 401
435 p1 = rcu_dereference_protected 402 p1 = rcu_dereference_protected(p->rcu_protected_pointer,
436 403 lockdep_is_held(&my_lock));
437 404
438 This can be extended to handle multipl 405 This can be extended to handle multiple locks as in #3 above,
439 and both can be extended to check othe 406 and both can be extended to check other conditions as well.
440 407
441 5. If the protection is supplied by the c 408 5. If the protection is supplied by the caller, and is thus unknown
442 to this code, that is the rare case wh 409 to this code, that is the rare case when rcu_dereference_raw()
443 is appropriate. In addition, rcu_dere 410 is appropriate. In addition, rcu_dereference_raw() might be
444 appropriate when the lockdep expressio 411 appropriate when the lockdep expression would be excessively
445 complex, except that a better approach 412 complex, except that a better approach in that case might be to
446 take a long hard look at your synchron 413 take a long hard look at your synchronization design. Still,
447 there are data-locking cases where any 414 there are data-locking cases where any one of a very large number
448 of locks or reference counters suffice 415 of locks or reference counters suffices to protect the pointer,
449 so rcu_dereference_raw() does have its 416 so rcu_dereference_raw() does have its place.
450 417
451 However, its place is probably quite a 418 However, its place is probably quite a bit smaller than one
452 might expect given the number of uses 419 might expect given the number of uses in the current kernel.
453 Ditto for its synonym, rcu_dereference 420 Ditto for its synonym, rcu_dereference_check( ... , 1), and
454 its close relative, rcu_dereference_pr 421 its close relative, rcu_dereference_protected(... , 1).
455 422
456 423
457 SPARSE CHECKING OF RCU-PROTECTED POINTERS 424 SPARSE CHECKING OF RCU-PROTECTED POINTERS
458 ----------------------------------------- 425 -----------------------------------------
459 426
460 The sparse static-analysis tool checks for non !! 427 The sparse static-analysis tool checks for direct access to RCU-protected
461 pointers, which can result in "interesting" bu 428 pointers, which can result in "interesting" bugs due to compiler
462 optimizations involving invented loads and per 429 optimizations involving invented loads and perhaps also load tearing.
463 For example, suppose someone mistakenly does s 430 For example, suppose someone mistakenly does something like this::
464 431
465 p = q->rcu_protected_pointer; 432 p = q->rcu_protected_pointer;
466 do_something_with(p->a); 433 do_something_with(p->a);
467 do_something_else_with(p->b); 434 do_something_else_with(p->b);
468 435
469 If register pressure is high, the compiler mig 436 If register pressure is high, the compiler might optimize "p" out
470 of existence, transforming the code to somethi 437 of existence, transforming the code to something like this::
471 438
472 do_something_with(q->rcu_protected_poi 439 do_something_with(q->rcu_protected_pointer->a);
473 do_something_else_with(q->rcu_protecte 440 do_something_else_with(q->rcu_protected_pointer->b);
474 441
475 This could fatally disappoint your code if q-> 442 This could fatally disappoint your code if q->rcu_protected_pointer
476 changed in the meantime. Nor is this a theore 443 changed in the meantime. Nor is this a theoretical problem: Exactly
477 this sort of bug cost Paul E. McKenney (and se 444 this sort of bug cost Paul E. McKenney (and several of his innocent
478 colleagues) a three-day weekend back in the ea 445 colleagues) a three-day weekend back in the early 1990s.
479 446
480 Load tearing could of course result in derefer 447 Load tearing could of course result in dereferencing a mashup of a pair
481 of pointers, which also might fatally disappoi 448 of pointers, which also might fatally disappoint your code.
482 449
483 These problems could have been avoided simply 450 These problems could have been avoided simply by making the code instead
484 read as follows:: 451 read as follows::
485 452
486 p = rcu_dereference(q->rcu_protected_p 453 p = rcu_dereference(q->rcu_protected_pointer);
487 do_something_with(p->a); 454 do_something_with(p->a);
488 do_something_else_with(p->b); 455 do_something_else_with(p->b);
489 456
490 Unfortunately, these sorts of bugs can be extr 457 Unfortunately, these sorts of bugs can be extremely hard to spot during
491 review. This is where the sparse tool comes i 458 review. This is where the sparse tool comes into play, along with the
492 "__rcu" marker. If you mark a pointer declara 459 "__rcu" marker. If you mark a pointer declaration, whether in a structure
493 or as a formal parameter, with "__rcu", which 460 or as a formal parameter, with "__rcu", which tells sparse to complain if
494 this pointer is accessed directly. It will al 461 this pointer is accessed directly. It will also cause sparse to complain
495 if a pointer not marked with "__rcu" is access 462 if a pointer not marked with "__rcu" is accessed using rcu_dereference()
496 and friends. For example, ->rcu_protected_poi 463 and friends. For example, ->rcu_protected_pointer might be declared as
497 follows:: 464 follows::
498 465
499 struct foo __rcu *rcu_protected_pointe 466 struct foo __rcu *rcu_protected_pointer;
500 467
501 Use of "__rcu" is opt-in. If you choose not t 468 Use of "__rcu" is opt-in. If you choose not to use it, then you should
502 ignore the sparse warnings. 469 ignore the sparse warnings.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.