1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ======== 3 ========
4 ORANGEFS 4 ORANGEFS
5 ======== 5 ========
6 6
7 OrangeFS is an LGPL userspace scale-out parall 7 OrangeFS is an LGPL userspace scale-out parallel storage system. It is ideal
8 for large storage problems faced by HPC, BigDa 8 for large storage problems faced by HPC, BigData, Streaming Video,
9 Genomics, Bioinformatics. 9 Genomics, Bioinformatics.
10 10
11 Orangefs, originally called PVFS, was first de 11 Orangefs, originally called PVFS, was first developed in 1993 by
12 Walt Ligon and Eric Blumer as a parallel file 12 Walt Ligon and Eric Blumer as a parallel file system for Parallel
13 Virtual Machine (PVM) as part of a NASA grant 13 Virtual Machine (PVM) as part of a NASA grant to study the I/O patterns
14 of parallel programs. 14 of parallel programs.
15 15
16 Orangefs features include: 16 Orangefs features include:
17 17
18 * Distributes file data among multiple file 18 * Distributes file data among multiple file servers
19 * Supports simultaneous access by multiple c 19 * Supports simultaneous access by multiple clients
20 * Stores file data and metadata on servers u 20 * Stores file data and metadata on servers using local file system
21 and access methods 21 and access methods
22 * Userspace implementation is easy to instal 22 * Userspace implementation is easy to install and maintain
23 * Direct MPI support 23 * Direct MPI support
24 * Stateless 24 * Stateless
25 25
26 26
27 Mailing List Archives 27 Mailing List Archives
28 ===================== 28 =====================
29 29
30 http://lists.orangefs.org/pipermail/devel_list 30 http://lists.orangefs.org/pipermail/devel_lists.orangefs.org/
31 31
32 32
33 Mailing List Submissions 33 Mailing List Submissions
34 ======================== 34 ========================
35 35
36 devel@lists.orangefs.org 36 devel@lists.orangefs.org
37 37
38 38
39 Documentation 39 Documentation
40 ============= 40 =============
41 41
42 http://www.orangefs.org/documentation/ 42 http://www.orangefs.org/documentation/
43 43
44 Running ORANGEFS On a Single Server 44 Running ORANGEFS On a Single Server
45 =================================== 45 ===================================
46 46
47 OrangeFS is usually run in large installations 47 OrangeFS is usually run in large installations with multiple servers and
48 clients, but a complete filesystem can be run 48 clients, but a complete filesystem can be run on a single machine for
49 development and testing. 49 development and testing.
50 50
51 On Fedora, install orangefs and orangefs-serve 51 On Fedora, install orangefs and orangefs-server::
52 52
53 dnf -y install orangefs orangefs-server 53 dnf -y install orangefs orangefs-server
54 54
55 There is an example server configuration file 55 There is an example server configuration file in
56 /etc/orangefs/orangefs.conf. Change localhost 56 /etc/orangefs/orangefs.conf. Change localhost to your hostname if
57 necessary. 57 necessary.
58 58
59 To generate a filesystem to run xfstests again 59 To generate a filesystem to run xfstests against, see below.
60 60
61 There is an example client configuration file 61 There is an example client configuration file in /etc/pvfs2tab. It is a
62 single line. Uncomment it and change the host 62 single line. Uncomment it and change the hostname if necessary. This
63 controls clients which use libpvfs2. This doe 63 controls clients which use libpvfs2. This does not control the
64 pvfs2-client-core. 64 pvfs2-client-core.
65 65
66 Create the filesystem:: 66 Create the filesystem::
67 67
68 pvfs2-server -f /etc/orangefs/orangefs.con 68 pvfs2-server -f /etc/orangefs/orangefs.conf
69 69
70 Start the server:: 70 Start the server::
71 71
72 systemctl start orangefs-server 72 systemctl start orangefs-server
73 73
74 Test the server:: 74 Test the server::
75 75
76 pvfs2-ping -m /pvfsmnt 76 pvfs2-ping -m /pvfsmnt
77 77
78 Start the client. The module must be compiled 78 Start the client. The module must be compiled in or loaded before this
79 point:: 79 point::
80 80
81 systemctl start orangefs-client 81 systemctl start orangefs-client
82 82
83 Mount the filesystem:: 83 Mount the filesystem::
84 84
85 mount -t pvfs2 tcp://localhost:3334/orange 85 mount -t pvfs2 tcp://localhost:3334/orangefs /pvfsmnt
86 86
87 Userspace Filesystem Source 87 Userspace Filesystem Source
88 =========================== 88 ===========================
89 89
90 http://www.orangefs.org/download 90 http://www.orangefs.org/download
91 91
92 Orangefs versions prior to 2.9.3 would not be 92 Orangefs versions prior to 2.9.3 would not be compatible with the
93 upstream version of the kernel client. 93 upstream version of the kernel client.
94 94
95 95
96 Building ORANGEFS on a Single Server 96 Building ORANGEFS on a Single Server
97 ==================================== 97 ====================================
98 98
99 Where OrangeFS cannot be installed from distri 99 Where OrangeFS cannot be installed from distribution packages, it may be
100 built from source. 100 built from source.
101 101
102 You can omit --prefix if you don't care that t 102 You can omit --prefix if you don't care that things are sprinkled around
103 in /usr/local. As of version 2.9.6, OrangeFS 103 in /usr/local. As of version 2.9.6, OrangeFS uses Berkeley DB by
104 default, we will probably be changing the defa 104 default, we will probably be changing the default to LMDB soon.
105 105
106 :: 106 ::
107 107
108 ./configure --prefix=/opt/ofs --with-db-ba 108 ./configure --prefix=/opt/ofs --with-db-backend=lmdb --disable-usrint
109 109
110 make 110 make
111 111
112 make install 112 make install
113 113
114 Create an orangefs config file by running pvfs 114 Create an orangefs config file by running pvfs2-genconfig and
115 specifying a target config file. Pvfs2-genconf 115 specifying a target config file. Pvfs2-genconfig will prompt you
116 through. Generally it works fine to take the d 116 through. Generally it works fine to take the defaults, but you
117 should use your server's hostname, rather than 117 should use your server's hostname, rather than "localhost" when
118 it comes to that question:: 118 it comes to that question::
119 119
120 /opt/ofs/bin/pvfs2-genconfig /etc/pvfs2.co 120 /opt/ofs/bin/pvfs2-genconfig /etc/pvfs2.conf
121 121
122 Create an /etc/pvfs2tab file (localhost is fin 122 Create an /etc/pvfs2tab file (localhost is fine)::
123 123
124 echo tcp://localhost:3334/orangefs /pvfsmn 124 echo tcp://localhost:3334/orangefs /pvfsmnt pvfs2 defaults,noauto 0 0 > \
125 /etc/pvfs2tab 125 /etc/pvfs2tab
126 126
127 Create the mount point you specified in the ta 127 Create the mount point you specified in the tab file if needed::
128 128
129 mkdir /pvfsmnt 129 mkdir /pvfsmnt
130 130
131 Bootstrap the server:: 131 Bootstrap the server::
132 132
133 /opt/ofs/sbin/pvfs2-server -f /etc/pvfs2.c 133 /opt/ofs/sbin/pvfs2-server -f /etc/pvfs2.conf
134 134
135 Start the server:: 135 Start the server::
136 136
137 /opt/ofs/sbin/pvfs2-server /etc/pvfs2.conf 137 /opt/ofs/sbin/pvfs2-server /etc/pvfs2.conf
138 138
139 Now the server should be running. Pvfs2-ls is 139 Now the server should be running. Pvfs2-ls is a simple
140 test to verify that the server is running:: 140 test to verify that the server is running::
141 141
142 /opt/ofs/bin/pvfs2-ls /pvfsmnt 142 /opt/ofs/bin/pvfs2-ls /pvfsmnt
143 143
144 If stuff seems to be working, load the kernel 144 If stuff seems to be working, load the kernel module and
145 turn on the client core:: 145 turn on the client core::
146 146
147 /opt/ofs/sbin/pvfs2-client -p /opt/ofs/sbi 147 /opt/ofs/sbin/pvfs2-client -p /opt/ofs/sbin/pvfs2-client-core
148 148
149 Mount your filesystem:: 149 Mount your filesystem::
150 150
151 mount -t pvfs2 tcp://`hostname`:3334/orang 151 mount -t pvfs2 tcp://`hostname`:3334/orangefs /pvfsmnt
152 152
153 153
154 Running xfstests 154 Running xfstests
155 ================ 155 ================
156 156
157 It is useful to use a scratch filesystem with 157 It is useful to use a scratch filesystem with xfstests. This can be
158 done with only one server. 158 done with only one server.
159 159
160 Make a second copy of the FileSystem section i 160 Make a second copy of the FileSystem section in the server configuration
161 file, which is /etc/orangefs/orangefs.conf. C 161 file, which is /etc/orangefs/orangefs.conf. Change the Name to scratch.
162 Change the ID to something other than the ID o 162 Change the ID to something other than the ID of the first FileSystem
163 section (2 is usually a good choice). 163 section (2 is usually a good choice).
164 164
165 Then there are two FileSystem sections: orange 165 Then there are two FileSystem sections: orangefs and scratch.
166 166
167 This change should be made before creating the 167 This change should be made before creating the filesystem.
168 168
169 :: 169 ::
170 170
171 pvfs2-server -f /etc/orangefs/orangefs.con 171 pvfs2-server -f /etc/orangefs/orangefs.conf
172 172
173 To run xfstests, create /etc/xfsqa.config:: 173 To run xfstests, create /etc/xfsqa.config::
174 174
175 TEST_DIR=/orangefs 175 TEST_DIR=/orangefs
176 TEST_DEV=tcp://localhost:3334/orangefs 176 TEST_DEV=tcp://localhost:3334/orangefs
177 SCRATCH_MNT=/scratch 177 SCRATCH_MNT=/scratch
178 SCRATCH_DEV=tcp://localhost:3334/scratch 178 SCRATCH_DEV=tcp://localhost:3334/scratch
179 179
180 Then xfstests can be run:: 180 Then xfstests can be run::
181 181
182 ./check -pvfs2 182 ./check -pvfs2
183 183
184 184
185 Options 185 Options
186 ======= 186 =======
187 187
188 The following mount options are accepted: 188 The following mount options are accepted:
189 189
190 acl 190 acl
191 Allow the use of Access Control Lists on f 191 Allow the use of Access Control Lists on files and directories.
192 192
193 intr 193 intr
194 Some operations between the kernel client 194 Some operations between the kernel client and the user space
195 filesystem can be interruptible, such as c 195 filesystem can be interruptible, such as changes in debug levels
196 and the setting of tunable parameters. 196 and the setting of tunable parameters.
197 197
198 local_lock 198 local_lock
199 Enable posix locking from the perspective 199 Enable posix locking from the perspective of "this" kernel. The
200 default file_operations lock action is to 200 default file_operations lock action is to return ENOSYS. Posix
201 locking kicks in if the filesystem is moun 201 locking kicks in if the filesystem is mounted with -o local_lock.
202 Distributed locking is being worked on for 202 Distributed locking is being worked on for the future.
203 203
204 204
205 Debugging 205 Debugging
206 ========= 206 =========
207 207
208 If you want the debug (GOSSIP) statements in a 208 If you want the debug (GOSSIP) statements in a particular
209 source file (inode.c for example) go to syslog 209 source file (inode.c for example) go to syslog::
210 210
211 echo inode > /sys/kernel/debug/orangefs/kern 211 echo inode > /sys/kernel/debug/orangefs/kernel-debug
212 212
213 No debugging (the default):: 213 No debugging (the default)::
214 214
215 echo none > /sys/kernel/debug/orangefs/kerne 215 echo none > /sys/kernel/debug/orangefs/kernel-debug
216 216
217 Debugging from several source files:: 217 Debugging from several source files::
218 218
219 echo inode,dir > /sys/kernel/debug/orangefs/ 219 echo inode,dir > /sys/kernel/debug/orangefs/kernel-debug
220 220
221 All debugging:: 221 All debugging::
222 222
223 echo all > /sys/kernel/debug/orangefs/kernel 223 echo all > /sys/kernel/debug/orangefs/kernel-debug
224 224
225 Get a list of all debugging keywords:: 225 Get a list of all debugging keywords::
226 226
227 cat /sys/kernel/debug/orangefs/debug-help 227 cat /sys/kernel/debug/orangefs/debug-help
228 228
229 229
230 Protocol between Kernel Module and Userspace 230 Protocol between Kernel Module and Userspace
231 ============================================ 231 ============================================
232 232
233 Orangefs is a user space filesystem and an ass 233 Orangefs is a user space filesystem and an associated kernel module.
234 We'll just refer to the user space part of Ora 234 We'll just refer to the user space part of Orangefs as "userspace"
235 from here on out. Orangefs descends from PVFS, 235 from here on out. Orangefs descends from PVFS, and userspace code
236 still uses PVFS for function and variable name 236 still uses PVFS for function and variable names. Userspace typedefs
237 many of the important structures. Function and 237 many of the important structures. Function and variable names in
238 the kernel module have been transitioned to "o 238 the kernel module have been transitioned to "orangefs", and The Linux
239 Coding Style avoids typedefs, so kernel module 239 Coding Style avoids typedefs, so kernel module structures that
240 correspond to userspace structures are not typ 240 correspond to userspace structures are not typedefed.
241 241
242 The kernel module implements a pseudo device t 242 The kernel module implements a pseudo device that userspace
243 can read from and write to. Userspace can also 243 can read from and write to. Userspace can also manipulate the
244 kernel module through the pseudo device with i 244 kernel module through the pseudo device with ioctl.
245 245
246 The Bufmap 246 The Bufmap
247 ---------- 247 ----------
248 248
249 At startup userspace allocates two page-size-a 249 At startup userspace allocates two page-size-aligned (posix_memalign)
250 mlocked memory buffers, one is used for IO and 250 mlocked memory buffers, one is used for IO and one is used for readdir
251 operations. The IO buffer is 41943040 bytes an 251 operations. The IO buffer is 41943040 bytes and the readdir buffer is
252 4194304 bytes. Each buffer contains logical ch 252 4194304 bytes. Each buffer contains logical chunks, or partitions, and
253 a pointer to each buffer is added to its own P 253 a pointer to each buffer is added to its own PVFS_dev_map_desc structure
254 which also describes its total size, as well a 254 which also describes its total size, as well as the size and number of
255 the partitions. 255 the partitions.
256 256
257 A pointer to the IO buffer's PVFS_dev_map_desc 257 A pointer to the IO buffer's PVFS_dev_map_desc structure is sent to a
258 mapping routine in the kernel module with an i 258 mapping routine in the kernel module with an ioctl. The structure is
259 copied from user space to kernel space with co 259 copied from user space to kernel space with copy_from_user and is used
260 to initialize the kernel module's "bufmap" (st 260 to initialize the kernel module's "bufmap" (struct orangefs_bufmap), which
261 then contains: 261 then contains:
262 262
263 * refcnt 263 * refcnt
264 - a reference counter 264 - a reference counter
265 * desc_size - PVFS2_BUFMAP_DEFAULT_DESC_SIZE 265 * desc_size - PVFS2_BUFMAP_DEFAULT_DESC_SIZE (4194304) - the IO buffer's
266 partition size, which represents the files 266 partition size, which represents the filesystem's block size and
267 is used for s_blocksize in super blocks. 267 is used for s_blocksize in super blocks.
268 * desc_count - PVFS2_BUFMAP_DEFAULT_DESC_COU 268 * desc_count - PVFS2_BUFMAP_DEFAULT_DESC_COUNT (10) - the number of
269 partitions in the IO buffer. 269 partitions in the IO buffer.
270 * desc_shift - log2(desc_size), used for s_b 270 * desc_shift - log2(desc_size), used for s_blocksize_bits in super blocks.
271 * total_size - the total size of the IO buff 271 * total_size - the total size of the IO buffer.
272 * page_count - the number of 4096 byte pages 272 * page_count - the number of 4096 byte pages in the IO buffer.
273 * page_array - a pointer to ``page_count * ( 273 * page_array - a pointer to ``page_count * (sizeof(struct page*))`` bytes
274 of kcalloced memory. This memory is used a 274 of kcalloced memory. This memory is used as an array of pointers
275 to each of the pages in the IO buffer thro 275 to each of the pages in the IO buffer through a call to get_user_pages.
276 * desc_array - a pointer to ``desc_count * ( 276 * desc_array - a pointer to ``desc_count * (sizeof(struct orangefs_bufmap_desc))``
277 bytes of kcalloced memory. This memory is 277 bytes of kcalloced memory. This memory is further initialized:
278 278
279 user_desc is the kernel's copy of the IO 279 user_desc is the kernel's copy of the IO buffer's ORANGEFS_dev_map_desc
280 structure. user_desc->ptr points to the 280 structure. user_desc->ptr points to the IO buffer.
281 281
282 :: 282 ::
283 283
284 pages_per_desc = bufmap->desc_size / P 284 pages_per_desc = bufmap->desc_size / PAGE_SIZE
285 offset = 0 285 offset = 0
286 286
287 bufmap->desc_array[0].page_array = &bu 287 bufmap->desc_array[0].page_array = &bufmap->page_array[offset]
288 bufmap->desc_array[0].array_count = pa 288 bufmap->desc_array[0].array_count = pages_per_desc = 1024
289 bufmap->desc_array[0].uaddr = (user_de 289 bufmap->desc_array[0].uaddr = (user_desc->ptr) + (0 * 1024 * 4096)
290 offset += 1024 290 offset += 1024
291 . 291 .
292 . 292 .
293 . 293 .
294 bufmap->desc_array[9].page_array = &bu 294 bufmap->desc_array[9].page_array = &bufmap->page_array[offset]
295 bufmap->desc_array[9].array_count = pa 295 bufmap->desc_array[9].array_count = pages_per_desc = 1024
296 bufmap->desc_array[9].uaddr = (user_de 296 bufmap->desc_array[9].uaddr = (user_desc->ptr) +
297 297 (9 * 1024 * 4096)
298 offset += 1024 298 offset += 1024
299 299
300 * buffer_index_array - a desc_count sized ar 300 * buffer_index_array - a desc_count sized array of ints, used to
301 indicate which of the IO buffer's partitio 301 indicate which of the IO buffer's partitions are available to use.
302 * buffer_index_lock - a spinlock to protect 302 * buffer_index_lock - a spinlock to protect buffer_index_array during update.
303 * readdir_index_array - a five (ORANGEFS_REA 303 * readdir_index_array - a five (ORANGEFS_READDIR_DEFAULT_DESC_COUNT) element
304 int array used to indicate which of the re 304 int array used to indicate which of the readdir buffer's partitions are
305 available to use. 305 available to use.
306 * readdir_index_lock - a spinlock to protect 306 * readdir_index_lock - a spinlock to protect readdir_index_array during
307 update. 307 update.
308 308
309 Operations 309 Operations
310 ---------- 310 ----------
311 311
312 The kernel module builds an "op" (struct orang 312 The kernel module builds an "op" (struct orangefs_kernel_op_s) when it
313 needs to communicate with userspace. Part of t 313 needs to communicate with userspace. Part of the op contains the "upcall"
314 which expresses the request to userspace. Part 314 which expresses the request to userspace. Part of the op eventually
315 contains the "downcall" which expresses the re 315 contains the "downcall" which expresses the results of the request.
316 316
317 The slab allocator is used to keep a cache of 317 The slab allocator is used to keep a cache of op structures handy.
318 318
319 At init time the kernel module defines and ini 319 At init time the kernel module defines and initializes a request list
320 and an in_progress hash table to keep track of 320 and an in_progress hash table to keep track of all the ops that are
321 in flight at any given time. 321 in flight at any given time.
322 322
323 Ops are stateful: 323 Ops are stateful:
324 324
325 * unknown 325 * unknown
326 - op was just initialized 326 - op was just initialized
327 * waiting 327 * waiting
328 - op is on request_list (upward bo 328 - op is on request_list (upward bound)
329 * inprogr 329 * inprogr
330 - op is in progress (waiting for d 330 - op is in progress (waiting for downcall)
331 * serviced 331 * serviced
332 - op has matching downcall; ok 332 - op has matching downcall; ok
333 * purged 333 * purged
334 - op has to start a timer since cl 334 - op has to start a timer since client-core
335 exited uncleanly before servicin 335 exited uncleanly before servicing op
336 * given up 336 * given up
337 - submitter has given up waiting f 337 - submitter has given up waiting for it
338 338
339 When some arbitrary userspace program needs to 339 When some arbitrary userspace program needs to perform a
340 filesystem operation on Orangefs (readdir, I/O 340 filesystem operation on Orangefs (readdir, I/O, create, whatever)
341 an op structure is initialized and tagged with 341 an op structure is initialized and tagged with a distinguishing ID
342 number. The upcall part of the op is filled ou 342 number. The upcall part of the op is filled out, and the op is
343 passed to the "service_operation" function. 343 passed to the "service_operation" function.
344 344
345 Service_operation changes the op's state to "w 345 Service_operation changes the op's state to "waiting", puts
346 it on the request list, and signals the Orange 346 it on the request list, and signals the Orangefs file_operations.poll
347 function through a wait queue. Userspace is po 347 function through a wait queue. Userspace is polling the pseudo-device
348 and thus becomes aware of the upcall request t 348 and thus becomes aware of the upcall request that needs to be read.
349 349
350 When the Orangefs file_operations.read functio 350 When the Orangefs file_operations.read function is triggered, the
351 request list is searched for an op that seems 351 request list is searched for an op that seems ready-to-process.
352 The op is removed from the request list. The t 352 The op is removed from the request list. The tag from the op and
353 the filled-out upcall struct are copy_to_user' 353 the filled-out upcall struct are copy_to_user'ed back to userspace.
354 354
355 If any of these (and some additional protocol) 355 If any of these (and some additional protocol) copy_to_users fail,
356 the op's state is set to "waiting" and the op 356 the op's state is set to "waiting" and the op is added back to
357 the request list. Otherwise, the op's state is 357 the request list. Otherwise, the op's state is changed to "in progress",
358 and the op is hashed on its tag and put onto t 358 and the op is hashed on its tag and put onto the end of a list in the
359 in_progress hash table at the index the tag ha 359 in_progress hash table at the index the tag hashed to.
360 360
361 When userspace has assembled the response to t 361 When userspace has assembled the response to the upcall, it
362 writes the response, which includes the distin 362 writes the response, which includes the distinguishing tag, back to
363 the pseudo device in a series of io_vecs. This 363 the pseudo device in a series of io_vecs. This triggers the Orangefs
364 file_operations.write_iter function to find th 364 file_operations.write_iter function to find the op with the associated
365 tag and remove it from the in_progress hash ta 365 tag and remove it from the in_progress hash table. As long as the op's
366 state is not "canceled" or "given up", its sta 366 state is not "canceled" or "given up", its state is set to "serviced".
367 The file_operations.write_iter function return 367 The file_operations.write_iter function returns to the waiting vfs,
368 and back to service_operation through wait_for 368 and back to service_operation through wait_for_matching_downcall.
369 369
370 Service operation returns to its caller with t 370 Service operation returns to its caller with the op's downcall
371 part (the response to the upcall) filled out. 371 part (the response to the upcall) filled out.
372 372
373 The "client-core" is the bridge between the ke 373 The "client-core" is the bridge between the kernel module and
374 userspace. The client-core is a daemon. The cl 374 userspace. The client-core is a daemon. The client-core has an
375 associated watchdog daemon. If the client-core 375 associated watchdog daemon. If the client-core is ever signaled
376 to die, the watchdog daemon restarts the clien 376 to die, the watchdog daemon restarts the client-core. Even though
377 the client-core is restarted "right away", the 377 the client-core is restarted "right away", there is a period of
378 time during such an event that the client-core 378 time during such an event that the client-core is dead. A dead client-core
379 can't be triggered by the Orangefs file_operat 379 can't be triggered by the Orangefs file_operations.poll function.
380 Ops that pass through service_operation during 380 Ops that pass through service_operation during a "dead spell" can timeout
381 on the wait queue and one attempt is made to r 381 on the wait queue and one attempt is made to recycle them. Obviously,
382 if the client-core stays dead too long, the ar 382 if the client-core stays dead too long, the arbitrary userspace processes
383 trying to use Orangefs will be negatively affe 383 trying to use Orangefs will be negatively affected. Waiting ops
384 that can't be serviced will be removed from th 384 that can't be serviced will be removed from the request list and
385 have their states set to "given up". In-progre 385 have their states set to "given up". In-progress ops that can't
386 be serviced will be removed from the in_progre 386 be serviced will be removed from the in_progress hash table and
387 have their states set to "given up". 387 have their states set to "given up".
388 388
389 Readdir and I/O ops are atypical with respect 389 Readdir and I/O ops are atypical with respect to their payloads.
390 390
391 - readdir ops use the smaller of the two pre 391 - readdir ops use the smaller of the two pre-allocated pre-partitioned
392 memory buffers. The readdir buffer is only 392 memory buffers. The readdir buffer is only available to userspace.
393 The kernel module obtains an index to a fr 393 The kernel module obtains an index to a free partition before launching
394 a readdir op. Userspace deposits the resul 394 a readdir op. Userspace deposits the results into the indexed partition
395 and then writes them to back to the pvfs d 395 and then writes them to back to the pvfs device.
396 396
397 - io (read and write) ops use the larger of 397 - io (read and write) ops use the larger of the two pre-allocated
398 pre-partitioned memory buffers. The IO buf 398 pre-partitioned memory buffers. The IO buffer is accessible from
399 both userspace and the kernel module. The 399 both userspace and the kernel module. The kernel module obtains an
400 index to a free partition before launching 400 index to a free partition before launching an io op. The kernel module
401 deposits write data into the indexed parti 401 deposits write data into the indexed partition, to be consumed
402 directly by userspace. Userspace deposits 402 directly by userspace. Userspace deposits the results of read
403 requests into the indexed partition, to be 403 requests into the indexed partition, to be consumed directly
404 by the kernel module. 404 by the kernel module.
405 405
406 Responses to kernel requests are all packaged 406 Responses to kernel requests are all packaged in pvfs2_downcall_t
407 structs. Besides a few other members, pvfs2_do 407 structs. Besides a few other members, pvfs2_downcall_t contains a
408 union of structs, each of which is associated 408 union of structs, each of which is associated with a particular
409 response type. 409 response type.
410 410
411 The several members outside of the union are: 411 The several members outside of the union are:
412 412
413 ``int32_t type`` 413 ``int32_t type``
414 - type of operation. 414 - type of operation.
415 ``int32_t status`` 415 ``int32_t status``
416 - return code for the operation. 416 - return code for the operation.
417 ``int64_t trailer_size`` 417 ``int64_t trailer_size``
418 - 0 unless readdir operation. 418 - 0 unless readdir operation.
419 ``char *trailer_buf`` 419 ``char *trailer_buf``
420 - initialized to NULL, used during readdir 420 - initialized to NULL, used during readdir operations.
421 421
422 The appropriate member inside the union is fil 422 The appropriate member inside the union is filled out for any
423 particular response. 423 particular response.
424 424
425 PVFS2_VFS_OP_FILE_IO 425 PVFS2_VFS_OP_FILE_IO
426 fill a pvfs2_io_response_t 426 fill a pvfs2_io_response_t
427 427
428 PVFS2_VFS_OP_LOOKUP 428 PVFS2_VFS_OP_LOOKUP
429 fill a PVFS_object_kref 429 fill a PVFS_object_kref
430 430
431 PVFS2_VFS_OP_CREATE 431 PVFS2_VFS_OP_CREATE
432 fill a PVFS_object_kref 432 fill a PVFS_object_kref
433 433
434 PVFS2_VFS_OP_SYMLINK 434 PVFS2_VFS_OP_SYMLINK
435 fill a PVFS_object_kref 435 fill a PVFS_object_kref
436 436
437 PVFS2_VFS_OP_GETATTR 437 PVFS2_VFS_OP_GETATTR
438 fill in a PVFS_sys_attr_s (tons of stuff t 438 fill in a PVFS_sys_attr_s (tons of stuff the kernel doesn't need)
439 fill in a string with the link target when 439 fill in a string with the link target when the object is a symlink.
440 440
441 PVFS2_VFS_OP_MKDIR 441 PVFS2_VFS_OP_MKDIR
442 fill a PVFS_object_kref 442 fill a PVFS_object_kref
443 443
444 PVFS2_VFS_OP_STATFS 444 PVFS2_VFS_OP_STATFS
445 fill a pvfs2_statfs_response_t with useles 445 fill a pvfs2_statfs_response_t with useless info <g>. It is hard for
446 us to know, in a timely fashion, these sta 446 us to know, in a timely fashion, these statistics about our
447 distributed network filesystem. 447 distributed network filesystem.
448 448
449 PVFS2_VFS_OP_FS_MOUNT 449 PVFS2_VFS_OP_FS_MOUNT
450 fill a pvfs2_fs_mount_response_t which is 450 fill a pvfs2_fs_mount_response_t which is just like a PVFS_object_kref
451 except its members are in a different orde 451 except its members are in a different order and "__pad1" is replaced
452 with "id". 452 with "id".
453 453
454 PVFS2_VFS_OP_GETXATTR 454 PVFS2_VFS_OP_GETXATTR
455 fill a pvfs2_getxattr_response_t 455 fill a pvfs2_getxattr_response_t
456 456
457 PVFS2_VFS_OP_LISTXATTR 457 PVFS2_VFS_OP_LISTXATTR
458 fill a pvfs2_listxattr_response_t 458 fill a pvfs2_listxattr_response_t
459 459
460 PVFS2_VFS_OP_PARAM 460 PVFS2_VFS_OP_PARAM
461 fill a pvfs2_param_response_t 461 fill a pvfs2_param_response_t
462 462
463 PVFS2_VFS_OP_PERF_COUNT 463 PVFS2_VFS_OP_PERF_COUNT
464 fill a pvfs2_perf_count_response_t 464 fill a pvfs2_perf_count_response_t
465 465
466 PVFS2_VFS_OP_FSKEY 466 PVFS2_VFS_OP_FSKEY
467 file a pvfs2_fs_key_response_t 467 file a pvfs2_fs_key_response_t
468 468
469 PVFS2_VFS_OP_READDIR 469 PVFS2_VFS_OP_READDIR
470 jamb everything needed to represent a pvfs 470 jamb everything needed to represent a pvfs2_readdir_response_t into
471 the readdir buffer descriptor specified in 471 the readdir buffer descriptor specified in the upcall.
472 472
473 Userspace uses writev() on /dev/pvfs2-req to p 473 Userspace uses writev() on /dev/pvfs2-req to pass responses to the requests
474 made by the kernel side. 474 made by the kernel side.
475 475
476 A buffer_list containing: 476 A buffer_list containing:
477 477
478 - a pointer to the prepared response to the 478 - a pointer to the prepared response to the request from the
479 kernel (struct pvfs2_downcall_t). 479 kernel (struct pvfs2_downcall_t).
480 - and also, in the case of a readdir request 480 - and also, in the case of a readdir request, a pointer to a
481 buffer containing descriptors for the obje 481 buffer containing descriptors for the objects in the target
482 directory. 482 directory.
483 483
484 ... is sent to the function (PINT_dev_write_li 484 ... is sent to the function (PINT_dev_write_list) which performs
485 the writev. 485 the writev.
486 486
487 PINT_dev_write_list has a local iovec array: s 487 PINT_dev_write_list has a local iovec array: struct iovec io_array[10];
488 488
489 The first four elements of io_array are initia 489 The first four elements of io_array are initialized like this for all
490 responses:: 490 responses::
491 491
492 io_array[0].iov_base = address of local vari 492 io_array[0].iov_base = address of local variable "proto_ver" (int32_t)
493 io_array[0].iov_len = sizeof(int32_t) 493 io_array[0].iov_len = sizeof(int32_t)
494 494
495 io_array[1].iov_base = address of global var 495 io_array[1].iov_base = address of global variable "pdev_magic" (int32_t)
496 io_array[1].iov_len = sizeof(int32_t) 496 io_array[1].iov_len = sizeof(int32_t)
497 497
498 io_array[2].iov_base = address of parameter 498 io_array[2].iov_base = address of parameter "tag" (PVFS_id_gen_t)
499 io_array[2].iov_len = sizeof(int64_t) 499 io_array[2].iov_len = sizeof(int64_t)
500 500
501 io_array[3].iov_base = address of out_downca 501 io_array[3].iov_base = address of out_downcall member (pvfs2_downcall_t)
502 of global variable vf 502 of global variable vfs_request (vfs_request_t)
503 io_array[3].iov_len = sizeof(pvfs2_downcall_ 503 io_array[3].iov_len = sizeof(pvfs2_downcall_t)
504 504
505 Readdir responses initialize the fifth element 505 Readdir responses initialize the fifth element io_array like this::
506 506
507 io_array[4].iov_base = contents of member tr 507 io_array[4].iov_base = contents of member trailer_buf (char *)
508 from out_downcall mem 508 from out_downcall member of global variable
509 vfs_request 509 vfs_request
510 io_array[4].iov_len = contents of member tra 510 io_array[4].iov_len = contents of member trailer_size (PVFS_size)
511 from out_downcall memb 511 from out_downcall member of global variable
512 vfs_request 512 vfs_request
513 513
514 Orangefs exploits the dcache in order to avoid 514 Orangefs exploits the dcache in order to avoid sending redundant
515 requests to userspace. We keep object inode at 515 requests to userspace. We keep object inode attributes up-to-date with
516 orangefs_inode_getattr. Orangefs_inode_getattr 516 orangefs_inode_getattr. Orangefs_inode_getattr uses two arguments to
517 help it decide whether or not to update an ino 517 help it decide whether or not to update an inode: "new" and "bypass".
518 Orangefs keeps private data in an object's ino 518 Orangefs keeps private data in an object's inode that includes a short
519 timeout value, getattr_time, which allows any 519 timeout value, getattr_time, which allows any iteration of
520 orangefs_inode_getattr to know how long it has 520 orangefs_inode_getattr to know how long it has been since the inode was
521 updated. When the object is not new (new == 0) 521 updated. When the object is not new (new == 0) and the bypass flag is not
522 set (bypass == 0) orangefs_inode_getattr retur 522 set (bypass == 0) orangefs_inode_getattr returns without updating the inode
523 if getattr_time has not timed out. Getattr_tim 523 if getattr_time has not timed out. Getattr_time is updated each time the
524 inode is updated. 524 inode is updated.
525 525
526 Creation of a new object (file, dir, sym-link) 526 Creation of a new object (file, dir, sym-link) includes the evaluation of
527 its pathname, resulting in a negative director 527 its pathname, resulting in a negative directory entry for the object.
528 A new inode is allocated and associated with t 528 A new inode is allocated and associated with the dentry, turning it from
529 a negative dentry into a "productive full memb 529 a negative dentry into a "productive full member of society". Orangefs
530 obtains the new inode from Linux with new_inod 530 obtains the new inode from Linux with new_inode() and associates
531 the inode with the dentry by sending the pair 531 the inode with the dentry by sending the pair back to Linux with
532 d_instantiate(). 532 d_instantiate().
533 533
534 The evaluation of a pathname for an object res 534 The evaluation of a pathname for an object resolves to its corresponding
535 dentry. If there is no corresponding dentry, o 535 dentry. If there is no corresponding dentry, one is created for it in
536 the dcache. Whenever a dentry is modified or v 536 the dcache. Whenever a dentry is modified or verified Orangefs stores a
537 short timeout value in the dentry's d_time, an 537 short timeout value in the dentry's d_time, and the dentry will be trusted
538 for that amount of time. Orangefs is a network 538 for that amount of time. Orangefs is a network filesystem, and objects
539 can potentially change out-of-band with any pa 539 can potentially change out-of-band with any particular Orangefs kernel module
540 instance, so trusting a dentry is risky. The a 540 instance, so trusting a dentry is risky. The alternative to trusting
541 dentries is to always obtain the needed inform 541 dentries is to always obtain the needed information from userspace - at
542 least a trip to the client-core, maybe to the 542 least a trip to the client-core, maybe to the servers. Obtaining information
543 from a dentry is cheap, obtaining it from user 543 from a dentry is cheap, obtaining it from userspace is relatively expensive,
544 hence the motivation to use the dentry when po 544 hence the motivation to use the dentry when possible.
545 545
546 The timeout values d_time and getattr_time are 546 The timeout values d_time and getattr_time are jiffy based, and the
547 code is designed to avoid the jiffy-wrap probl 547 code is designed to avoid the jiffy-wrap problem::
548 548
549 "In general, if the clock may have wrapped 549 "In general, if the clock may have wrapped around more than once, there
550 is no way to tell how much time has elapse 550 is no way to tell how much time has elapsed. However, if the times t1
551 and t2 are known to be fairly close, we ca 551 and t2 are known to be fairly close, we can reliably compute the
552 difference in a way that takes into accoun 552 difference in a way that takes into account the possibility that the
553 clock may have wrapped between times." 553 clock may have wrapped between times."
554 554
555 from course notes by instructor Andy Wang 555 from course notes by instructor Andy Wang
556 556
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.