1 .. _filesystems_index: !! 1 =====================
>> 2 Linux Filesystems API
>> 3 =====================
2 4
3 =============================== !! 5 The Linux VFS
4 Filesystems in the Linux kernel !! 6 =============
5 =============================== <<
6 <<
7 This under-development manual will, some glori <<
8 comprehensive information on how the Linux vir <<
9 works, along with the filesystems that sit bel <<
10 can be found below. <<
11 7
12 Core VFS documentation !! 8 The Filesystem types
13 ====================== !! 9 --------------------
14 10
15 See these manuals for documentation about the !! 11 .. kernel-doc:: include/linux/fs.h
16 algorithms work. !! 12 :internal:
17 13
18 .. toctree:: !! 14 The Directory Cache
19 :maxdepth: 2 !! 15 -------------------
>> 16
>> 17 .. kernel-doc:: fs/dcache.c
>> 18 :export:
>> 19
>> 20 .. kernel-doc:: include/linux/dcache.h
>> 21 :internal:
>> 22
>> 23 Inode Handling
>> 24 --------------
>> 25
>> 26 .. kernel-doc:: fs/inode.c
>> 27 :export:
>> 28
>> 29 .. kernel-doc:: fs/bad_inode.c
>> 30 :export:
>> 31
>> 32 Registration and Superblocks
>> 33 ----------------------------
>> 34
>> 35 .. kernel-doc:: fs/super.c
>> 36 :export:
>> 37
>> 38 File Locks
>> 39 ----------
>> 40
>> 41 .. kernel-doc:: fs/locks.c
>> 42 :export:
>> 43
>> 44 .. kernel-doc:: fs/locks.c
>> 45 :internal:
>> 46
>> 47 Other Functions
>> 48 ---------------
>> 49
>> 50 .. kernel-doc:: fs/mpage.c
>> 51 :export:
>> 52
>> 53 .. kernel-doc:: fs/namei.c
>> 54 :export:
>> 55
>> 56 .. kernel-doc:: fs/buffer.c
>> 57 :export:
>> 58
>> 59 .. kernel-doc:: block/bio.c
>> 60 :export:
>> 61
>> 62 .. kernel-doc:: fs/seq_file.c
>> 63 :export:
>> 64
>> 65 .. kernel-doc:: fs/filesystems.c
>> 66 :export:
>> 67
>> 68 .. kernel-doc:: fs/fs-writeback.c
>> 69 :export:
>> 70
>> 71 .. kernel-doc:: fs/block_dev.c
>> 72 :export:
>> 73
>> 74 .. kernel-doc:: fs/anon_inodes.c
>> 75 :export:
>> 76
>> 77 .. kernel-doc:: fs/attr.c
>> 78 :export:
20 79
21 vfs !! 80 .. kernel-doc:: fs/d_path.c
22 path-lookup !! 81 :export:
23 api-summary <<
24 splice <<
25 locking <<
26 directory-locking <<
27 devpts <<
28 dnotify <<
29 fiemap <<
30 files <<
31 locks <<
32 mount_api <<
33 quota <<
34 seq_file <<
35 sharedsubtree <<
36 idmappings <<
37 iomap/index <<
38 82
39 automount-support !! 83 .. kernel-doc:: fs/dax.c
>> 84 :export:
40 85
41 caching/index !! 86 .. kernel-doc:: fs/direct-io.c
>> 87 :export:
42 88
43 porting !! 89 .. kernel-doc:: fs/file_table.c
>> 90 :export:
44 91
45 Filesystem support layers !! 92 .. kernel-doc:: fs/libfs.c
>> 93 :export:
>> 94
>> 95 .. kernel-doc:: fs/posix_acl.c
>> 96 :export:
>> 97
>> 98 .. kernel-doc:: fs/stat.c
>> 99 :export:
>> 100
>> 101 .. kernel-doc:: fs/sync.c
>> 102 :export:
>> 103
>> 104 .. kernel-doc:: fs/xattr.c
>> 105 :export:
>> 106
>> 107 The proc filesystem
>> 108 ===================
>> 109
>> 110 sysctl interface
>> 111 ----------------
>> 112
>> 113 .. kernel-doc:: kernel/sysctl.c
>> 114 :export:
>> 115
>> 116 proc filesystem interface
>> 117 -------------------------
>> 118
>> 119 .. kernel-doc:: fs/proc/base.c
>> 120 :internal:
>> 121
>> 122 Events based on file descriptors
>> 123 ================================
>> 124
>> 125 .. kernel-doc:: fs/eventfd.c
>> 126 :export:
>> 127
>> 128 The Filesystem for Exporting Kernel Objects
>> 129 ===========================================
>> 130
>> 131 .. kernel-doc:: fs/sysfs/file.c
>> 132 :export:
>> 133
>> 134 .. kernel-doc:: fs/sysfs/symlink.c
>> 135 :export:
>> 136
>> 137 The debugfs filesystem
>> 138 ======================
>> 139
>> 140 debugfs interface
>> 141 -----------------
>> 142
>> 143 .. kernel-doc:: fs/debugfs/inode.c
>> 144 :export:
>> 145
>> 146 .. kernel-doc:: fs/debugfs/file.c
>> 147 :export:
>> 148
>> 149 The Linux Journalling API
46 ========================= 150 =========================
47 151
48 Documentation for the support code within the !! 152 Overview
49 filesystem implementations. !! 153 --------
50 154
51 .. toctree:: !! 155 Details
52 :maxdepth: 2 !! 156 ~~~~~~~
>> 157
>> 158 The journalling layer is easy to use. You need to first of all create a
>> 159 journal_t data structure. There are two calls to do this dependent on
>> 160 how you decide to allocate the physical media on which the journal
>> 161 resides. The :c:func:`jbd2_journal_init_inode` call is for journals stored in
>> 162 filesystem inodes, or the :c:func:`jbd2_journal_init_dev` call can be used
>> 163 for journal stored on a raw device (in a continuous range of blocks). A
>> 164 journal_t is a typedef for a struct pointer, so when you are finally
>> 165 finished make sure you call :c:func:`jbd2_journal_destroy` on it to free up
>> 166 any used kernel memory.
>> 167
>> 168 Once you have got your journal_t object you need to 'mount' or load the
>> 169 journal file. The journalling layer expects the space for the journal
>> 170 was already allocated and initialized properly by the userspace tools.
>> 171 When loading the journal you must call :c:func:`jbd2_journal_load` to process
>> 172 journal contents. If the client file system detects the journal contents
>> 173 does not need to be processed (or even need not have valid contents), it
>> 174 may call :c:func:`jbd2_journal_wipe` to clear the journal contents before
>> 175 calling :c:func:`jbd2_journal_load`.
>> 176
>> 177 Note that jbd2_journal_wipe(..,0) calls
>> 178 :c:func:`jbd2_journal_skip_recovery` for you if it detects any outstanding
>> 179 transactions in the journal and similarly :c:func:`jbd2_journal_load` will
>> 180 call :c:func:`jbd2_journal_recover` if necessary. I would advise reading
>> 181 :c:func:`ext4_load_journal` in fs/ext4/super.c for examples on this stage.
>> 182
>> 183 Now you can go ahead and start modifying the underlying filesystem.
>> 184 Almost.
>> 185
>> 186 You still need to actually journal your filesystem changes, this is done
>> 187 by wrapping them into transactions. Additionally you also need to wrap
>> 188 the modification of each of the buffers with calls to the journal layer,
>> 189 so it knows what the modifications you are actually making are. To do
>> 190 this use :c:func:`jbd2_journal_start` which returns a transaction handle.
>> 191
>> 192 :c:func:`jbd2_journal_start` and its counterpart :c:func:`jbd2_journal_stop`,
>> 193 which indicates the end of a transaction are nestable calls, so you can
>> 194 reenter a transaction if necessary, but remember you must call
>> 195 :c:func:`jbd2_journal_stop` the same number of times as
>> 196 :c:func:`jbd2_journal_start` before the transaction is completed (or more
>> 197 accurately leaves the update phase). Ext4/VFS makes use of this feature to
>> 198 simplify handling of inode dirtying, quota support, etc.
>> 199
>> 200 Inside each transaction you need to wrap the modifications to the
>> 201 individual buffers (blocks). Before you start to modify a buffer you
>> 202 need to call :c:func:`jbd2_journal_get_create_access()` /
>> 203 :c:func:`jbd2_journal_get_write_access()` /
>> 204 :c:func:`jbd2_journal_get_undo_access()` as appropriate, this allows the
>> 205 journalling layer to copy the unmodified
>> 206 data if it needs to. After all the buffer may be part of a previously
>> 207 uncommitted transaction. At this point you are at last ready to modify a
>> 208 buffer, and once you are have done so you need to call
>> 209 :c:func:`jbd2_journal_dirty_metadata`. Or if you've asked for access to a
>> 210 buffer you now know is now longer required to be pushed back on the
>> 211 device you can call :c:func:`jbd2_journal_forget` in much the same way as you
>> 212 might have used :c:func:`bforget` in the past.
>> 213
>> 214 A :c:func:`jbd2_journal_flush` may be called at any time to commit and
>> 215 checkpoint all your transactions.
>> 216
>> 217 Then at umount time , in your :c:func:`put_super` you can then call
>> 218 :c:func:`jbd2_journal_destroy` to clean up your in-core journal object.
>> 219
>> 220 Unfortunately there a couple of ways the journal layer can cause a
>> 221 deadlock. The first thing to note is that each task can only have a
>> 222 single outstanding transaction at any one time, remember nothing commits
>> 223 until the outermost :c:func:`jbd2_journal_stop`. This means you must complete
>> 224 the transaction at the end of each file/inode/address etc. operation you
>> 225 perform, so that the journalling system isn't re-entered on another
>> 226 journal. Since transactions can't be nested/batched across differing
>> 227 journals, and another filesystem other than yours (say ext4) may be
>> 228 modified in a later syscall.
>> 229
>> 230 The second case to bear in mind is that :c:func:`jbd2_journal_start` can block
>> 231 if there isn't enough space in the journal for your transaction (based
>> 232 on the passed nblocks param) - when it blocks it merely(!) needs to wait
>> 233 for transactions to complete and be committed from other tasks, so
>> 234 essentially we are waiting for :c:func:`jbd2_journal_stop`. So to avoid
>> 235 deadlocks you must treat :c:func:`jbd2_journal_start` /
>> 236 :c:func:`jbd2_journal_stop` as if they were semaphores and include them in
>> 237 your semaphore ordering rules to prevent
>> 238 deadlocks. Note that :c:func:`jbd2_journal_extend` has similar blocking
>> 239 behaviour to :c:func:`jbd2_journal_start` so you can deadlock here just as
>> 240 easily as on :c:func:`jbd2_journal_start`.
>> 241
>> 242 Try to reserve the right number of blocks the first time. ;-). This will
>> 243 be the maximum number of blocks you are going to touch in this
>> 244 transaction. I advise having a look at at least ext4_jbd.h to see the
>> 245 basis on which ext4 uses to make these decisions.
>> 246
>> 247 Another wriggle to watch out for is your on-disk block allocation
>> 248 strategy. Why? Because, if you do a delete, you need to ensure you
>> 249 haven't reused any of the freed blocks until the transaction freeing
>> 250 these blocks commits. If you reused these blocks and crash happens,
>> 251 there is no way to restore the contents of the reallocated blocks at the
>> 252 end of the last fully committed transaction. One simple way of doing
>> 253 this is to mark blocks as free in internal in-memory block allocation
>> 254 structures only after the transaction freeing them commits. Ext4 uses
>> 255 journal commit callback for this purpose.
>> 256
>> 257 With journal commit callbacks you can ask the journalling layer to call
>> 258 a callback function when the transaction is finally committed to disk,
>> 259 so that you can do some of your own management. You ask the journalling
>> 260 layer for calling the callback by simply setting
>> 261 ``journal->j_commit_callback`` function pointer and that function is
>> 262 called after each transaction commit. You can also use
>> 263 ``transaction->t_private_list`` for attaching entries to a transaction
>> 264 that need processing when the transaction commits.
>> 265
>> 266 JBD2 also provides a way to block all transaction updates via
>> 267 :c:func:`jbd2_journal_lock_updates()` /
>> 268 :c:func:`jbd2_journal_unlock_updates()`. Ext4 uses this when it wants a
>> 269 window with a clean and stable fs for a moment. E.g.
>> 270
>> 271 ::
>> 272
>> 273
>> 274 jbd2_journal_lock_updates() //stop new stuff happening..
>> 275 jbd2_journal_flush() // checkpoint everything.
>> 276 ..do stuff on stable fs
>> 277 jbd2_journal_unlock_updates() // carry on with filesystem use.
>> 278
>> 279 The opportunities for abuse and DOS attacks with this should be obvious,
>> 280 if you allow unprivileged userspace to trigger codepaths containing
>> 281 these calls.
>> 282
>> 283 Summary
>> 284 ~~~~~~~
>> 285
>> 286 Using the journal is a matter of wrapping the different context changes,
>> 287 being each mount, each modification (transaction) and each changed
>> 288 buffer to tell the journalling layer about them.
>> 289
>> 290 Data Types
>> 291 ----------
>> 292
>> 293 The journalling layer uses typedefs to 'hide' the concrete definitions
>> 294 of the structures used. As a client of the JBD2 layer you can just rely
>> 295 on the using the pointer as a magic cookie of some sort. Obviously the
>> 296 hiding is not enforced as this is 'C'.
>> 297
>> 298 Structures
>> 299 ~~~~~~~~~~
>> 300
>> 301 .. kernel-doc:: include/linux/jbd2.h
>> 302 :internal:
>> 303
>> 304 Functions
>> 305 ---------
>> 306
>> 307 The functions here are split into two groups those that affect a journal
>> 308 as a whole, and those which are used to manage transactions
>> 309
>> 310 Journal Level
>> 311 ~~~~~~~~~~~~~
>> 312
>> 313 .. kernel-doc:: fs/jbd2/journal.c
>> 314 :export:
>> 315
>> 316 .. kernel-doc:: fs/jbd2/recovery.c
>> 317 :internal:
>> 318
>> 319 Transasction Level
>> 320 ~~~~~~~~~~~~~~~~~~
>> 321
>> 322 .. kernel-doc:: fs/jbd2/transaction.c
>> 323
>> 324 See also
>> 325 --------
>> 326
>> 327 `Journaling the Linux ext2fs Filesystem, LinuxExpo 98, Stephen
>> 328 Tweedie <http://kernel.org/pub/linux/kernel/people/sct/ext3/journal-design.ps.gz>`__
>> 329
>> 330 `Ext3 Journalling FileSystem, OLS 2000, Dr. Stephen
>> 331 Tweedie <http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html>`__
>> 332
>> 333 splice API
>> 334 ==========
>> 335
>> 336 splice is a method for moving blocks of data around inside the kernel,
>> 337 without continually transferring them between the kernel and user space.
>> 338
>> 339 .. kernel-doc:: fs/splice.c
>> 340
>> 341 pipes API
>> 342 =========
>> 343
>> 344 Pipe interfaces are all for in-kernel (builtin image) use. They are not
>> 345 exported for use by modules.
>> 346
>> 347 .. kernel-doc:: include/linux/pipe_fs_i.h
>> 348 :internal:
53 349
54 buffer !! 350 .. kernel-doc:: fs/pipe.c
55 journalling <<
56 fscrypt <<
57 fsverity <<
58 netfs_library <<
59 351
60 Filesystems !! 352 Encryption API
61 =========== !! 353 ==============
62 354
63 Documentation for filesystem implementations. !! 355 A library which filesystems can hook into to support transparent
>> 356 encryption of files and directories.
64 357
65 .. toctree:: 358 .. toctree::
66 :maxdepth: 2 !! 359 :maxdepth: 2
67 360
68 9p !! 361 fscrypt
69 adfs <<
70 affs <<
71 afs <<
72 autofs <<
73 autofs-mount-control <<
74 bcachefs/index <<
75 befs <<
76 bfs <<
77 btrfs <<
78 ceph <<
79 coda <<
80 configfs <<
81 cramfs <<
82 dax <<
83 debugfs <<
84 dlmfs <<
85 ecryptfs <<
86 efivarfs <<
87 erofs <<
88 ext2 <<
89 ext3 <<
90 ext4/index <<
91 f2fs <<
92 gfs2 <<
93 gfs2-uevents <<
94 gfs2-glocks <<
95 hfs <<
96 hfsplus <<
97 hpfs <<
98 fuse <<
99 fuse-io <<
100 inotify <<
101 isofs <<
102 nilfs2 <<
103 nfs/index <<
104 ntfs3 <<
105 ocfs2 <<
106 ocfs2-online-filecheck <<
107 omfs <<
108 orangefs <<
109 overlayfs <<
110 proc <<
111 qnx6 <<
112 ramfs-rootfs-initramfs <<
113 relay <<
114 romfs <<
115 smb/index <<
116 spufs/index <<
117 squashfs <<
118 sysfs <<
119 sysv-fs <<
120 tmpfs <<
121 ubifs <<
122 ubifs-authentication <<
123 udf <<
124 virtiofs <<
125 vfat <<
126 xfs/index <<
127 zonefs <<
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.