1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 Written by: Neil Brown 3 Written by: Neil Brown
4 Please see MAINTAINERS file for where to send 4 Please see MAINTAINERS file for where to send questions.
5 5
6 Overlay Filesystem 6 Overlay Filesystem
7 ================== 7 ==================
8 8
9 This document describes a prototype for a new 9 This document describes a prototype for a new approach to providing
10 overlay-filesystem functionality in Linux (som 10 overlay-filesystem functionality in Linux (sometimes referred to as
11 union-filesystems). An overlay-filesystem tri 11 union-filesystems). An overlay-filesystem tries to present a
12 filesystem which is the result over overlaying 12 filesystem which is the result over overlaying one filesystem on top
13 of the other. 13 of the other.
14 14
15 15
16 Overlay objects 16 Overlay objects
17 --------------- 17 ---------------
18 18
19 The overlay filesystem approach is 'hybrid', b 19 The overlay filesystem approach is 'hybrid', because the objects that
20 appear in the filesystem do not always appear 20 appear in the filesystem do not always appear to belong to that filesystem.
21 In many cases, an object accessed in the union 21 In many cases, an object accessed in the union will be indistinguishable
22 from accessing the corresponding object from t 22 from accessing the corresponding object from the original filesystem.
23 This is most obvious from the 'st_dev' field r 23 This is most obvious from the 'st_dev' field returned by stat(2).
24 24
25 While directories will report an st_dev from t 25 While directories will report an st_dev from the overlay-filesystem,
26 non-directory objects may report an st_dev fro 26 non-directory objects may report an st_dev from the lower filesystem or
27 upper filesystem that is providing the object. 27 upper filesystem that is providing the object. Similarly st_ino will
28 only be unique when combined with st_dev, and 28 only be unique when combined with st_dev, and both of these can change
29 over the lifetime of a non-directory object. 29 over the lifetime of a non-directory object. Many applications and
30 tools ignore these values and will not be affe 30 tools ignore these values and will not be affected.
31 31
32 In the special case of all overlay layers on t 32 In the special case of all overlay layers on the same underlying
33 filesystem, all objects will report an st_dev 33 filesystem, all objects will report an st_dev from the overlay
34 filesystem and st_ino from the underlying file 34 filesystem and st_ino from the underlying filesystem. This will
35 make the overlay mount more compliant with fil 35 make the overlay mount more compliant with filesystem scanners and
36 overlay objects will be distinguishable from t 36 overlay objects will be distinguishable from the corresponding
37 objects in the original filesystem. 37 objects in the original filesystem.
38 38
39 On 64bit systems, even if all overlay layers a 39 On 64bit systems, even if all overlay layers are not on the same
40 underlying filesystem, the same compliant beha 40 underlying filesystem, the same compliant behavior could be achieved
41 with the "xino" feature. The "xino" feature c 41 with the "xino" feature. The "xino" feature composes a unique object
42 identifier from the real object st_ino and an !! 42 identifier from the real object st_ino and an underlying fsid index.
43 The "xino" feature uses the high inode number !! 43 If all underlying filesystems support NFS file handles and export file
44 underlying filesystems rarely use the high ino !! 44 handles with 32bit inode number encoding (e.g. ext4), overlay filesystem
45 the underlying inode number does overflow into !! 45 will use the high inode number bits for fsid. Even when the underlying
46 filesystem will fall back to the non xino beha !! 46 filesystem uses 64bit inode numbers, users can still enable the "xino"
47 !! 47 feature with the "-o xino=on" overlay mount option. That is useful for the
48 The "xino" feature can be enabled with the "-o !! 48 case of underlying filesystems like xfs and tmpfs, which use 64bit inode
49 If all underlying filesystems support NFS file !! 49 numbers, but are very unlikely to use the high inode number bit.
50 for overlay filesystem objects is not only uni !! 50
51 the lifetime of the filesystem. The "-o xino= <<
52 enables the "xino" feature only if the persist <<
53 <<
54 The following table summarizes what can be exp <<
55 configurations. <<
56 <<
57 Inode properties <<
58 ```````````````` <<
59 <<
60 +--------------+------------+------------+---- <<
61 |Configuration | Persistent | Uniform | st_ <<
62 | | st_ino | st_dev | <<
63 +==============+=====+======+=====+======+==== <<
64 | | dir | !dir | dir | !dir | di <<
65 +--------------+-----+------+-----+------+---- <<
66 | All layers | Y | Y | Y | Y | Y <<
67 | on same fs | | | | | <<
68 +--------------+-----+------+-----+------+---- <<
69 | Layers not | N | N | Y | N | N <<
70 | on same fs, | | | | | <<
71 | xino=off | | | | | <<
72 +--------------+-----+------+-----+------+---- <<
73 | xino=on/auto | Y | Y | Y | Y | Y <<
74 +--------------+-----+------+-----+------+---- <<
75 | xino=on/auto,| N | N | Y | N | N <<
76 | ino overflow | | | | | <<
77 +--------------+-----+------+-----+------+---- <<
78 <<
79 [*] nfsd v3 readdirplus verifies d_ino == i_in <<
80 /proc files, such as /proc/locks and /proc/sel <<
81 file descriptor. <<
82 51
83 Upper and Lower 52 Upper and Lower
84 --------------- 53 ---------------
85 54
86 An overlay filesystem combines two filesystems 55 An overlay filesystem combines two filesystems - an 'upper' filesystem
87 and a 'lower' filesystem. When a name exists 56 and a 'lower' filesystem. When a name exists in both filesystems, the
88 object in the 'upper' filesystem is visible wh 57 object in the 'upper' filesystem is visible while the object in the
89 'lower' filesystem is either hidden or, in the 58 'lower' filesystem is either hidden or, in the case of directories,
90 merged with the 'upper' object. 59 merged with the 'upper' object.
91 60
92 It would be more correct to refer to an upper 61 It would be more correct to refer to an upper and lower 'directory
93 tree' rather than 'filesystem' as it is quite 62 tree' rather than 'filesystem' as it is quite possible for both
94 directory trees to be in the same filesystem a 63 directory trees to be in the same filesystem and there is no
95 requirement that the root of a filesystem be g 64 requirement that the root of a filesystem be given for either upper or
96 lower. 65 lower.
97 66
98 A wide range of filesystems supported by Linux !! 67 The lower filesystem can be any filesystem supported by Linux and does
99 but not all filesystems that are mountable by !! 68 not need to be writable. The lower filesystem can even be another
100 needed for OverlayFS to work. The lower files !! 69 overlayfs. The upper filesystem will normally be writable and if it
101 writable. The lower filesystem can even be an !! 70 is it must support the creation of trusted.* extended attributes, and
102 filesystem will normally be writable and if it !! 71 must provide valid d_type in readdir responses, so NFS is not suitable.
103 creation of trusted.* and/or user.* extended a <<
104 valid d_type in readdir responses, so NFS is n <<
105 72
106 A read-only overlay of two read-only filesyste 73 A read-only overlay of two read-only filesystems may use any
107 filesystem type. 74 filesystem type.
108 75
109 Directories 76 Directories
110 ----------- 77 -----------
111 78
112 Overlaying mainly involves directories. If a 79 Overlaying mainly involves directories. If a given name appears in both
113 upper and lower filesystems and refers to a no 80 upper and lower filesystems and refers to a non-directory in either,
114 then the lower object is hidden - the name ref 81 then the lower object is hidden - the name refers only to the upper
115 object. 82 object.
116 83
117 Where both upper and lower objects are directo 84 Where both upper and lower objects are directories, a merged directory
118 is formed. 85 is formed.
119 86
120 At mount time, the two directories given as mo 87 At mount time, the two directories given as mount options "lowerdir" and
121 "upperdir" are combined into a merged director !! 88 "upperdir" are combined into a merged directory:
122 89
123 mount -t overlay overlay -olowerdir=/lower,u 90 mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\
124 workdir=/work /merged 91 workdir=/work /merged
125 92
126 The "workdir" needs to be an empty directory o 93 The "workdir" needs to be an empty directory on the same filesystem
127 as upperdir. 94 as upperdir.
128 95
129 Then whenever a lookup is requested in such a 96 Then whenever a lookup is requested in such a merged directory, the
130 lookup is performed in each actual directory a 97 lookup is performed in each actual directory and the combined result
131 is cached in the dentry belonging to the overl 98 is cached in the dentry belonging to the overlay filesystem. If both
132 actual lookups find directories, both are stor 99 actual lookups find directories, both are stored and a merged
133 directory is created, otherwise only one is st 100 directory is created, otherwise only one is stored: the upper if it
134 exists, else the lower. 101 exists, else the lower.
135 102
136 Only the lists of names from directories are m 103 Only the lists of names from directories are merged. Other content
137 such as metadata and extended attributes are r 104 such as metadata and extended attributes are reported for the upper
138 directory only. These attributes of the lower 105 directory only. These attributes of the lower directory are hidden.
139 106
140 whiteouts and opaque directories 107 whiteouts and opaque directories
141 -------------------------------- 108 --------------------------------
142 109
143 In order to support rm and rmdir without chang 110 In order to support rm and rmdir without changing the lower
144 filesystem, an overlay filesystem needs to rec 111 filesystem, an overlay filesystem needs to record in the upper filesystem
145 that files have been removed. This is done us 112 that files have been removed. This is done using whiteouts and opaque
146 directories (non-directories are always opaque 113 directories (non-directories are always opaque).
147 114
148 A whiteout is created as a character device wi !! 115 A whiteout is created as a character device with 0/0 device number.
149 as a zero-size regular file with the xattr "tr <<
150 <<
151 When a whiteout is found in the upper level of 116 When a whiteout is found in the upper level of a merged directory, any
152 matching name in the lower level is ignored, a 117 matching name in the lower level is ignored, and the whiteout itself
153 is also hidden. 118 is also hidden.
154 119
155 A directory is made opaque by setting the xatt 120 A directory is made opaque by setting the xattr "trusted.overlay.opaque"
156 to "y". Where the upper filesystem contains a 121 to "y". Where the upper filesystem contains an opaque directory, any
157 directory in the lower filesystem with the sam 122 directory in the lower filesystem with the same name is ignored.
158 123
159 An opaque directory should not conntain any wh <<
160 serve any purpose. A merge directory containi <<
161 "trusted.overlay.whiteout", should be addition <<
162 "trusted.overlay.opaque" to "x" on the merge d <<
163 This is needed to avoid the overhead of checki <<
164 on all entries during readdir in the common ca <<
165 <<
166 readdir 124 readdir
167 ------- 125 -------
168 126
169 When a 'readdir' request is made on a merged d 127 When a 'readdir' request is made on a merged directory, the upper and
170 lower directories are each read and the name l 128 lower directories are each read and the name lists merged in the
171 obvious way (upper is read first, then lower - 129 obvious way (upper is read first, then lower - entries that already
172 exist are not re-added). This merged name lis 130 exist are not re-added). This merged name list is cached in the
173 'struct file' and so remains as long as the fi 131 'struct file' and so remains as long as the file is kept open. If the
174 directory is opened and read by two processes 132 directory is opened and read by two processes at the same time, they
175 will each have separate caches. A seekdir to 133 will each have separate caches. A seekdir to the start of the
176 directory (offset 0) followed by a readdir wil 134 directory (offset 0) followed by a readdir will cause the cache to be
177 discarded and rebuilt. 135 discarded and rebuilt.
178 136
179 This means that changes to the merged director 137 This means that changes to the merged directory do not appear while a
180 directory is being read. This is unlikely to 138 directory is being read. This is unlikely to be noticed by many
181 programs. 139 programs.
182 140
183 seek offsets are assigned sequentially when th 141 seek offsets are assigned sequentially when the directories are read.
184 Thus if: !! 142 Thus if
185 143
186 - read part of a directory !! 144 - read part of a directory
187 - remember an offset, and close the directory !! 145 - remember an offset, and close the directory
188 - re-open the directory some time later !! 146 - re-open the directory some time later
189 - seek to the remembered offset !! 147 - seek to the remembered offset
190 148
191 there may be little correlation between the ol 149 there may be little correlation between the old and new locations in
192 the list of filenames, particularly if anythin 150 the list of filenames, particularly if anything has changed in the
193 directory. 151 directory.
194 152
195 Readdir on directories that are not merged is 153 Readdir on directories that are not merged is simply handled by the
196 underlying directory (upper or lower). 154 underlying directory (upper or lower).
197 155
198 renaming directories 156 renaming directories
199 -------------------- 157 --------------------
200 158
201 When renaming a directory that is on the lower 159 When renaming a directory that is on the lower layer or merged (i.e. the
202 directory was not created on the upper layer t 160 directory was not created on the upper layer to start with) overlayfs can
203 handle it in two different ways: 161 handle it in two different ways:
204 162
205 1. return EXDEV error: this error is returned 163 1. return EXDEV error: this error is returned by rename(2) when trying to
206 move a file or directory across filesystem 164 move a file or directory across filesystem boundaries. Hence
207 applications are usually prepared to handle !! 165 applications are usually prepared to hande this error (mv(1) for example
208 recursively copies the directory tree). Th 166 recursively copies the directory tree). This is the default behavior.
209 167
210 2. If the "redirect_dir" feature is enabled, t 168 2. If the "redirect_dir" feature is enabled, then the directory will be
211 copied up (but not the contents). Then the 169 copied up (but not the contents). Then the "trusted.overlay.redirect"
212 extended attribute is set to the path of th 170 extended attribute is set to the path of the original location from the
213 root of the overlay. Finally the directory 171 root of the overlay. Finally the directory is moved to the new
214 location. 172 location.
215 173
216 There are several ways to tune the "redirect_d 174 There are several ways to tune the "redirect_dir" feature.
217 175
218 Kernel config options: 176 Kernel config options:
219 177
220 - OVERLAY_FS_REDIRECT_DIR: 178 - OVERLAY_FS_REDIRECT_DIR:
221 If this is enabled, then redirect_dir is t 179 If this is enabled, then redirect_dir is turned on by default.
222 - OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW: 180 - OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW:
223 If this is enabled, then redirects are alw 181 If this is enabled, then redirects are always followed by default. Enabling
224 this results in a less secure configuratio 182 this results in a less secure configuration. Enable this option only when
225 worried about backward compatibility with 183 worried about backward compatibility with kernels that have the redirect_dir
226 feature and follow redirects even if turne 184 feature and follow redirects even if turned off.
227 185
228 Module options (can also be changed through /s 186 Module options (can also be changed through /sys/module/overlay/parameters/):
229 187
230 - "redirect_dir=BOOL": 188 - "redirect_dir=BOOL":
231 See OVERLAY_FS_REDIRECT_DIR kernel config 189 See OVERLAY_FS_REDIRECT_DIR kernel config option above.
232 - "redirect_always_follow=BOOL": 190 - "redirect_always_follow=BOOL":
233 See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kern 191 See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kernel config option above.
234 - "redirect_max=NUM": 192 - "redirect_max=NUM":
235 The maximum number of bytes in an absolute 193 The maximum number of bytes in an absolute redirect (default is 256).
236 194
237 Mount options: 195 Mount options:
238 196
239 - "redirect_dir=on": 197 - "redirect_dir=on":
240 Redirects are enabled. 198 Redirects are enabled.
241 - "redirect_dir=follow": 199 - "redirect_dir=follow":
242 Redirects are not created, but followed. 200 Redirects are not created, but followed.
243 - "redirect_dir=nofollow": <<
244 Redirects are not created and not followed <<
245 - "redirect_dir=off": 201 - "redirect_dir=off":
246 If "redirect_always_follow" is enabled in !! 202 Redirects are not created and only followed if "redirect_always_follow"
247 this "off" translates to "follow", otherwi !! 203 feature is enabled in the kernel/module config.
>> 204 - "redirect_dir=nofollow":
>> 205 Redirects are not created and not followed (equivalent to "redirect_dir=off"
>> 206 if "redirect_always_follow" feature is not enabled).
248 207
249 When the NFS export feature is enabled, every 208 When the NFS export feature is enabled, every copied up directory is
250 indexed by the file handle of the lower inode 209 indexed by the file handle of the lower inode and a file handle of the
251 upper directory is stored in a "trusted.overla 210 upper directory is stored in a "trusted.overlay.upper" extended attribute
252 on the index entry. On lookup of a merged dir 211 on the index entry. On lookup of a merged directory, if the upper
253 directory does not match the file handle store 212 directory does not match the file handle stores in the index, that is an
254 indication that multiple upper directories may 213 indication that multiple upper directories may be redirected to the same
255 lower directory. In that case, lookup returns 214 lower directory. In that case, lookup returns an error and warns about
256 a possible inconsistency. 215 a possible inconsistency.
257 216
258 Because lower layer redirects cannot be verifi 217 Because lower layer redirects cannot be verified with the index, enabling
259 NFS export support on an overlay filesystem wi 218 NFS export support on an overlay filesystem with no upper layer requires
260 turning off redirect follow (e.g. "redirect_di 219 turning off redirect follow (e.g. "redirect_dir=nofollow").
261 220
262 221
263 Non-directories 222 Non-directories
264 --------------- 223 ---------------
265 224
266 Objects that are not directories (files, symli 225 Objects that are not directories (files, symlinks, device-special
267 files etc.) are presented either from the uppe 226 files etc.) are presented either from the upper or lower filesystem as
268 appropriate. When a file in the lower filesys 227 appropriate. When a file in the lower filesystem is accessed in a way
269 the requires write-access, such as opening for 228 the requires write-access, such as opening for write access, changing
270 some metadata etc., the file is first copied f 229 some metadata etc., the file is first copied from the lower filesystem
271 to the upper filesystem (copy_up). Note that 230 to the upper filesystem (copy_up). Note that creating a hard-link
272 also requires copy_up, though of course creati 231 also requires copy_up, though of course creation of a symlink does
273 not. 232 not.
274 233
275 The copy_up may turn out to be unnecessary, fo 234 The copy_up may turn out to be unnecessary, for example if the file is
276 opened for read-write but the data is not modi 235 opened for read-write but the data is not modified.
277 236
278 The copy_up process first makes sure that the 237 The copy_up process first makes sure that the containing directory
279 exists in the upper filesystem - creating it a 238 exists in the upper filesystem - creating it and any parents as
280 necessary. It then creates the object with th 239 necessary. It then creates the object with the same metadata (owner,
281 mode, mtime, symlink-target etc.) and then if 240 mode, mtime, symlink-target etc.) and then if the object is a file, the
282 data is copied from the lower to the upper fil 241 data is copied from the lower to the upper filesystem. Finally any
283 extended attributes are copied up. 242 extended attributes are copied up.
284 243
285 Once the copy_up is complete, the overlay file 244 Once the copy_up is complete, the overlay filesystem simply
286 provides direct access to the newly created fi 245 provides direct access to the newly created file in the upper
287 filesystem - future operations on the file are 246 filesystem - future operations on the file are barely noticed by the
288 overlay filesystem (though an operation on the 247 overlay filesystem (though an operation on the name of the file such as
289 rename or unlink will of course be noticed and 248 rename or unlink will of course be noticed and handled).
290 249
291 250
292 Permission model <<
293 ---------------- <<
294 <<
295 Permission checking in the overlay filesystem <<
296 <<
297 1) permission check SHOULD return the same re <<
298 <<
299 2) task creating the overlay mount MUST NOT g <<
300 <<
301 3) non-mounting task MAY gain additional priv <<
302 compared to direct access on underlying lo <<
303 <<
304 This is achieved by performing two permission <<
305 <<
306 a) check if current task is allowed access ba <<
307 group, mode and posix acl), as well as MAC <<
308 <<
309 b) check if mounting task would be allowed re <<
310 upper layer based on underlying filesystem <<
311 MAC checks <<
312 <<
313 Check (a) ensures consistency (1) since owner, <<
314 are copied up. On the other hand it can resul <<
315 permissions (used by NFS, for example) being i <<
316 <<
317 Check (b) ensures that no task gains permissio <<
318 the mounting task does not have (2). This als <<
319 to create setups where the consistency rule (1 <<
320 however, the mounting task will have sufficien <<
321 operations. <<
322 <<
323 Another way to demonstrate this model is drawi <<
324 <<
325 mount -t overlay overlay -olowerdir=/lower,u <<
326 <<
327 and:: <<
328 <<
329 cp -a /lower /upper <<
330 mount --bind /upper /merged <<
331 <<
332 The resulting access permissions should be the <<
333 the time of copy (on-demand vs. up-front). <<
334 <<
335 <<
336 Multiple lower layers 251 Multiple lower layers
337 --------------------- 252 ---------------------
338 253
339 Multiple lower layers can now be given using t !! 254 Multiple lower layers can now be given using the the colon (":") as a
340 separator character between the directory name !! 255 separator character between the directory names. For example:
341 256
342 mount -t overlay overlay -olowerdir=/lower1: 257 mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged
343 258
344 As the example shows, "upperdir=" and "workdir 259 As the example shows, "upperdir=" and "workdir=" may be omitted. In
345 that case the overlay will be read-only. 260 that case the overlay will be read-only.
346 261
347 The specified lower directories will be stacke 262 The specified lower directories will be stacked beginning from the
348 rightmost one and going left. In the above ex 263 rightmost one and going left. In the above example lower1 will be the
349 top, lower2 the middle and lower3 the bottom l 264 top, lower2 the middle and lower3 the bottom layer.
350 265
351 Note: directory names containing colons can be <<
352 escaping the colons with a single backslash. <<
353 <<
354 mount -t overlay overlay -olowerdir=/a\:lowe <<
355 <<
356 Since kernel version v6.8, directory names con <<
357 be configured as lower layer using the "lowerd <<
358 fsconfig syscall from new mount api. For exam <<
359 <<
360 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerd <<
361 <<
362 In the latter case, colons in lower layer dire <<
363 as an octal characters (\072) when displayed i <<
364 266
365 Metadata only copy up 267 Metadata only copy up
366 --------------------- 268 ---------------------
367 269
368 When the "metacopy" feature is enabled, overla !! 270 When metadata only copy up feature is enabled, overlayfs will only copy
369 up metadata (as opposed to whole file), when a 271 up metadata (as opposed to whole file), when a metadata specific operation
370 like chown/chmod is performed. Full file will 272 like chown/chmod is performed. Full file will be copied up later when
371 file is opened for WRITE operation. 273 file is opened for WRITE operation.
372 274
373 In other words, this is delayed data copy up o 275 In other words, this is delayed data copy up operation and data is copied
374 up when there is a need to actually modify dat 276 up when there is a need to actually modify data.
375 277
376 There are multiple ways to enable/disable this 278 There are multiple ways to enable/disable this feature. A config option
377 CONFIG_OVERLAY_FS_METACOPY can be set/unset to 279 CONFIG_OVERLAY_FS_METACOPY can be set/unset to enable/disable this feature
378 by default. Or one can enable/disable it at mo 280 by default. Or one can enable/disable it at module load time with module
379 parameter metacopy=on/off. Lastly, there is al 281 parameter metacopy=on/off. Lastly, there is also a per mount option
380 metacopy=on/off to enable/disable this feature 282 metacopy=on/off to enable/disable this feature per mount.
381 283
382 Do not use metacopy=on with untrusted upper/lo 284 Do not use metacopy=on with untrusted upper/lower directories. Otherwise
383 it is possible that an attacker can create a h 285 it is possible that an attacker can create a handcrafted file with
384 appropriate REDIRECT and METACOPY xattrs, and 286 appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower
385 pointed by REDIRECT. This should not be possib 287 pointed by REDIRECT. This should not be possible on local system as setting
386 "trusted." xattrs will require CAP_SYS_ADMIN. 288 "trusted." xattrs will require CAP_SYS_ADMIN. But it should be possible
387 for untrusted layers like from a pen drive. 289 for untrusted layers like from a pen drive.
388 290
389 Note: redirect_dir={off|nofollow|follow[*]} an !! 291 Note: redirect_dir={off|nofollow|follow[*]} conflicts with metacopy=on, and
390 conflict with metacopy=on, and will result in !! 292 results in an error.
391 293
392 [*] redirect_dir=follow only conflicts with me 294 [*] redirect_dir=follow only conflicts with metacopy=on if upperdir=... is
393 given. 295 given.
394 296
395 <<
396 Data-only lower layers <<
397 ---------------------- <<
398 <<
399 With "metacopy" feature enabled, an overlayfs <<
400 of information from up to three different laye <<
401 <<
402 1) metadata from a file in the upper layer <<
403 <<
404 2) st_ino and st_dev object identifier from a <<
405 <<
406 3) data from a file in another lower layer (f <<
407 <<
408 The "lower data" file can be on any lower laye <<
409 lower layer. <<
410 <<
411 Below the top most lower layer, any number of <<
412 as "data-only" lower layers, using double colo <<
413 A normal lower layer is not allowed to be belo <<
414 colon separators are not allowed to the right <<
415 <<
416 <<
417 For example:: <<
418 <<
419 mount -t overlay overlay -olowerdir=/l1:/l2: <<
420 <<
421 The paths of files in the "data-only" lower la <<
422 merged overlayfs directories and the metadata <<
423 in the "data-only" lower layers are not visibl <<
424 <<
425 Only the data of the files in the "data-only" <<
426 when a "metacopy" file in one of the lower lay <<
427 to the absolute path of the "lower data" file <<
428 <<
429 Since kernel version v6.8, "data-only" lower l <<
430 the "datadir+" mount options and the fsconfig <<
431 For example:: <<
432 <<
433 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerd <<
434 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerd <<
435 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerd <<
436 fsconfig(fs_fd, FSCONFIG_SET_STRING, "datadi <<
437 fsconfig(fs_fd, FSCONFIG_SET_STRING, "datadi <<
438 <<
439 <<
440 fs-verity support <<
441 ----------------- <<
442 <<
443 During metadata copy up of a lower file, if th <<
444 fs-verity enabled and overlay verity support i <<
445 digest of the lower file is added to the "trus <<
446 xattr. This is then used to verify the content <<
447 each the time the metacopy file is opened. <<
448 <<
449 When a layer containing verity xattrs is used, <<
450 metacopy file in the upper layer is guaranteed <<
451 that was in the lower at the time of the copy- <<
452 (during a mount, after a remount, etc) such a <<
453 replaced or modified in any way, access to the <<
454 overlayfs will result in EIO errors (either on <<
455 digest check, or from a later read due to fs-v <<
456 error is printed to the kernel logs. For more <<
457 file access works, see :ref:`Documentation/fil <<
458 <accessing_verity_files>`. <<
459 <<
460 Verity can be used as a general robustness che <<
461 changes in the overlayfs directories in use. B <<
462 it can also give more powerful guarantees. For <<
463 layer is fully trusted (by using dm-verity or <<
464 an untrusted lower layer can be used to supply <<
465 for all metacopy files. If additionally the u <<
466 directories are specified as "Data-only", then <<
467 such file content, and the entire mount can be <<
468 upper layer. <<
469 <<
470 This feature is controlled by the "verity" mou <<
471 supports these values: <<
472 <<
473 - "off": <<
474 The metacopy digest is never generated or <<
475 default if verity option is not specified. <<
476 - "on": <<
477 Whenever a metacopy files specifies an exp <<
478 corresponding data file must match the spe <<
479 generating a metacopy file the verity dige <<
480 based on the source file (if it has one). <<
481 - "require": <<
482 Same as "on", but additionally all metacop <<
483 digest (or EIO is returned on open). This <<
484 will only be used if the data file has fs- <<
485 otherwise a full copy-up is used. <<
486 <<
487 Sharing and copying layers 297 Sharing and copying layers
488 -------------------------- 298 --------------------------
489 299
490 Lower layers may be shared among several overl 300 Lower layers may be shared among several overlay mounts and that is indeed
491 a very common practice. An overlay mount may 301 a very common practice. An overlay mount may use the same lower layer
492 path as another overlay mount and it may use a 302 path as another overlay mount and it may use a lower layer path that is
493 beneath or above the path of another overlay l 303 beneath or above the path of another overlay lower layer path.
494 304
495 Using an upper layer path and/or a workdir pat 305 Using an upper layer path and/or a workdir path that are already used by
496 another overlay mount is not allowed and may f 306 another overlay mount is not allowed and may fail with EBUSY. Using
497 partially overlapping paths is not allowed and 307 partially overlapping paths is not allowed and may fail with EBUSY.
498 If files are accessed from two overlayfs mount 308 If files are accessed from two overlayfs mounts which share or overlap the
499 upper layer and/or workdir path the behavior o 309 upper layer and/or workdir path the behavior of the overlay is undefined,
500 though it will not result in a crash or deadlo 310 though it will not result in a crash or deadlock.
501 311
502 Mounting an overlay using an upper layer path, 312 Mounting an overlay using an upper layer path, where the upper layer path
503 was previously used by another mounted overlay 313 was previously used by another mounted overlay in combination with a
504 different lower layer path, is allowed, unless !! 314 different lower layer path, is allowed, unless the "inodes index" feature
505 features are enabled. !! 315 or "metadata only copy up" feature is enabled.
506 316
507 With the "index" feature, on the first time mo !! 317 With the "inodes index" feature, on the first time mount, an NFS file
508 handle of the lower layer root directory, alon 318 handle of the lower layer root directory, along with the UUID of the lower
509 filesystem, are encoded and stored in the "tru 319 filesystem, are encoded and stored in the "trusted.overlay.origin" extended
510 attribute on the upper layer root directory. 320 attribute on the upper layer root directory. On subsequent mount attempts,
511 the lower root directory file handle and lower 321 the lower root directory file handle and lower filesystem UUID are compared
512 to the stored origin in upper root directory. 322 to the stored origin in upper root directory. On failure to verify the
513 lower root origin, mount will fail with ESTALE 323 lower root origin, mount will fail with ESTALE. An overlayfs mount with
514 "index" enabled will fail with EOPNOTSUPP if t !! 324 "inodes index" enabled will fail with EOPNOTSUPP if the lower filesystem
515 does not support NFS export, lower filesystem 325 does not support NFS export, lower filesystem does not have a valid UUID or
516 if the upper filesystem does not support exten 326 if the upper filesystem does not support extended attributes.
517 327
518 For the "metacopy" feature, there is no verifi !! 328 For "metadata only copy up" feature there is no verification mechanism at
519 mount time. So if same upper is mounted with d 329 mount time. So if same upper is mounted with different set of lower, mount
520 probably will succeed but expect the unexpecte 330 probably will succeed but expect the unexpected later on. So don't do it.
521 331
522 It is quite a common practice to copy overlay 332 It is quite a common practice to copy overlay layers to a different
523 directory tree on the same or different underl 333 directory tree on the same or different underlying filesystem, and even
524 to a different machine. With the "index" feat !! 334 to a different machine. With the "inodes index" feature, trying to mount
525 the copied layers will fail the verification o 335 the copied layers will fail the verification of the lower root file handle.
526 336
527 Nesting overlayfs mounts <<
528 ------------------------ <<
529 <<
530 It is possible to use a lower directory that i <<
531 mount. For regular files this does not need an <<
532 that have overlayfs attributes, such as whiteo <<
533 interpreted by the underlying overlayfs mount <<
534 allow the second overlayfs mount to see the at <<
535 <<
536 Overlayfs specific xattrs are escaped by using <<
537 "overlay.overlay.". So, a file with a "trusted <<
538 in the lower dir will be exposed as a regular <<
539 "trusted.overlay.metacopy" xattr in the overla <<
540 repeating the prefix multiple time, as each in <<
541 <<
542 A lower dir with a regular whiteout will alway <<
543 mount, so to support storing an effective whit <<
544 alternative form of whiteout is supported. Thi <<
545 file with the "overlay.whiteout" xattr set, in <<
546 "overlay.opaque" xattr set to "x" (see `whiteo <<
547 These alternative whiteouts are never created <<
548 userspace tools (like containers) that generat <<
549 These alternative whiteouts can be escaped usi <<
550 mechanism in order to properly nest to any dep <<
551 337
552 Non-standard behavior 338 Non-standard behavior
553 --------------------- 339 ---------------------
554 340
555 Current version of overlayfs can act as a most 341 Current version of overlayfs can act as a mostly POSIX compliant
556 filesystem. 342 filesystem.
557 343
558 This is the list of cases that overlayfs doesn 344 This is the list of cases that overlayfs doesn't currently handle:
559 345
560 a) POSIX mandates updating st_atime for reads !! 346 a) POSIX mandates updating st_atime for reads. This is currently not
561 done in the case when the file resides on !! 347 done in the case when the file resides on a lower layer.
562 <<
563 b) If a file residing on a lower layer is ope <<
564 memory mapped with MAP_SHARED, then subseq <<
565 reflected in the memory mapping. <<
566 348
567 c) If a file residing on a lower layer is bei !! 349 b) If a file residing on a lower layer is opened for read-only and then
568 file for write or truncating the file will !! 350 memory mapped with MAP_SHARED, then subsequent changes to the file are not
>> 351 reflected in the memory mapping.
569 352
570 The following options allow overlayfs to act m 353 The following options allow overlayfs to act more like a standards
571 compliant filesystem: 354 compliant filesystem:
572 355
573 redirect_dir !! 356 1) "redirect_dir"
574 ```````````` <<
575 357
576 Enabled with the mount option or module option 358 Enabled with the mount option or module option: "redirect_dir=on" or with
577 the kernel config option CONFIG_OVERLAY_FS_RED 359 the kernel config option CONFIG_OVERLAY_FS_REDIRECT_DIR=y.
578 360
579 If this feature is disabled, then rename(2) on 361 If this feature is disabled, then rename(2) on a lower or merged directory
580 will fail with EXDEV ("Invalid cross-device li 362 will fail with EXDEV ("Invalid cross-device link").
581 363
582 index !! 364 2) "inode index"
583 ````` <<
584 365
585 Enabled with the mount option or module option 366 Enabled with the mount option or module option "index=on" or with the
586 kernel config option CONFIG_OVERLAY_FS_INDEX=y 367 kernel config option CONFIG_OVERLAY_FS_INDEX=y.
587 368
588 If this feature is disabled and a file with mu 369 If this feature is disabled and a file with multiple hard links is copied
589 up, then this will "break" the link. Changes 370 up, then this will "break" the link. Changes will not be propagated to
590 other names referring to the same inode. 371 other names referring to the same inode.
591 372
592 xino !! 373 3) "xino"
593 ```` <<
594 374
595 Enabled with the mount option "xino=auto" or " 375 Enabled with the mount option "xino=auto" or "xino=on", with the module
596 option "xino_auto=on" or with the kernel confi 376 option "xino_auto=on" or with the kernel config option
597 CONFIG_OVERLAY_FS_XINO_AUTO=y. Also implicitl 377 CONFIG_OVERLAY_FS_XINO_AUTO=y. Also implicitly enabled by using the same
598 underlying filesystem for all layers making up 378 underlying filesystem for all layers making up the overlay.
599 379
600 If this feature is disabled or the underlying 380 If this feature is disabled or the underlying filesystem doesn't have
601 enough free bits in the inode number, then ove 381 enough free bits in the inode number, then overlayfs will not be able to
602 guarantee that the values of st_ino and st_dev 382 guarantee that the values of st_ino and st_dev returned by stat(2) and the
603 value of d_ino returned by readdir(3) will act 383 value of d_ino returned by readdir(3) will act like on a normal filesystem.
604 E.g. the value of st_dev may be different for 384 E.g. the value of st_dev may be different for two objects in the same
605 overlay filesystem and the value of st_ino for !! 385 overlay filesystem and the value of st_ino for directory objects may not be
606 persistent and could change even while the ove !! 386 persistent and could change even while the overlay filesystem is mounted.
607 summarized in the `Inode properties`_ table ab <<
608 387
609 388
610 Changes to underlying filesystems 389 Changes to underlying filesystems
611 --------------------------------- 390 ---------------------------------
612 391
>> 392 Offline changes, when the overlay is not mounted, are allowed to either
>> 393 the upper or the lower trees.
>> 394
613 Changes to the underlying filesystems while pa 395 Changes to the underlying filesystems while part of a mounted overlay
614 filesystem are not allowed. If the underlying 396 filesystem are not allowed. If the underlying filesystem is changed,
615 the behavior of the overlay is undefined, thou 397 the behavior of the overlay is undefined, though it will not result in
616 a crash or deadlock. 398 a crash or deadlock.
617 399
618 Offline changes, when the overlay is not mount <<
619 upper tree. Offline changes to the lower tree <<
620 "metacopy", "index", "xino" and "redirect_dir" <<
621 have not been used. If the lower tree is modi <<
622 features has been used, the behavior of the ov <<
623 though it will not result in a crash or deadlo <<
624 <<
625 When the overlay NFS export feature is enabled 400 When the overlay NFS export feature is enabled, overlay filesystems
626 behavior on offline changes of the underlying 401 behavior on offline changes of the underlying lower layer is different
627 than the behavior when NFS export is disabled. 402 than the behavior when NFS export is disabled.
628 403
629 On every copy_up, an NFS file handle of the lo 404 On every copy_up, an NFS file handle of the lower inode, along with the
630 UUID of the lower filesystem, are encoded and 405 UUID of the lower filesystem, are encoded and stored in an extended
631 attribute "trusted.overlay.origin" on the uppe 406 attribute "trusted.overlay.origin" on the upper inode.
632 407
633 When the NFS export feature is enabled, a look 408 When the NFS export feature is enabled, a lookup of a merged directory,
634 that found a lower directory at the lookup pat 409 that found a lower directory at the lookup path or at the path pointed
635 to by the "trusted.overlay.redirect" extended 410 to by the "trusted.overlay.redirect" extended attribute, will verify
636 that the found lower directory file handle and 411 that the found lower directory file handle and lower filesystem UUID
637 match the origin file handle that was stored a 412 match the origin file handle that was stored at copy_up time. If a
638 found lower directory does not match the store 413 found lower directory does not match the stored origin, that directory
639 will not be merged with the upper directory. 414 will not be merged with the upper directory.
640 415
641 416
642 417
643 NFS export 418 NFS export
644 ---------- 419 ----------
645 420
646 When the underlying filesystems supports NFS e 421 When the underlying filesystems supports NFS export and the "nfs_export"
647 feature is enabled, an overlay filesystem may 422 feature is enabled, an overlay filesystem may be exported to NFS.
648 423
649 With the "nfs_export" feature, on copy_up of a 424 With the "nfs_export" feature, on copy_up of any lower object, an index
650 entry is created under the index directory. T 425 entry is created under the index directory. The index entry name is the
651 hexadecimal representation of the copy up orig 426 hexadecimal representation of the copy up origin file handle. For a
652 non-directory object, the index entry is a har 427 non-directory object, the index entry is a hard link to the upper inode.
653 For a directory object, the index entry has an 428 For a directory object, the index entry has an extended attribute
654 "trusted.overlay.upper" with an encoded file h 429 "trusted.overlay.upper" with an encoded file handle of the upper
655 directory inode. 430 directory inode.
656 431
657 When encoding a file handle from an overlay fi 432 When encoding a file handle from an overlay filesystem object, the
658 following rules apply: 433 following rules apply:
659 434
660 1. For a non-upper object, encode a lower fil !! 435 1. For a non-upper object, encode a lower file handle from lower inode
661 2. For an indexed object, encode a lower file !! 436 2. For an indexed object, encode a lower file handle from copy_up origin
662 3. For a pure-upper object and for an existin !! 437 3. For a pure-upper object and for an existing non-indexed upper object,
663 encode an upper file handle from upper ino !! 438 encode an upper file handle from upper inode
664 439
665 The encoded overlay file handle includes: 440 The encoded overlay file handle includes:
666 <<
667 - Header including path type information (e.g 441 - Header including path type information (e.g. lower/upper)
668 - UUID of the underlying filesystem 442 - UUID of the underlying filesystem
669 - Underlying filesystem encoding of underlyin 443 - Underlying filesystem encoding of underlying inode
670 444
671 This encoding format is identical to the encod 445 This encoding format is identical to the encoding format file handles that
672 are stored in extended attribute "trusted.over 446 are stored in extended attribute "trusted.overlay.origin".
673 447
674 When decoding an overlay file handle, the foll 448 When decoding an overlay file handle, the following steps are followed:
675 449
676 1. Find underlying layer by UUID and path typ !! 450 1. Find underlying layer by UUID and path type information.
677 2. Decode the underlying filesystem file hand !! 451 2. Decode the underlying filesystem file handle to underlying dentry.
678 3. For a lower file handle, lookup the handle !! 452 3. For a lower file handle, lookup the handle in index directory by name.
679 4. If a whiteout is found in index, return ES !! 453 4. If a whiteout is found in index, return ESTALE. This represents an
680 overlay object that was deleted after its !! 454 overlay object that was deleted after its file handle was encoded.
681 5. For a non-directory, instantiate a disconn !! 455 5. For a non-directory, instantiate a disconnected overlay dentry from the
682 decoded underlying dentry, the path type a !! 456 decoded underlying dentry, the path type and index inode, if found.
683 6. For a directory, use the connected underly !! 457 6. For a directory, use the connected underlying decoded dentry, path type
684 and index, to lookup a connected overlay d !! 458 and index, to lookup a connected overlay dentry.
685 459
686 Decoding a non-directory file handle may retur 460 Decoding a non-directory file handle may return a disconnected dentry.
687 copy_up of that disconnected dentry will creat 461 copy_up of that disconnected dentry will create an upper index entry with
688 no upper alias. 462 no upper alias.
689 463
690 When overlay filesystem has multiple lower lay 464 When overlay filesystem has multiple lower layers, a middle layer
691 directory may have a "redirect" to lower direc 465 directory may have a "redirect" to lower directory. Because middle layer
692 "redirects" are not indexed, a lower file hand 466 "redirects" are not indexed, a lower file handle that was encoded from the
693 "redirect" origin directory, cannot be used to 467 "redirect" origin directory, cannot be used to find the middle or upper
694 layer directory. Similarly, a lower file hand 468 layer directory. Similarly, a lower file handle that was encoded from a
695 descendant of the "redirect" origin directory, 469 descendant of the "redirect" origin directory, cannot be used to
696 reconstruct a connected overlay path. To miti 470 reconstruct a connected overlay path. To mitigate the cases of
697 directories that cannot be decoded from a lowe 471 directories that cannot be decoded from a lower file handle, these
698 directories are copied up on encode and encode 472 directories are copied up on encode and encoded as an upper file handle.
699 On an overlay filesystem with no upper layer t 473 On an overlay filesystem with no upper layer this mitigation cannot be
700 used NFS export in this setup requires turning 474 used NFS export in this setup requires turning off redirect follow (e.g.
701 "redirect_dir=nofollow"). 475 "redirect_dir=nofollow").
702 476
703 The overlay filesystem does not support non-di 477 The overlay filesystem does not support non-directory connectable file
704 handles, so exporting with the 'subtree_check' 478 handles, so exporting with the 'subtree_check' exportfs configuration will
705 cause failures to lookup files over NFS. 479 cause failures to lookup files over NFS.
706 480
707 When the NFS export feature is enabled, all di 481 When the NFS export feature is enabled, all directory index entries are
708 verified on mount time to check that upper fil 482 verified on mount time to check that upper file handles are not stale.
709 This verification may cause significant overhe 483 This verification may cause significant overhead in some cases.
710 484
711 Note: the mount options index=off,nfs_export=o <<
712 read-write mount and will result in an error. <<
713 <<
714 Note: the mount option uuid=off can be used to <<
715 filesystem in file handles with null, and effe <<
716 can be useful in case the underlying disk is c <<
717 is changed. This is only applicable if all low <<
718 the same filesystem, otherwise it will fallbac <<
719 <<
720 <<
721 UUID and fsid <<
722 ------------- <<
723 <<
724 The UUID of overlayfs instance itself and the <<
725 controlled by the "uuid" mount option, which s <<
726 <<
727 - "null": <<
728 UUID of overlayfs is null. fsid is taken f <<
729 - "off": <<
730 UUID of overlayfs is null. fsid is taken f <<
731 UUID of underlying layers is ignored. <<
732 - "on": <<
733 UUID of overlayfs is generated and used to <<
734 UUID is stored in xattr "trusted.overlay.u <<
735 unique and persistent. This option requir <<
736 filesystem that supports xattrs. <<
737 - "auto": (default) <<
738 UUID is taken from xattr "trusted.overlay. <<
739 Upgrade to "uuid=on" on first time mount o <<
740 meets the prerequites. <<
741 Downgrade to "uuid=null" for existing over <<
742 mounted with "uuid=on". <<
743 <<
744 <<
745 Volatile mount <<
746 -------------- <<
747 <<
748 This is enabled with the "volatile" mount opti <<
749 guaranteed to survive a crash. It is strongly <<
750 mounts are only used if data written to the ov <<
751 without significant effort. <<
752 <<
753 The advantage of mounting with the "volatile" <<
754 sync calls to the upper filesystem are omitted <<
755 <<
756 In order to avoid a giving a false sense of sa <<
757 semantics of volatile mounts are slightly diff <<
758 VFS. If any writeback error occurs on the upp <<
759 volatile mount takes place, all sync functions <<
760 condition is reached, the filesystem will not <<
761 call will return an error, even if the upperdi <<
762 since the last sync call. <<
763 <<
764 When overlay is mounted with "volatile" option <<
765 "$workdir/work/incompat/volatile" is created. <<
766 checks for this directory and refuses to mount <<
767 indicator that user should throw away upper an <<
768 fresh one. In very limited cases where the use <<
769 not crashed and contents of upperdir are intac <<
770 can be removed. <<
771 <<
772 <<
773 User xattr <<
774 ---------- <<
775 <<
776 The "-o userxattr" mount option forces overlay <<
777 "user.overlay." xattr namespace instead of "tr <<
778 useful for unprivileged mounting of overlayfs. <<
779 <<
780 485
781 Testsuite 486 Testsuite
782 --------- 487 ---------
783 488
784 There's a testsuite originally developed by Da 489 There's a testsuite originally developed by David Howells and currently
785 maintained by Amir Goldstein at: 490 maintained by Amir Goldstein at:
786 491
787 https://github.com/amir73il/unionmount-testsui !! 492 https://github.com/amir73il/unionmount-testsuite.git
788 493
789 Run as root:: !! 494 Run as root:
790 495
791 # cd unionmount-testsuite 496 # cd unionmount-testsuite
792 # ./run --ov --verify 497 # ./run --ov --verify
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.