1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0 2 2 3 Written by: Neil Brown 3 Written by: Neil Brown 4 Please see MAINTAINERS file for where to send 4 Please see MAINTAINERS file for where to send questions. 5 5 6 Overlay Filesystem 6 Overlay Filesystem 7 ================== 7 ================== 8 8 9 This document describes a prototype for a new 9 This document describes a prototype for a new approach to providing 10 overlay-filesystem functionality in Linux (som 10 overlay-filesystem functionality in Linux (sometimes referred to as 11 union-filesystems). An overlay-filesystem tri 11 union-filesystems). An overlay-filesystem tries to present a 12 filesystem which is the result over overlaying 12 filesystem which is the result over overlaying one filesystem on top 13 of the other. 13 of the other. 14 14 15 15 16 Overlay objects 16 Overlay objects 17 --------------- 17 --------------- 18 18 19 The overlay filesystem approach is 'hybrid', b 19 The overlay filesystem approach is 'hybrid', because the objects that 20 appear in the filesystem do not always appear 20 appear in the filesystem do not always appear to belong to that filesystem. 21 In many cases, an object accessed in the union 21 In many cases, an object accessed in the union will be indistinguishable 22 from accessing the corresponding object from t 22 from accessing the corresponding object from the original filesystem. 23 This is most obvious from the 'st_dev' field r 23 This is most obvious from the 'st_dev' field returned by stat(2). 24 24 25 While directories will report an st_dev from t 25 While directories will report an st_dev from the overlay-filesystem, 26 non-directory objects may report an st_dev fro 26 non-directory objects may report an st_dev from the lower filesystem or 27 upper filesystem that is providing the object. 27 upper filesystem that is providing the object. Similarly st_ino will 28 only be unique when combined with st_dev, and 28 only be unique when combined with st_dev, and both of these can change 29 over the lifetime of a non-directory object. 29 over the lifetime of a non-directory object. Many applications and 30 tools ignore these values and will not be affe 30 tools ignore these values and will not be affected. 31 31 32 In the special case of all overlay layers on t 32 In the special case of all overlay layers on the same underlying 33 filesystem, all objects will report an st_dev 33 filesystem, all objects will report an st_dev from the overlay 34 filesystem and st_ino from the underlying file 34 filesystem and st_ino from the underlying filesystem. This will 35 make the overlay mount more compliant with fil 35 make the overlay mount more compliant with filesystem scanners and 36 overlay objects will be distinguishable from t 36 overlay objects will be distinguishable from the corresponding 37 objects in the original filesystem. 37 objects in the original filesystem. 38 38 39 On 64bit systems, even if all overlay layers a 39 On 64bit systems, even if all overlay layers are not on the same 40 underlying filesystem, the same compliant beha 40 underlying filesystem, the same compliant behavior could be achieved 41 with the "xino" feature. The "xino" feature c 41 with the "xino" feature. The "xino" feature composes a unique object 42 identifier from the real object st_ino and an !! 42 identifier from the real object st_ino and an underlying fsid index. 43 The "xino" feature uses the high inode number !! 43 44 underlying filesystems rarely use the high ino !! 44 If all underlying filesystems support NFS file handles and export file >> 45 handles with 32bit inode number encoding (e.g. ext4), overlay filesystem >> 46 will use the high inode number bits for fsid. Even when the underlying >> 47 filesystem uses 64bit inode numbers, users can still enable the "xino" >> 48 feature with the "-o xino=on" overlay mount option. That is useful for the >> 49 case of underlying filesystems like xfs and tmpfs, which use 64bit inode >> 50 numbers, but are very unlikely to use the high inode number bits. In case 45 the underlying inode number does overflow into 51 the underlying inode number does overflow into the high xino bits, overlay 46 filesystem will fall back to the non xino beha 52 filesystem will fall back to the non xino behavior for that inode. 47 53 48 The "xino" feature can be enabled with the "-o << 49 If all underlying filesystems support NFS file << 50 for overlay filesystem objects is not only uni << 51 the lifetime of the filesystem. The "-o xino= << 52 enables the "xino" feature only if the persist << 53 << 54 The following table summarizes what can be exp 54 The following table summarizes what can be expected in different overlay 55 configurations. 55 configurations. 56 56 57 Inode properties 57 Inode properties 58 ```````````````` 58 ```````````````` 59 59 60 +--------------+------------+------------+---- 60 +--------------+------------+------------+-----------------+----------------+ 61 |Configuration | Persistent | Uniform | st_ 61 |Configuration | Persistent | Uniform | st_ino == d_ino | d_ino == i_ino | 62 | | st_ino | st_dev | 62 | | st_ino | st_dev | | [*] | 63 +==============+=====+======+=====+======+==== 63 +==============+=====+======+=====+======+========+========+========+=======+ 64 | | dir | !dir | dir | !dir | di 64 | | dir | !dir | dir | !dir | dir + !dir | dir | !dir | 65 +--------------+-----+------+-----+------+---- 65 +--------------+-----+------+-----+------+--------+--------+--------+-------+ 66 | All layers | Y | Y | Y | Y | Y 66 | All layers | Y | Y | Y | Y | Y | Y | Y | Y | 67 | on same fs | | | | | 67 | on same fs | | | | | | | | | 68 +--------------+-----+------+-----+------+---- 68 +--------------+-----+------+-----+------+--------+--------+--------+-------+ 69 | Layers not | N | N | Y | N | N !! 69 | Layers not | N | Y | Y | N | N | Y | N | Y | 70 | on same fs, | | | | | 70 | on same fs, | | | | | | | | | 71 | xino=off | | | | | 71 | xino=off | | | | | | | | | 72 +--------------+-----+------+-----+------+---- 72 +--------------+-----+------+-----+------+--------+--------+--------+-------+ 73 | xino=on/auto | Y | Y | Y | Y | Y 73 | xino=on/auto | Y | Y | Y | Y | Y | Y | Y | Y | >> 74 | | | | | | | | | | 74 +--------------+-----+------+-----+------+---- 75 +--------------+-----+------+-----+------+--------+--------+--------+-------+ 75 | xino=on/auto,| N | N | Y | N | N !! 76 | xino=on/auto,| N | Y | Y | N | N | Y | N | Y | 76 | ino overflow | | | | | 77 | ino overflow | | | | | | | | | 77 +--------------+-----+------+-----+------+---- 78 +--------------+-----+------+-----+------+--------+--------+--------+-------+ 78 79 79 [*] nfsd v3 readdirplus verifies d_ino == i_in 80 [*] nfsd v3 readdirplus verifies d_ino == i_ino. i_ino is exposed via several 80 /proc files, such as /proc/locks and /proc/sel 81 /proc files, such as /proc/locks and /proc/self/fdinfo/<fd> of an inotify 81 file descriptor. 82 file descriptor. 82 83 >> 84 83 Upper and Lower 85 Upper and Lower 84 --------------- 86 --------------- 85 87 86 An overlay filesystem combines two filesystems 88 An overlay filesystem combines two filesystems - an 'upper' filesystem 87 and a 'lower' filesystem. When a name exists 89 and a 'lower' filesystem. When a name exists in both filesystems, the 88 object in the 'upper' filesystem is visible wh 90 object in the 'upper' filesystem is visible while the object in the 89 'lower' filesystem is either hidden or, in the 91 'lower' filesystem is either hidden or, in the case of directories, 90 merged with the 'upper' object. 92 merged with the 'upper' object. 91 93 92 It would be more correct to refer to an upper 94 It would be more correct to refer to an upper and lower 'directory 93 tree' rather than 'filesystem' as it is quite 95 tree' rather than 'filesystem' as it is quite possible for both 94 directory trees to be in the same filesystem a 96 directory trees to be in the same filesystem and there is no 95 requirement that the root of a filesystem be g 97 requirement that the root of a filesystem be given for either upper or 96 lower. 98 lower. 97 99 98 A wide range of filesystems supported by Linux !! 100 The lower filesystem can be any filesystem supported by Linux and does 99 but not all filesystems that are mountable by !! 101 not need to be writable. The lower filesystem can even be another 100 needed for OverlayFS to work. The lower files !! 102 overlayfs. The upper filesystem will normally be writable and if it 101 writable. The lower filesystem can even be an !! 103 is it must support the creation of trusted.* extended attributes, and 102 filesystem will normally be writable and if it !! 104 must provide valid d_type in readdir responses, so NFS is not suitable. 103 creation of trusted.* and/or user.* extended a << 104 valid d_type in readdir responses, so NFS is n << 105 105 106 A read-only overlay of two read-only filesyste 106 A read-only overlay of two read-only filesystems may use any 107 filesystem type. 107 filesystem type. 108 108 109 Directories 109 Directories 110 ----------- 110 ----------- 111 111 112 Overlaying mainly involves directories. If a 112 Overlaying mainly involves directories. If a given name appears in both 113 upper and lower filesystems and refers to a no 113 upper and lower filesystems and refers to a non-directory in either, 114 then the lower object is hidden - the name ref 114 then the lower object is hidden - the name refers only to the upper 115 object. 115 object. 116 116 117 Where both upper and lower objects are directo 117 Where both upper and lower objects are directories, a merged directory 118 is formed. 118 is formed. 119 119 120 At mount time, the two directories given as mo 120 At mount time, the two directories given as mount options "lowerdir" and 121 "upperdir" are combined into a merged director !! 121 "upperdir" are combined into a merged directory: 122 122 123 mount -t overlay overlay -olowerdir=/lower,u 123 mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\ 124 workdir=/work /merged 124 workdir=/work /merged 125 125 126 The "workdir" needs to be an empty directory o 126 The "workdir" needs to be an empty directory on the same filesystem 127 as upperdir. 127 as upperdir. 128 128 129 Then whenever a lookup is requested in such a 129 Then whenever a lookup is requested in such a merged directory, the 130 lookup is performed in each actual directory a 130 lookup is performed in each actual directory and the combined result 131 is cached in the dentry belonging to the overl 131 is cached in the dentry belonging to the overlay filesystem. If both 132 actual lookups find directories, both are stor 132 actual lookups find directories, both are stored and a merged 133 directory is created, otherwise only one is st 133 directory is created, otherwise only one is stored: the upper if it 134 exists, else the lower. 134 exists, else the lower. 135 135 136 Only the lists of names from directories are m 136 Only the lists of names from directories are merged. Other content 137 such as metadata and extended attributes are r 137 such as metadata and extended attributes are reported for the upper 138 directory only. These attributes of the lower 138 directory only. These attributes of the lower directory are hidden. 139 139 140 whiteouts and opaque directories 140 whiteouts and opaque directories 141 -------------------------------- 141 -------------------------------- 142 142 143 In order to support rm and rmdir without chang 143 In order to support rm and rmdir without changing the lower 144 filesystem, an overlay filesystem needs to rec 144 filesystem, an overlay filesystem needs to record in the upper filesystem 145 that files have been removed. This is done us 145 that files have been removed. This is done using whiteouts and opaque 146 directories (non-directories are always opaque 146 directories (non-directories are always opaque). 147 147 148 A whiteout is created as a character device wi !! 148 A whiteout is created as a character device with 0/0 device number. 149 as a zero-size regular file with the xattr "tr << 150 << 151 When a whiteout is found in the upper level of 149 When a whiteout is found in the upper level of a merged directory, any 152 matching name in the lower level is ignored, a 150 matching name in the lower level is ignored, and the whiteout itself 153 is also hidden. 151 is also hidden. 154 152 155 A directory is made opaque by setting the xatt 153 A directory is made opaque by setting the xattr "trusted.overlay.opaque" 156 to "y". Where the upper filesystem contains a 154 to "y". Where the upper filesystem contains an opaque directory, any 157 directory in the lower filesystem with the sam 155 directory in the lower filesystem with the same name is ignored. 158 156 159 An opaque directory should not conntain any wh << 160 serve any purpose. A merge directory containi << 161 "trusted.overlay.whiteout", should be addition << 162 "trusted.overlay.opaque" to "x" on the merge d << 163 This is needed to avoid the overhead of checki << 164 on all entries during readdir in the common ca << 165 << 166 readdir 157 readdir 167 ------- 158 ------- 168 159 169 When a 'readdir' request is made on a merged d 160 When a 'readdir' request is made on a merged directory, the upper and 170 lower directories are each read and the name l 161 lower directories are each read and the name lists merged in the 171 obvious way (upper is read first, then lower - 162 obvious way (upper is read first, then lower - entries that already 172 exist are not re-added). This merged name lis 163 exist are not re-added). This merged name list is cached in the 173 'struct file' and so remains as long as the fi 164 'struct file' and so remains as long as the file is kept open. If the 174 directory is opened and read by two processes 165 directory is opened and read by two processes at the same time, they 175 will each have separate caches. A seekdir to 166 will each have separate caches. A seekdir to the start of the 176 directory (offset 0) followed by a readdir wil 167 directory (offset 0) followed by a readdir will cause the cache to be 177 discarded and rebuilt. 168 discarded and rebuilt. 178 169 179 This means that changes to the merged director 170 This means that changes to the merged directory do not appear while a 180 directory is being read. This is unlikely to 171 directory is being read. This is unlikely to be noticed by many 181 programs. 172 programs. 182 173 183 seek offsets are assigned sequentially when th 174 seek offsets are assigned sequentially when the directories are read. 184 Thus if: !! 175 Thus if 185 176 186 - read part of a directory !! 177 - read part of a directory 187 - remember an offset, and close the directory !! 178 - remember an offset, and close the directory 188 - re-open the directory some time later !! 179 - re-open the directory some time later 189 - seek to the remembered offset !! 180 - seek to the remembered offset 190 181 191 there may be little correlation between the ol 182 there may be little correlation between the old and new locations in 192 the list of filenames, particularly if anythin 183 the list of filenames, particularly if anything has changed in the 193 directory. 184 directory. 194 185 195 Readdir on directories that are not merged is 186 Readdir on directories that are not merged is simply handled by the 196 underlying directory (upper or lower). 187 underlying directory (upper or lower). 197 188 198 renaming directories 189 renaming directories 199 -------------------- 190 -------------------- 200 191 201 When renaming a directory that is on the lower 192 When renaming a directory that is on the lower layer or merged (i.e. the 202 directory was not created on the upper layer t 193 directory was not created on the upper layer to start with) overlayfs can 203 handle it in two different ways: 194 handle it in two different ways: 204 195 205 1. return EXDEV error: this error is returned 196 1. return EXDEV error: this error is returned by rename(2) when trying to 206 move a file or directory across filesystem 197 move a file or directory across filesystem boundaries. Hence 207 applications are usually prepared to handle !! 198 applications are usually prepared to hande this error (mv(1) for example 208 recursively copies the directory tree). Th 199 recursively copies the directory tree). This is the default behavior. 209 200 210 2. If the "redirect_dir" feature is enabled, t 201 2. If the "redirect_dir" feature is enabled, then the directory will be 211 copied up (but not the contents). Then the 202 copied up (but not the contents). Then the "trusted.overlay.redirect" 212 extended attribute is set to the path of th 203 extended attribute is set to the path of the original location from the 213 root of the overlay. Finally the directory 204 root of the overlay. Finally the directory is moved to the new 214 location. 205 location. 215 206 216 There are several ways to tune the "redirect_d 207 There are several ways to tune the "redirect_dir" feature. 217 208 218 Kernel config options: 209 Kernel config options: 219 210 220 - OVERLAY_FS_REDIRECT_DIR: 211 - OVERLAY_FS_REDIRECT_DIR: 221 If this is enabled, then redirect_dir is t 212 If this is enabled, then redirect_dir is turned on by default. 222 - OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW: 213 - OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW: 223 If this is enabled, then redirects are alw 214 If this is enabled, then redirects are always followed by default. Enabling 224 this results in a less secure configuratio 215 this results in a less secure configuration. Enable this option only when 225 worried about backward compatibility with 216 worried about backward compatibility with kernels that have the redirect_dir 226 feature and follow redirects even if turne 217 feature and follow redirects even if turned off. 227 218 228 Module options (can also be changed through /s 219 Module options (can also be changed through /sys/module/overlay/parameters/): 229 220 230 - "redirect_dir=BOOL": 221 - "redirect_dir=BOOL": 231 See OVERLAY_FS_REDIRECT_DIR kernel config 222 See OVERLAY_FS_REDIRECT_DIR kernel config option above. 232 - "redirect_always_follow=BOOL": 223 - "redirect_always_follow=BOOL": 233 See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kern 224 See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kernel config option above. 234 - "redirect_max=NUM": 225 - "redirect_max=NUM": 235 The maximum number of bytes in an absolute 226 The maximum number of bytes in an absolute redirect (default is 256). 236 227 237 Mount options: 228 Mount options: 238 229 239 - "redirect_dir=on": 230 - "redirect_dir=on": 240 Redirects are enabled. 231 Redirects are enabled. 241 - "redirect_dir=follow": 232 - "redirect_dir=follow": 242 Redirects are not created, but followed. 233 Redirects are not created, but followed. 243 - "redirect_dir=nofollow": << 244 Redirects are not created and not followed << 245 - "redirect_dir=off": 234 - "redirect_dir=off": 246 If "redirect_always_follow" is enabled in !! 235 Redirects are not created and only followed if "redirect_always_follow" 247 this "off" translates to "follow", otherwi !! 236 feature is enabled in the kernel/module config. >> 237 - "redirect_dir=nofollow": >> 238 Redirects are not created and not followed (equivalent to "redirect_dir=off" >> 239 if "redirect_always_follow" feature is not enabled). 248 240 249 When the NFS export feature is enabled, every 241 When the NFS export feature is enabled, every copied up directory is 250 indexed by the file handle of the lower inode 242 indexed by the file handle of the lower inode and a file handle of the 251 upper directory is stored in a "trusted.overla 243 upper directory is stored in a "trusted.overlay.upper" extended attribute 252 on the index entry. On lookup of a merged dir 244 on the index entry. On lookup of a merged directory, if the upper 253 directory does not match the file handle store 245 directory does not match the file handle stores in the index, that is an 254 indication that multiple upper directories may 246 indication that multiple upper directories may be redirected to the same 255 lower directory. In that case, lookup returns 247 lower directory. In that case, lookup returns an error and warns about 256 a possible inconsistency. 248 a possible inconsistency. 257 249 258 Because lower layer redirects cannot be verifi 250 Because lower layer redirects cannot be verified with the index, enabling 259 NFS export support on an overlay filesystem wi 251 NFS export support on an overlay filesystem with no upper layer requires 260 turning off redirect follow (e.g. "redirect_di 252 turning off redirect follow (e.g. "redirect_dir=nofollow"). 261 253 262 254 263 Non-directories 255 Non-directories 264 --------------- 256 --------------- 265 257 266 Objects that are not directories (files, symli 258 Objects that are not directories (files, symlinks, device-special 267 files etc.) are presented either from the uppe 259 files etc.) are presented either from the upper or lower filesystem as 268 appropriate. When a file in the lower filesys 260 appropriate. When a file in the lower filesystem is accessed in a way 269 the requires write-access, such as opening for 261 the requires write-access, such as opening for write access, changing 270 some metadata etc., the file is first copied f 262 some metadata etc., the file is first copied from the lower filesystem 271 to the upper filesystem (copy_up). Note that 263 to the upper filesystem (copy_up). Note that creating a hard-link 272 also requires copy_up, though of course creati 264 also requires copy_up, though of course creation of a symlink does 273 not. 265 not. 274 266 275 The copy_up may turn out to be unnecessary, fo 267 The copy_up may turn out to be unnecessary, for example if the file is 276 opened for read-write but the data is not modi 268 opened for read-write but the data is not modified. 277 269 278 The copy_up process first makes sure that the 270 The copy_up process first makes sure that the containing directory 279 exists in the upper filesystem - creating it a 271 exists in the upper filesystem - creating it and any parents as 280 necessary. It then creates the object with th 272 necessary. It then creates the object with the same metadata (owner, 281 mode, mtime, symlink-target etc.) and then if 273 mode, mtime, symlink-target etc.) and then if the object is a file, the 282 data is copied from the lower to the upper fil 274 data is copied from the lower to the upper filesystem. Finally any 283 extended attributes are copied up. 275 extended attributes are copied up. 284 276 285 Once the copy_up is complete, the overlay file 277 Once the copy_up is complete, the overlay filesystem simply 286 provides direct access to the newly created fi 278 provides direct access to the newly created file in the upper 287 filesystem - future operations on the file are 279 filesystem - future operations on the file are barely noticed by the 288 overlay filesystem (though an operation on the 280 overlay filesystem (though an operation on the name of the file such as 289 rename or unlink will of course be noticed and 281 rename or unlink will of course be noticed and handled). 290 282 291 283 292 Permission model 284 Permission model 293 ---------------- 285 ---------------- 294 286 295 Permission checking in the overlay filesystem 287 Permission checking in the overlay filesystem follows these principles: 296 288 297 1) permission check SHOULD return the same re 289 1) permission check SHOULD return the same result before and after copy up 298 290 299 2) task creating the overlay mount MUST NOT g 291 2) task creating the overlay mount MUST NOT gain additional privileges 300 292 301 3) non-mounting task MAY gain additional priv 293 3) non-mounting task MAY gain additional privileges through the overlay, 302 compared to direct access on underlying lo !! 294 compared to direct access on underlying lower or upper filesystems 303 295 304 This is achieved by performing two permission !! 296 This is achieved by performing two permission checks on each access 305 297 306 a) check if current task is allowed access ba 298 a) check if current task is allowed access based on local DAC (owner, 307 group, mode and posix acl), as well as MAC 299 group, mode and posix acl), as well as MAC checks 308 300 309 b) check if mounting task would be allowed re 301 b) check if mounting task would be allowed real operation on lower or 310 upper layer based on underlying filesystem 302 upper layer based on underlying filesystem permissions, again including 311 MAC checks 303 MAC checks 312 304 313 Check (a) ensures consistency (1) since owner, 305 Check (a) ensures consistency (1) since owner, group, mode and posix acls 314 are copied up. On the other hand it can resul 306 are copied up. On the other hand it can result in server enforced 315 permissions (used by NFS, for example) being i 307 permissions (used by NFS, for example) being ignored (3). 316 308 317 Check (b) ensures that no task gains permissio 309 Check (b) ensures that no task gains permissions to underlying layers that 318 the mounting task does not have (2). This als 310 the mounting task does not have (2). This also means that it is possible 319 to create setups where the consistency rule (1 311 to create setups where the consistency rule (1) does not hold; normally, 320 however, the mounting task will have sufficien 312 however, the mounting task will have sufficient privileges to perform all 321 operations. 313 operations. 322 314 323 Another way to demonstrate this model is drawi !! 315 Another way to demonstrate this model is drawing parallels between 324 316 325 mount -t overlay overlay -olowerdir=/lower,u 317 mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,... /merged 326 318 327 and:: !! 319 and 328 320 329 cp -a /lower /upper 321 cp -a /lower /upper 330 mount --bind /upper /merged 322 mount --bind /upper /merged 331 323 332 The resulting access permissions should be the 324 The resulting access permissions should be the same. The difference is in 333 the time of copy (on-demand vs. up-front). 325 the time of copy (on-demand vs. up-front). 334 326 335 327 336 Multiple lower layers 328 Multiple lower layers 337 --------------------- 329 --------------------- 338 330 339 Multiple lower layers can now be given using t !! 331 Multiple lower layers can now be given using the the colon (":") as a 340 separator character between the directory name !! 332 separator character between the directory names. For example: 341 333 342 mount -t overlay overlay -olowerdir=/lower1: 334 mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged 343 335 344 As the example shows, "upperdir=" and "workdir 336 As the example shows, "upperdir=" and "workdir=" may be omitted. In 345 that case the overlay will be read-only. 337 that case the overlay will be read-only. 346 338 347 The specified lower directories will be stacke 339 The specified lower directories will be stacked beginning from the 348 rightmost one and going left. In the above ex 340 rightmost one and going left. In the above example lower1 will be the 349 top, lower2 the middle and lower3 the bottom l 341 top, lower2 the middle and lower3 the bottom layer. 350 342 351 Note: directory names containing colons can be << 352 escaping the colons with a single backslash. << 353 << 354 mount -t overlay overlay -olowerdir=/a\:lowe << 355 << 356 Since kernel version v6.8, directory names con << 357 be configured as lower layer using the "lowerd << 358 fsconfig syscall from new mount api. For exam << 359 << 360 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerd << 361 << 362 In the latter case, colons in lower layer dire << 363 as an octal characters (\072) when displayed i << 364 343 365 Metadata only copy up 344 Metadata only copy up 366 --------------------- 345 --------------------- 367 346 368 When the "metacopy" feature is enabled, overla !! 347 When metadata only copy up feature is enabled, overlayfs will only copy 369 up metadata (as opposed to whole file), when a 348 up metadata (as opposed to whole file), when a metadata specific operation 370 like chown/chmod is performed. Full file will 349 like chown/chmod is performed. Full file will be copied up later when 371 file is opened for WRITE operation. 350 file is opened for WRITE operation. 372 351 373 In other words, this is delayed data copy up o 352 In other words, this is delayed data copy up operation and data is copied 374 up when there is a need to actually modify dat 353 up when there is a need to actually modify data. 375 354 376 There are multiple ways to enable/disable this 355 There are multiple ways to enable/disable this feature. A config option 377 CONFIG_OVERLAY_FS_METACOPY can be set/unset to 356 CONFIG_OVERLAY_FS_METACOPY can be set/unset to enable/disable this feature 378 by default. Or one can enable/disable it at mo 357 by default. Or one can enable/disable it at module load time with module 379 parameter metacopy=on/off. Lastly, there is al 358 parameter metacopy=on/off. Lastly, there is also a per mount option 380 metacopy=on/off to enable/disable this feature 359 metacopy=on/off to enable/disable this feature per mount. 381 360 382 Do not use metacopy=on with untrusted upper/lo 361 Do not use metacopy=on with untrusted upper/lower directories. Otherwise 383 it is possible that an attacker can create a h 362 it is possible that an attacker can create a handcrafted file with 384 appropriate REDIRECT and METACOPY xattrs, and 363 appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower 385 pointed by REDIRECT. This should not be possib 364 pointed by REDIRECT. This should not be possible on local system as setting 386 "trusted." xattrs will require CAP_SYS_ADMIN. 365 "trusted." xattrs will require CAP_SYS_ADMIN. But it should be possible 387 for untrusted layers like from a pen drive. 366 for untrusted layers like from a pen drive. 388 367 389 Note: redirect_dir={off|nofollow|follow[*]} an 368 Note: redirect_dir={off|nofollow|follow[*]} and nfs_export=on mount options 390 conflict with metacopy=on, and will result in 369 conflict with metacopy=on, and will result in an error. 391 370 392 [*] redirect_dir=follow only conflicts with me 371 [*] redirect_dir=follow only conflicts with metacopy=on if upperdir=... is 393 given. 372 given. 394 373 395 << 396 Data-only lower layers << 397 ---------------------- << 398 << 399 With "metacopy" feature enabled, an overlayfs << 400 of information from up to three different laye << 401 << 402 1) metadata from a file in the upper layer << 403 << 404 2) st_ino and st_dev object identifier from a << 405 << 406 3) data from a file in another lower layer (f << 407 << 408 The "lower data" file can be on any lower laye << 409 lower layer. << 410 << 411 Below the top most lower layer, any number of << 412 as "data-only" lower layers, using double colo << 413 A normal lower layer is not allowed to be belo << 414 colon separators are not allowed to the right << 415 << 416 << 417 For example:: << 418 << 419 mount -t overlay overlay -olowerdir=/l1:/l2: << 420 << 421 The paths of files in the "data-only" lower la << 422 merged overlayfs directories and the metadata << 423 in the "data-only" lower layers are not visibl << 424 << 425 Only the data of the files in the "data-only" << 426 when a "metacopy" file in one of the lower lay << 427 to the absolute path of the "lower data" file << 428 << 429 Since kernel version v6.8, "data-only" lower l << 430 the "datadir+" mount options and the fsconfig << 431 For example:: << 432 << 433 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerd << 434 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerd << 435 fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerd << 436 fsconfig(fs_fd, FSCONFIG_SET_STRING, "datadi << 437 fsconfig(fs_fd, FSCONFIG_SET_STRING, "datadi << 438 << 439 << 440 fs-verity support << 441 ----------------- << 442 << 443 During metadata copy up of a lower file, if th << 444 fs-verity enabled and overlay verity support i << 445 digest of the lower file is added to the "trus << 446 xattr. This is then used to verify the content << 447 each the time the metacopy file is opened. << 448 << 449 When a layer containing verity xattrs is used, << 450 metacopy file in the upper layer is guaranteed << 451 that was in the lower at the time of the copy- << 452 (during a mount, after a remount, etc) such a << 453 replaced or modified in any way, access to the << 454 overlayfs will result in EIO errors (either on << 455 digest check, or from a later read due to fs-v << 456 error is printed to the kernel logs. For more << 457 file access works, see :ref:`Documentation/fil << 458 <accessing_verity_files>`. << 459 << 460 Verity can be used as a general robustness che << 461 changes in the overlayfs directories in use. B << 462 it can also give more powerful guarantees. For << 463 layer is fully trusted (by using dm-verity or << 464 an untrusted lower layer can be used to supply << 465 for all metacopy files. If additionally the u << 466 directories are specified as "Data-only", then << 467 such file content, and the entire mount can be << 468 upper layer. << 469 << 470 This feature is controlled by the "verity" mou << 471 supports these values: << 472 << 473 - "off": << 474 The metacopy digest is never generated or << 475 default if verity option is not specified. << 476 - "on": << 477 Whenever a metacopy files specifies an exp << 478 corresponding data file must match the spe << 479 generating a metacopy file the verity dige << 480 based on the source file (if it has one). << 481 - "require": << 482 Same as "on", but additionally all metacop << 483 digest (or EIO is returned on open). This << 484 will only be used if the data file has fs- << 485 otherwise a full copy-up is used. << 486 << 487 Sharing and copying layers 374 Sharing and copying layers 488 -------------------------- 375 -------------------------- 489 376 490 Lower layers may be shared among several overl 377 Lower layers may be shared among several overlay mounts and that is indeed 491 a very common practice. An overlay mount may 378 a very common practice. An overlay mount may use the same lower layer 492 path as another overlay mount and it may use a 379 path as another overlay mount and it may use a lower layer path that is 493 beneath or above the path of another overlay l 380 beneath or above the path of another overlay lower layer path. 494 381 495 Using an upper layer path and/or a workdir pat 382 Using an upper layer path and/or a workdir path that are already used by 496 another overlay mount is not allowed and may f 383 another overlay mount is not allowed and may fail with EBUSY. Using 497 partially overlapping paths is not allowed and 384 partially overlapping paths is not allowed and may fail with EBUSY. 498 If files are accessed from two overlayfs mount 385 If files are accessed from two overlayfs mounts which share or overlap the 499 upper layer and/or workdir path the behavior o 386 upper layer and/or workdir path the behavior of the overlay is undefined, 500 though it will not result in a crash or deadlo 387 though it will not result in a crash or deadlock. 501 388 502 Mounting an overlay using an upper layer path, 389 Mounting an overlay using an upper layer path, where the upper layer path 503 was previously used by another mounted overlay 390 was previously used by another mounted overlay in combination with a 504 different lower layer path, is allowed, unless !! 391 different lower layer path, is allowed, unless the "inodes index" feature 505 features are enabled. !! 392 or "metadata only copy up" feature is enabled. 506 393 507 With the "index" feature, on the first time mo !! 394 With the "inodes index" feature, on the first time mount, an NFS file 508 handle of the lower layer root directory, alon 395 handle of the lower layer root directory, along with the UUID of the lower 509 filesystem, are encoded and stored in the "tru 396 filesystem, are encoded and stored in the "trusted.overlay.origin" extended 510 attribute on the upper layer root directory. 397 attribute on the upper layer root directory. On subsequent mount attempts, 511 the lower root directory file handle and lower 398 the lower root directory file handle and lower filesystem UUID are compared 512 to the stored origin in upper root directory. 399 to the stored origin in upper root directory. On failure to verify the 513 lower root origin, mount will fail with ESTALE 400 lower root origin, mount will fail with ESTALE. An overlayfs mount with 514 "index" enabled will fail with EOPNOTSUPP if t !! 401 "inodes index" enabled will fail with EOPNOTSUPP if the lower filesystem 515 does not support NFS export, lower filesystem 402 does not support NFS export, lower filesystem does not have a valid UUID or 516 if the upper filesystem does not support exten 403 if the upper filesystem does not support extended attributes. 517 404 518 For the "metacopy" feature, there is no verifi !! 405 For "metadata only copy up" feature there is no verification mechanism at 519 mount time. So if same upper is mounted with d 406 mount time. So if same upper is mounted with different set of lower, mount 520 probably will succeed but expect the unexpecte 407 probably will succeed but expect the unexpected later on. So don't do it. 521 408 522 It is quite a common practice to copy overlay 409 It is quite a common practice to copy overlay layers to a different 523 directory tree on the same or different underl 410 directory tree on the same or different underlying filesystem, and even 524 to a different machine. With the "index" feat !! 411 to a different machine. With the "inodes index" feature, trying to mount 525 the copied layers will fail the verification o 412 the copied layers will fail the verification of the lower root file handle. 526 413 527 Nesting overlayfs mounts << 528 ------------------------ << 529 << 530 It is possible to use a lower directory that i << 531 mount. For regular files this does not need an << 532 that have overlayfs attributes, such as whiteo << 533 interpreted by the underlying overlayfs mount << 534 allow the second overlayfs mount to see the at << 535 << 536 Overlayfs specific xattrs are escaped by using << 537 "overlay.overlay.". So, a file with a "trusted << 538 in the lower dir will be exposed as a regular << 539 "trusted.overlay.metacopy" xattr in the overla << 540 repeating the prefix multiple time, as each in << 541 << 542 A lower dir with a regular whiteout will alway << 543 mount, so to support storing an effective whit << 544 alternative form of whiteout is supported. Thi << 545 file with the "overlay.whiteout" xattr set, in << 546 "overlay.opaque" xattr set to "x" (see `whiteo << 547 These alternative whiteouts are never created << 548 userspace tools (like containers) that generat << 549 These alternative whiteouts can be escaped usi << 550 mechanism in order to properly nest to any dep << 551 414 552 Non-standard behavior 415 Non-standard behavior 553 --------------------- 416 --------------------- 554 417 555 Current version of overlayfs can act as a most 418 Current version of overlayfs can act as a mostly POSIX compliant 556 filesystem. 419 filesystem. 557 420 558 This is the list of cases that overlayfs doesn 421 This is the list of cases that overlayfs doesn't currently handle: 559 422 560 a) POSIX mandates updating st_atime for reads !! 423 a) POSIX mandates updating st_atime for reads. This is currently not 561 done in the case when the file resides on !! 424 done in the case when the file resides on a lower layer. 562 << 563 b) If a file residing on a lower layer is ope << 564 memory mapped with MAP_SHARED, then subseq << 565 reflected in the memory mapping. << 566 425 567 c) If a file residing on a lower layer is bei !! 426 b) If a file residing on a lower layer is opened for read-only and then 568 file for write or truncating the file will !! 427 memory mapped with MAP_SHARED, then subsequent changes to the file are not >> 428 reflected in the memory mapping. 569 429 570 The following options allow overlayfs to act m 430 The following options allow overlayfs to act more like a standards 571 compliant filesystem: 431 compliant filesystem: 572 432 573 redirect_dir !! 433 1) "redirect_dir" 574 ```````````` << 575 434 576 Enabled with the mount option or module option 435 Enabled with the mount option or module option: "redirect_dir=on" or with 577 the kernel config option CONFIG_OVERLAY_FS_RED 436 the kernel config option CONFIG_OVERLAY_FS_REDIRECT_DIR=y. 578 437 579 If this feature is disabled, then rename(2) on 438 If this feature is disabled, then rename(2) on a lower or merged directory 580 will fail with EXDEV ("Invalid cross-device li 439 will fail with EXDEV ("Invalid cross-device link"). 581 440 582 index !! 441 2) "inode index" 583 ````` << 584 442 585 Enabled with the mount option or module option 443 Enabled with the mount option or module option "index=on" or with the 586 kernel config option CONFIG_OVERLAY_FS_INDEX=y 444 kernel config option CONFIG_OVERLAY_FS_INDEX=y. 587 445 588 If this feature is disabled and a file with mu 446 If this feature is disabled and a file with multiple hard links is copied 589 up, then this will "break" the link. Changes 447 up, then this will "break" the link. Changes will not be propagated to 590 other names referring to the same inode. 448 other names referring to the same inode. 591 449 592 xino !! 450 3) "xino" 593 ```` << 594 451 595 Enabled with the mount option "xino=auto" or " 452 Enabled with the mount option "xino=auto" or "xino=on", with the module 596 option "xino_auto=on" or with the kernel confi 453 option "xino_auto=on" or with the kernel config option 597 CONFIG_OVERLAY_FS_XINO_AUTO=y. Also implicitl 454 CONFIG_OVERLAY_FS_XINO_AUTO=y. Also implicitly enabled by using the same 598 underlying filesystem for all layers making up 455 underlying filesystem for all layers making up the overlay. 599 456 600 If this feature is disabled or the underlying 457 If this feature is disabled or the underlying filesystem doesn't have 601 enough free bits in the inode number, then ove 458 enough free bits in the inode number, then overlayfs will not be able to 602 guarantee that the values of st_ino and st_dev 459 guarantee that the values of st_ino and st_dev returned by stat(2) and the 603 value of d_ino returned by readdir(3) will act 460 value of d_ino returned by readdir(3) will act like on a normal filesystem. 604 E.g. the value of st_dev may be different for 461 E.g. the value of st_dev may be different for two objects in the same 605 overlay filesystem and the value of st_ino for !! 462 overlay filesystem and the value of st_ino for directory objects may not be 606 persistent and could change even while the ove 463 persistent and could change even while the overlay filesystem is mounted, as 607 summarized in the `Inode properties`_ table ab 464 summarized in the `Inode properties`_ table above. 608 465 609 466 610 Changes to underlying filesystems 467 Changes to underlying filesystems 611 --------------------------------- 468 --------------------------------- 612 469 >> 470 Offline changes, when the overlay is not mounted, are allowed to either >> 471 the upper or the lower trees. >> 472 613 Changes to the underlying filesystems while pa 473 Changes to the underlying filesystems while part of a mounted overlay 614 filesystem are not allowed. If the underlying 474 filesystem are not allowed. If the underlying filesystem is changed, 615 the behavior of the overlay is undefined, thou 475 the behavior of the overlay is undefined, though it will not result in 616 a crash or deadlock. 476 a crash or deadlock. 617 477 618 Offline changes, when the overlay is not mount << 619 upper tree. Offline changes to the lower tree << 620 "metacopy", "index", "xino" and "redirect_dir" << 621 have not been used. If the lower tree is modi << 622 features has been used, the behavior of the ov << 623 though it will not result in a crash or deadlo << 624 << 625 When the overlay NFS export feature is enabled 478 When the overlay NFS export feature is enabled, overlay filesystems 626 behavior on offline changes of the underlying 479 behavior on offline changes of the underlying lower layer is different 627 than the behavior when NFS export is disabled. 480 than the behavior when NFS export is disabled. 628 481 629 On every copy_up, an NFS file handle of the lo 482 On every copy_up, an NFS file handle of the lower inode, along with the 630 UUID of the lower filesystem, are encoded and 483 UUID of the lower filesystem, are encoded and stored in an extended 631 attribute "trusted.overlay.origin" on the uppe 484 attribute "trusted.overlay.origin" on the upper inode. 632 485 633 When the NFS export feature is enabled, a look 486 When the NFS export feature is enabled, a lookup of a merged directory, 634 that found a lower directory at the lookup pat 487 that found a lower directory at the lookup path or at the path pointed 635 to by the "trusted.overlay.redirect" extended 488 to by the "trusted.overlay.redirect" extended attribute, will verify 636 that the found lower directory file handle and 489 that the found lower directory file handle and lower filesystem UUID 637 match the origin file handle that was stored a 490 match the origin file handle that was stored at copy_up time. If a 638 found lower directory does not match the store 491 found lower directory does not match the stored origin, that directory 639 will not be merged with the upper directory. 492 will not be merged with the upper directory. 640 493 641 494 642 495 643 NFS export 496 NFS export 644 ---------- 497 ---------- 645 498 646 When the underlying filesystems supports NFS e 499 When the underlying filesystems supports NFS export and the "nfs_export" 647 feature is enabled, an overlay filesystem may 500 feature is enabled, an overlay filesystem may be exported to NFS. 648 501 649 With the "nfs_export" feature, on copy_up of a 502 With the "nfs_export" feature, on copy_up of any lower object, an index 650 entry is created under the index directory. T 503 entry is created under the index directory. The index entry name is the 651 hexadecimal representation of the copy up orig 504 hexadecimal representation of the copy up origin file handle. For a 652 non-directory object, the index entry is a har 505 non-directory object, the index entry is a hard link to the upper inode. 653 For a directory object, the index entry has an 506 For a directory object, the index entry has an extended attribute 654 "trusted.overlay.upper" with an encoded file h 507 "trusted.overlay.upper" with an encoded file handle of the upper 655 directory inode. 508 directory inode. 656 509 657 When encoding a file handle from an overlay fi 510 When encoding a file handle from an overlay filesystem object, the 658 following rules apply: 511 following rules apply: 659 512 660 1. For a non-upper object, encode a lower fil !! 513 1. For a non-upper object, encode a lower file handle from lower inode 661 2. For an indexed object, encode a lower file !! 514 2. For an indexed object, encode a lower file handle from copy_up origin 662 3. For a pure-upper object and for an existin !! 515 3. For a pure-upper object and for an existing non-indexed upper object, 663 encode an upper file handle from upper ino !! 516 encode an upper file handle from upper inode 664 517 665 The encoded overlay file handle includes: 518 The encoded overlay file handle includes: 666 << 667 - Header including path type information (e.g 519 - Header including path type information (e.g. lower/upper) 668 - UUID of the underlying filesystem 520 - UUID of the underlying filesystem 669 - Underlying filesystem encoding of underlyin 521 - Underlying filesystem encoding of underlying inode 670 522 671 This encoding format is identical to the encod 523 This encoding format is identical to the encoding format file handles that 672 are stored in extended attribute "trusted.over 524 are stored in extended attribute "trusted.overlay.origin". 673 525 674 When decoding an overlay file handle, the foll 526 When decoding an overlay file handle, the following steps are followed: 675 527 676 1. Find underlying layer by UUID and path typ !! 528 1. Find underlying layer by UUID and path type information. 677 2. Decode the underlying filesystem file hand !! 529 2. Decode the underlying filesystem file handle to underlying dentry. 678 3. For a lower file handle, lookup the handle !! 530 3. For a lower file handle, lookup the handle in index directory by name. 679 4. If a whiteout is found in index, return ES !! 531 4. If a whiteout is found in index, return ESTALE. This represents an 680 overlay object that was deleted after its !! 532 overlay object that was deleted after its file handle was encoded. 681 5. For a non-directory, instantiate a disconn !! 533 5. For a non-directory, instantiate a disconnected overlay dentry from the 682 decoded underlying dentry, the path type a !! 534 decoded underlying dentry, the path type and index inode, if found. 683 6. For a directory, use the connected underly !! 535 6. For a directory, use the connected underlying decoded dentry, path type 684 and index, to lookup a connected overlay d !! 536 and index, to lookup a connected overlay dentry. 685 537 686 Decoding a non-directory file handle may retur 538 Decoding a non-directory file handle may return a disconnected dentry. 687 copy_up of that disconnected dentry will creat 539 copy_up of that disconnected dentry will create an upper index entry with 688 no upper alias. 540 no upper alias. 689 541 690 When overlay filesystem has multiple lower lay 542 When overlay filesystem has multiple lower layers, a middle layer 691 directory may have a "redirect" to lower direc 543 directory may have a "redirect" to lower directory. Because middle layer 692 "redirects" are not indexed, a lower file hand 544 "redirects" are not indexed, a lower file handle that was encoded from the 693 "redirect" origin directory, cannot be used to 545 "redirect" origin directory, cannot be used to find the middle or upper 694 layer directory. Similarly, a lower file hand 546 layer directory. Similarly, a lower file handle that was encoded from a 695 descendant of the "redirect" origin directory, 547 descendant of the "redirect" origin directory, cannot be used to 696 reconstruct a connected overlay path. To miti 548 reconstruct a connected overlay path. To mitigate the cases of 697 directories that cannot be decoded from a lowe 549 directories that cannot be decoded from a lower file handle, these 698 directories are copied up on encode and encode 550 directories are copied up on encode and encoded as an upper file handle. 699 On an overlay filesystem with no upper layer t 551 On an overlay filesystem with no upper layer this mitigation cannot be 700 used NFS export in this setup requires turning 552 used NFS export in this setup requires turning off redirect follow (e.g. 701 "redirect_dir=nofollow"). 553 "redirect_dir=nofollow"). 702 554 703 The overlay filesystem does not support non-di 555 The overlay filesystem does not support non-directory connectable file 704 handles, so exporting with the 'subtree_check' 556 handles, so exporting with the 'subtree_check' exportfs configuration will 705 cause failures to lookup files over NFS. 557 cause failures to lookup files over NFS. 706 558 707 When the NFS export feature is enabled, all di 559 When the NFS export feature is enabled, all directory index entries are 708 verified on mount time to check that upper fil 560 verified on mount time to check that upper file handles are not stale. 709 This verification may cause significant overhe 561 This verification may cause significant overhead in some cases. 710 562 711 Note: the mount options index=off,nfs_export=o 563 Note: the mount options index=off,nfs_export=on are conflicting for a 712 read-write mount and will result in an error. 564 read-write mount and will result in an error. 713 565 714 Note: the mount option uuid=off can be used to << 715 filesystem in file handles with null, and effe << 716 can be useful in case the underlying disk is c << 717 is changed. This is only applicable if all low << 718 the same filesystem, otherwise it will fallbac << 719 << 720 << 721 UUID and fsid << 722 ------------- << 723 << 724 The UUID of overlayfs instance itself and the << 725 controlled by the "uuid" mount option, which s << 726 << 727 - "null": << 728 UUID of overlayfs is null. fsid is taken f << 729 - "off": << 730 UUID of overlayfs is null. fsid is taken f << 731 UUID of underlying layers is ignored. << 732 - "on": << 733 UUID of overlayfs is generated and used to << 734 UUID is stored in xattr "trusted.overlay.u << 735 unique and persistent. This option requir << 736 filesystem that supports xattrs. << 737 - "auto": (default) << 738 UUID is taken from xattr "trusted.overlay. << 739 Upgrade to "uuid=on" on first time mount o << 740 meets the prerequites. << 741 Downgrade to "uuid=null" for existing over << 742 mounted with "uuid=on". << 743 << 744 << 745 Volatile mount << 746 -------------- << 747 << 748 This is enabled with the "volatile" mount opti << 749 guaranteed to survive a crash. It is strongly << 750 mounts are only used if data written to the ov << 751 without significant effort. << 752 << 753 The advantage of mounting with the "volatile" << 754 sync calls to the upper filesystem are omitted << 755 << 756 In order to avoid a giving a false sense of sa << 757 semantics of volatile mounts are slightly diff << 758 VFS. If any writeback error occurs on the upp << 759 volatile mount takes place, all sync functions << 760 condition is reached, the filesystem will not << 761 call will return an error, even if the upperdi << 762 since the last sync call. << 763 << 764 When overlay is mounted with "volatile" option << 765 "$workdir/work/incompat/volatile" is created. << 766 checks for this directory and refuses to mount << 767 indicator that user should throw away upper an << 768 fresh one. In very limited cases where the use << 769 not crashed and contents of upperdir are intac << 770 can be removed. << 771 << 772 << 773 User xattr << 774 ---------- << 775 << 776 The "-o userxattr" mount option forces overlay << 777 "user.overlay." xattr namespace instead of "tr << 778 useful for unprivileged mounting of overlayfs. << 779 << 780 566 781 Testsuite 567 Testsuite 782 --------- 568 --------- 783 569 784 There's a testsuite originally developed by Da 570 There's a testsuite originally developed by David Howells and currently 785 maintained by Amir Goldstein at: 571 maintained by Amir Goldstein at: 786 572 787 https://github.com/amir73il/unionmount-testsui !! 573 https://github.com/amir73il/unionmount-testsuite.git 788 574 789 Run as root:: !! 575 Run as root: 790 576 791 # cd unionmount-testsuite 577 # cd unionmount-testsuite 792 # ./run --ov --verify 578 # ./run --ov --verify
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.