1 :orphan: 1 :orphan:
2 2
3 Making Filesystems Exportable 3 Making Filesystems Exportable
4 ============================= 4 =============================
5 5
6 Overview 6 Overview
7 -------- 7 --------
8 8
9 All filesystem operations require a dentry (or 9 All filesystem operations require a dentry (or two) as a starting
10 point. Local applications have a reference-co 10 point. Local applications have a reference-counted hold on suitable
11 dentries via open file descriptors or cwd/root 11 dentries via open file descriptors or cwd/root. However remote
12 applications that access a filesystem via a re 12 applications that access a filesystem via a remote filesystem protocol
13 such as NFS may not be able to hold such a ref 13 such as NFS may not be able to hold such a reference, and so need a
14 different way to refer to a particular dentry. 14 different way to refer to a particular dentry. As the alternative
15 form of reference needs to be stable across re 15 form of reference needs to be stable across renames, truncates, and
16 server-reboot (among other things, though thes 16 server-reboot (among other things, though these tend to be the most
17 problematic), there is no simple answer like ' 17 problematic), there is no simple answer like 'filename'.
18 18
19 The mechanism discussed here allows each files 19 The mechanism discussed here allows each filesystem implementation to
20 specify how to generate an opaque (outside of 20 specify how to generate an opaque (outside of the filesystem) byte
21 string for any dentry, and how to find an appr 21 string for any dentry, and how to find an appropriate dentry for any
22 given opaque byte string. 22 given opaque byte string.
23 This byte string will be called a "filehandle 23 This byte string will be called a "filehandle fragment" as it
24 corresponds to part of an NFS filehandle. 24 corresponds to part of an NFS filehandle.
25 25
26 A filesystem which supports the mapping betwee 26 A filesystem which supports the mapping between filehandle fragments
27 and dentries will be termed "exportable". 27 and dentries will be termed "exportable".
28 28
29 29
30 30
31 Dcache Issues 31 Dcache Issues
32 ------------- 32 -------------
33 33
34 The dcache normally contains a proper prefix o 34 The dcache normally contains a proper prefix of any given filesystem
35 tree. This means that if any filesystem objec 35 tree. This means that if any filesystem object is in the dcache, then
36 all of the ancestors of that filesystem object 36 all of the ancestors of that filesystem object are also in the dcache.
37 As normal access is by filename this prefix is 37 As normal access is by filename this prefix is created naturally and
38 maintained easily (by each object maintaining 38 maintained easily (by each object maintaining a reference count on
39 its parent). 39 its parent).
40 40
41 However when objects are included into the dca 41 However when objects are included into the dcache by interpreting a
42 filehandle fragment, there is no automatic cre 42 filehandle fragment, there is no automatic creation of a path prefix
43 for the object. This leads to two related but 43 for the object. This leads to two related but distinct features of
44 the dcache that are not needed for normal file 44 the dcache that are not needed for normal filesystem access.
45 45
46 1. The dcache must sometimes contain objects t 46 1. The dcache must sometimes contain objects that are not part of the
47 proper prefix. i.e that are not connected t 47 proper prefix. i.e that are not connected to the root.
48 2. The dcache must be prepared for a newly fou 48 2. The dcache must be prepared for a newly found (via ->lookup) directory
49 to already have a (non-connected) dentry, a 49 to already have a (non-connected) dentry, and must be able to move
50 that dentry into place (based on the parent 50 that dentry into place (based on the parent and name in the
51 ->lookup). This is particularly needed fo 51 ->lookup). This is particularly needed for directories as
52 it is a dcache invariant that directories o 52 it is a dcache invariant that directories only have one dentry.
53 53
54 To implement these features, the dcache has: 54 To implement these features, the dcache has:
55 55
56 a. A dentry flag DCACHE_DISCONNECTED which is 56 a. A dentry flag DCACHE_DISCONNECTED which is set on
57 any dentry that might not be part of the pr 57 any dentry that might not be part of the proper prefix.
58 This is set when anonymous dentries are cre 58 This is set when anonymous dentries are created, and cleared when a
59 dentry is noticed to be a child of a dentry 59 dentry is noticed to be a child of a dentry which is in the proper
60 prefix. If the refcount on a dentry with t 60 prefix. If the refcount on a dentry with this flag set
61 becomes zero, the dentry is immediately dis 61 becomes zero, the dentry is immediately discarded, rather than being
62 kept in the dcache. If a dentry that is no 62 kept in the dcache. If a dentry that is not already in the dcache
63 is repeatedly accessed by filehandle (as NF 63 is repeatedly accessed by filehandle (as NFSD might do), an new dentry
64 will be a allocated for each access, and di 64 will be a allocated for each access, and discarded at the end of
65 the access. 65 the access.
66 66
67 Note that such a dentry can acquire childre 67 Note that such a dentry can acquire children, name, ancestors, etc.
68 without losing DCACHE_DISCONNECTED - that f 68 without losing DCACHE_DISCONNECTED - that flag is only cleared when
69 subtree is successfully reconnected to root 69 subtree is successfully reconnected to root. Until then dentries
70 in such subtree are retained only as long a 70 in such subtree are retained only as long as there are references;
71 refcount reaching zero means immediate evic 71 refcount reaching zero means immediate eviction, same as for unhashed
72 dentries. That guarantees that we won't ne 72 dentries. That guarantees that we won't need to hunt them down upon
73 umount. 73 umount.
74 74
75 b. A primitive for creation of secondary roots 75 b. A primitive for creation of secondary roots - d_obtain_root(inode).
76 Those do _not_ bear DCACHE_DISCONNECTED. T 76 Those do _not_ bear DCACHE_DISCONNECTED. They are placed on the
77 per-superblock list (->s_roots), so they ca 77 per-superblock list (->s_roots), so they can be located at umount
78 time for eviction purposes. 78 time for eviction purposes.
79 79
80 c. Helper routines to allocate anonymous dentr 80 c. Helper routines to allocate anonymous dentries, and to help attach
81 loose directory dentries at lookup time. Th 81 loose directory dentries at lookup time. They are:
82 82
83 d_obtain_alias(inode) will return a dentry 83 d_obtain_alias(inode) will return a dentry for the given inode.
84 If the inode already has a dentry, one o 84 If the inode already has a dentry, one of those is returned.
85 85
86 If it doesn't, a new anonymous (IS_ROOT 86 If it doesn't, a new anonymous (IS_ROOT and
87 DCACHE_DISCONNECTED) dentry is allocated 87 DCACHE_DISCONNECTED) dentry is allocated and attached.
88 88
89 In the case of a directory, care is take 89 In the case of a directory, care is taken that only one dentry
90 can ever be attached. 90 can ever be attached.
91 91
92 d_splice_alias(inode, dentry) will introdu 92 d_splice_alias(inode, dentry) will introduce a new dentry into the tree;
93 either the passed-in dentry or a preexis 93 either the passed-in dentry or a preexisting alias for the given inode
94 (such as an anonymous one created by d_o 94 (such as an anonymous one created by d_obtain_alias), if appropriate.
95 It returns NULL when the passed-in dentr 95 It returns NULL when the passed-in dentry is used, following the calling
96 convention of ->lookup. 96 convention of ->lookup.
97 97
98 Filesystem Issues 98 Filesystem Issues
99 ----------------- 99 -----------------
100 100
101 For a filesystem to be exportable it must: 101 For a filesystem to be exportable it must:
102 102
103 1. provide the filehandle fragment routines 103 1. provide the filehandle fragment routines described below.
104 2. make sure that d_splice_alias is used ra 104 2. make sure that d_splice_alias is used rather than d_add
105 when ->lookup finds an inode for a given 105 when ->lookup finds an inode for a given parent and name.
106 106
107 If inode is NULL, d_splice_alias(inode, 107 If inode is NULL, d_splice_alias(inode, dentry) is equivalent to::
108 108
109 d_add(dentry, inode), NULL 109 d_add(dentry, inode), NULL
110 110
111 Similarly, d_splice_alias(ERR_PTR(err), 111 Similarly, d_splice_alias(ERR_PTR(err), dentry) = ERR_PTR(err)
112 112
113 Typically the ->lookup routine will simp 113 Typically the ->lookup routine will simply end with a::
114 114
115 return d_splice_alias(inode, d 115 return d_splice_alias(inode, dentry);
116 } 116 }
117 117
118 118
119 119
120 A file system implementation declares that ins 120 A file system implementation declares that instances of the filesystem
121 are exportable by setting the s_export_op fiel 121 are exportable by setting the s_export_op field in the struct
122 super_block. This field must point to a "stru 122 super_block. This field must point to a "struct export_operations"
123 struct which has the following members: 123 struct which has the following members:
124 124
125 encode_fh (mandatory) 125 encode_fh (mandatory)
126 Takes a dentry and creates a filehandle fr 126 Takes a dentry and creates a filehandle fragment which may later be used
127 to find or create a dentry for the same ob 127 to find or create a dentry for the same object.
128 128
129 fh_to_dentry (mandatory) 129 fh_to_dentry (mandatory)
130 Given a filehandle fragment, this should f 130 Given a filehandle fragment, this should find the implied object and
131 create a dentry for it (possibly with d_ob 131 create a dentry for it (possibly with d_obtain_alias).
132 132
133 fh_to_parent (optional but strongly recommen 133 fh_to_parent (optional but strongly recommended)
134 Given a filehandle fragment, this should f 134 Given a filehandle fragment, this should find the parent of the
135 implied object and create a dentry for it 135 implied object and create a dentry for it (possibly with
136 d_obtain_alias). May fail if the filehand 136 d_obtain_alias). May fail if the filehandle fragment is too small.
137 137
138 get_parent (optional but strongly recommende 138 get_parent (optional but strongly recommended)
139 When given a dentry for a directory, this 139 When given a dentry for a directory, this should return a dentry for
140 the parent. Quite possibly the parent den 140 the parent. Quite possibly the parent dentry will have been allocated
141 by d_alloc_anon. The default get_parent f 141 by d_alloc_anon. The default get_parent function just returns an error
142 so any filehandle lookup that requires fin 142 so any filehandle lookup that requires finding a parent will fail.
143 ->lookup("..") is *not* used as a default 143 ->lookup("..") is *not* used as a default as it can leave ".." entries
144 in the dcache which are too messy to work 144 in the dcache which are too messy to work with.
145 145
146 get_name (optional) 146 get_name (optional)
147 When given a parent dentry and a child den 147 When given a parent dentry and a child dentry, this should find a name
148 in the directory identified by the parent 148 in the directory identified by the parent dentry, which leads to the
149 object identified by the child dentry. If 149 object identified by the child dentry. If no get_name function is
150 supplied, a default implementation is prov 150 supplied, a default implementation is provided which uses vfs_readdir
151 to find potential names, and matches inode 151 to find potential names, and matches inode numbers to find the correct
152 match. 152 match.
153 153
154 flags 154 flags
155 Some filesystems may need to be handled di 155 Some filesystems may need to be handled differently than others. The
156 export_operations struct also includes a f 156 export_operations struct also includes a flags field that allows the
157 filesystem to communicate such information 157 filesystem to communicate such information to nfsd. See the Export
158 Operations Flags section below for more ex 158 Operations Flags section below for more explanation.
159 159
160 A filehandle fragment consists of an array of 160 A filehandle fragment consists of an array of 1 or more 4byte words,
161 together with a one byte "type". 161 together with a one byte "type".
162 The decode_fh routine should not depend on the 162 The decode_fh routine should not depend on the stated size that is
163 passed to it. This size may be larger than th 163 passed to it. This size may be larger than the original filehandle
164 generated by encode_fh, in which case it will 164 generated by encode_fh, in which case it will have been padded with
165 nuls. Rather, the encode_fh routine should ch 165 nuls. Rather, the encode_fh routine should choose a "type" which
166 indicates the decode_fh how much of the fileha 166 indicates the decode_fh how much of the filehandle is valid, and how
167 it should be interpreted. 167 it should be interpreted.
168 168
169 Export Operations Flags 169 Export Operations Flags
170 ----------------------- 170 -----------------------
171 In addition to the operation vector pointers, 171 In addition to the operation vector pointers, struct export_operations also
172 contains a "flags" field that allows the files 172 contains a "flags" field that allows the filesystem to communicate to nfsd
173 that it may want to do things differently when 173 that it may want to do things differently when dealing with it. The
174 following flags are defined: 174 following flags are defined:
175 175
176 EXPORT_OP_NOWCC - disable NFSv3 WCC attribut 176 EXPORT_OP_NOWCC - disable NFSv3 WCC attributes on this filesystem
177 RFC 1813 recommends that servers always se 177 RFC 1813 recommends that servers always send weak cache consistency
178 (WCC) data to the client after each operat 178 (WCC) data to the client after each operation. The server should
179 atomically collect attributes about the in 179 atomically collect attributes about the inode, do an operation on it,
180 and then collect the attributes afterward. 180 and then collect the attributes afterward. This allows the client to
181 skip issuing GETATTRs in some situations b 181 skip issuing GETATTRs in some situations but means that the server
182 is calling vfs_getattr for almost all RPCs 182 is calling vfs_getattr for almost all RPCs. On some filesystems
183 (particularly those that are clustered or 183 (particularly those that are clustered or networked) this is expensive
184 and atomicity is difficult to guarantee. T 184 and atomicity is difficult to guarantee. This flag indicates to nfsd
185 that it should skip providing WCC attribut 185 that it should skip providing WCC attributes to the client in NFSv3
186 replies when doing operations on this file 186 replies when doing operations on this filesystem. Consider enabling
187 this on filesystems that have an expensive 187 this on filesystems that have an expensive ->getattr inode operation,
188 or when atomicity between pre and post ope 188 or when atomicity between pre and post operation attribute collection
189 is impossible to guarantee. 189 is impossible to guarantee.
190 190
191 EXPORT_OP_NOSUBTREECHK - disallow subtree ch 191 EXPORT_OP_NOSUBTREECHK - disallow subtree checking on this fs
192 Many NFS operations deal with filehandles, 192 Many NFS operations deal with filehandles, which the server must then
193 vet to ensure that they live inside of an 193 vet to ensure that they live inside of an exported tree. When the
194 export consists of an entire filesystem, t 194 export consists of an entire filesystem, this is trivial. nfsd can just
195 ensure that the filehandle live on the fil 195 ensure that the filehandle live on the filesystem. When only part of a
196 filesystem is exported however, then nfsd 196 filesystem is exported however, then nfsd must walk the ancestors of the
197 inode to ensure that it's within an export 197 inode to ensure that it's within an exported subtree. This is an
198 expensive operation and not all filesystem 198 expensive operation and not all filesystems can support it properly.
199 This flag exempts the filesystem from subt 199 This flag exempts the filesystem from subtree checking and causes
200 exportfs to get back an error if it tries 200 exportfs to get back an error if it tries to enable subtree checking
201 on it. 201 on it.
202 202
203 EXPORT_OP_CLOSE_BEFORE_UNLINK - always close 203 EXPORT_OP_CLOSE_BEFORE_UNLINK - always close cached files before unlinking
204 On some exportable filesystems (such as NF 204 On some exportable filesystems (such as NFS) unlinking a file that
205 is still open can cause a fair bit of extr 205 is still open can cause a fair bit of extra work. For instance,
206 the NFS client will do a "sillyrename" to 206 the NFS client will do a "sillyrename" to ensure that the file
207 sticks around while it's still open. When 207 sticks around while it's still open. When reexporting, that open
208 file is held by nfsd so we usually end up 208 file is held by nfsd so we usually end up doing a sillyrename, and
209 then immediately deleting the sillyrenamed 209 then immediately deleting the sillyrenamed file just afterward when
210 the link count actually goes to zero. Some 210 the link count actually goes to zero. Sometimes this delete can race
211 with other operations (for instance an rmd 211 with other operations (for instance an rmdir of the parent directory).
212 This flag causes nfsd to close any open fi 212 This flag causes nfsd to close any open files for this inode _before_
213 calling into the vfs to do an unlink or a 213 calling into the vfs to do an unlink or a rename that would replace
214 an existing file. 214 an existing file.
215 215
216 EXPORT_OP_REMOTE_FS - Backing storage for th 216 EXPORT_OP_REMOTE_FS - Backing storage for this filesystem is remote
217 PF_LOCAL_THROTTLE exists for loopback NFSD 217 PF_LOCAL_THROTTLE exists for loopback NFSD, where a thread needs to
218 write to one bdi (the final bdi) in order 218 write to one bdi (the final bdi) in order to free up writes queued
219 to another bdi (the client bdi). Such thre 219 to another bdi (the client bdi). Such threads get a private balance
220 of dirty pages so that dirty pages for the 220 of dirty pages so that dirty pages for the client bdi do not imact
221 the daemon writing to the final bdi. For f 221 the daemon writing to the final bdi. For filesystems whose durable
222 storage is not local (such as exported NFS 222 storage is not local (such as exported NFS filesystems), this
223 constraint has negative consequences. EXPO 223 constraint has negative consequences. EXPORT_OP_REMOTE_FS enables
224 an export to disable writeback throttling. 224 an export to disable writeback throttling.
225 225
226 EXPORT_OP_NOATOMIC_ATTR - Filesystem does no 226 EXPORT_OP_NOATOMIC_ATTR - Filesystem does not update attributes atomically
227 EXPORT_OP_NOATOMIC_ATTR indicates that the 227 EXPORT_OP_NOATOMIC_ATTR indicates that the exported filesystem
228 cannot provide the semantics required by t 228 cannot provide the semantics required by the "atomic" boolean in
229 NFSv4's change_info4. This boolean indicat 229 NFSv4's change_info4. This boolean indicates to a client whether the
230 returned before and after change attribute 230 returned before and after change attributes were obtained atomically
231 with the respect to the requested metadata 231 with the respect to the requested metadata operation (UNLINK,
232 OPEN/CREATE, MKDIR, etc). 232 OPEN/CREATE, MKDIR, etc).
233 233
234 EXPORT_OP_FLUSH_ON_CLOSE - Filesystem flushe 234 EXPORT_OP_FLUSH_ON_CLOSE - Filesystem flushes file data on close(2)
235 On most filesystems, inodes can remain und 235 On most filesystems, inodes can remain under writeback after the
236 file is closed. NFSD relies on client acti 236 file is closed. NFSD relies on client activity or local flusher
237 threads to handle writeback. Certain files 237 threads to handle writeback. Certain filesystems, such as NFS, flush
238 all of an inode's dirty data on last close 238 all of an inode's dirty data on last close. Exports that behave this
239 way should set EXPORT_OP_FLUSH_ON_CLOSE so 239 way should set EXPORT_OP_FLUSH_ON_CLOSE so that NFSD knows to skip
240 waiting for writeback when closing such fi 240 waiting for writeback when closing such files.
241 241
242 EXPORT_OP_ASYNC_LOCK - Indicates a capable f 242 EXPORT_OP_ASYNC_LOCK - Indicates a capable filesystem to do async lock
243 requests from lockd. Only set EXPORT_OP_AS 243 requests from lockd. Only set EXPORT_OP_ASYNC_LOCK if the filesystem has
244 it's own ->lock() functionality as core po 244 it's own ->lock() functionality as core posix_lock_file() implementation
245 has no async lock request handling yet. Fo 245 has no async lock request handling yet. For more information about how to
246 indicate an async lock request from a ->lo 246 indicate an async lock request from a ->lock() file_operations struct, see
247 fs/locks.c and comment for the function vf 247 fs/locks.c and comment for the function vfs_lock_file().
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.