1 ===========
2 Userfaultfd
3 ===========
4
5 Objective
6 =========
7
8 Userfaults allow the implementation of on-dema
9 and more generally they allow userland to take
10 memory page faults, something otherwise only t
11
12 For example userfaults allows a proper and mor
13 of the ``PROT_NONE+SIGSEGV`` trick.
14
15 Design
16 ======
17
18 Userspace creates a new userfaultfd, initializ
19 regions of virtual memory with it. Then, any p
20 region(s) result in a message being delivered
21 userspace of the fault.
22
23 The ``userfaultfd`` (aside from registering an
24 memory ranges) provides two primary functional
25
26 1) ``read/POLLIN`` protocol to notify a userla
27 happening
28
29 2) various ``UFFDIO_*`` ioctls that can manage
30 registered in the ``userfaultfd`` that allo
31 resolve the userfaults it receives via 1) o
32 memory in the background
33
34 The real advantage of userfaults if compared t
35 management of mremap/mprotect is that the user
36 operations never involve heavyweight structure
37 ``userfaultfd`` runtime load never takes the m
38 Vmas are not suitable for page- (or hugepage)
39 when dealing with virtual address spaces that
40 Terabytes. Too many vmas would be needed for t
41
42 The ``userfaultfd``, once created, can also be
43 passed using unix domain sockets to a manager
44 manager process could handle the userfaults of
45 different processes without them being aware a
46 (well of course unless they later try to use t
47 themselves on the same region the manager is a
48 is a corner case that would currently return `
49
50 API
51 ===
52
53 Creating a userfaultfd
54 ----------------------
55
56 There are two ways to create a new userfaultfd
57 restrict access to this functionality (since h
58 handle kernel page faults have been a useful t
59
60 The first way, supported since userfaultfd was
61 userfaultfd(2) syscall. Access to this is cont
62
63 - Any user can always create a userfaultfd whi
64 only. Such a userfaultfd can be created usin
65 with the flag UFFD_USER_MODE_ONLY.
66
67 - In order to also trap kernel page faults for
68 process needs the CAP_SYS_PTRACE capability,
69 vm.unprivileged_userfaultfd set to 1. By def
70 is set to 0.
71
72 The second way, added to the kernel more recen
73 /dev/userfaultfd and issuing a USERFAULTFD_IOC
74 yields equivalent userfaultfds to the userfaul
75
76 Unlike userfaultfd(2), access to /dev/userfaul
77 filesystem permissions (user/group/mode), whic
78 userfaultfd specifically, without also grantin
79 the same time (as e.g. granting CAP_SYS_PTRACE
80 to /dev/userfaultfd can always create userfaul
81 vm.unprivileged_userfaultfd is not considered.
82
83 Initializing a userfaultfd
84 --------------------------
85
86 When first opened the ``userfaultfd`` must be
87 ``UFFDIO_API`` ioctl specifying a ``uffdio_api
88 a later API version) which will specify the ``
89 userland intends to speak on the ``UFFD`` and
90 userland requires. The ``UFFDIO_API`` ioctl if
91 requested ``uffdio_api.api`` is spoken also by
92 requested features are going to be enabled) wi
93 ``uffdio_api.features`` and ``uffdio_api.ioctl
94 respectively all the available features of the
95 the generic ioctl available.
96
97 The ``uffdio_api.features`` bitmask returned b
98 defines what memory types are supported by the
99 events, except page fault notifications, may b
100
101 - The ``UFFD_FEATURE_EVENT_*`` flags indicate
102 other than page faults are supported. These
103 detail below in the `Non-cooperative userfau
104
105 - ``UFFD_FEATURE_MISSING_HUGETLBFS`` and ``UFF
106 indicate that the kernel supports ``UFFDIO_R
107 registrations for hugetlbfs and shared memor
108 i.e. tmpfs, ``IPCSHM``, ``/dev/zero``, ``MAP
109 etc) virtual memory areas, respectively.
110
111 - ``UFFD_FEATURE_MINOR_HUGETLBFS`` indicates t
112 ``UFFDIO_REGISTER_MODE_MINOR`` registration
113 areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the a
114 support for shmem virtual memory areas.
115
116 - ``UFFD_FEATURE_MOVE`` indicates that the ker
117 existing page contents from userspace.
118
119 The userland application should set the featur
120 when invoking the ``UFFDIO_API`` ioctl, to req
121 enabled if supported.
122
123 Once the ``userfaultfd`` API has been enabled
124 ioctl should be invoked (if present in the ret
125 bitmask) to register a memory range in the ``u
126 uffdio_register structure accordingly. The ``u
127 bitmask will specify to the kernel which kind
128 the range. The ``UFFDIO_REGISTER`` ioctl will
129 ``uffdio_register.ioctls`` bitmask of ioctls t
130 userfaults on the range registered. Not all io
131 supported for all memory types (e.g. anonymous
132 hugetlbfs), or all types of intercepted faults
133
134 Userland can use the ``uffdio_register.ioctls`
135 address space in the background (to add or pot
136 memory from the ``userfaultfd`` registered ran
137 could be triggering just before userland maps
138 user-faulted page.
139
140 Resolving Userfaults
141 --------------------
142
143 There are three basic ways to resolve userfaul
144
145 - ``UFFDIO_COPY`` atomically copies some exist
146 userspace.
147
148 - ``UFFDIO_ZEROPAGE`` atomically zeros the new
149
150 - ``UFFDIO_CONTINUE`` maps an existing, previo
151
152 These operations are atomic in the sense that
153 see a half-populated page, since readers will
154 operation has finished.
155
156 By default, these wake up userfaults blocked o
157 They support a ``UFFDIO_*_MODE_DONTWAKE`` ``mo
158 that waking will be done separately at some la
159
160 Which ioctl to choose depends on the kind of p
161 like to do to resolve it:
162
163 - For ``UFFDIO_REGISTER_MODE_MISSING`` faults,
164 resolved by either providing a new page (``U
165 the zero page (``UFFDIO_ZEROPAGE``). By defa
166 the zero page for a missing fault. With user
167 decide what content to provide before the fa
168
169 - For ``UFFDIO_REGISTER_MODE_MINOR`` faults, t
170 the page cache). Userspace has the option of
171 contents before resolving the fault. Once th
172 (modified or not), userspace asks the kernel
173 faulting thread continue with ``UFFDIO_CONTI
174
175 Notes:
176
177 - You can tell which kind of fault occurred by
178 ``pagefault.flags`` within the ``uffd_msg``,
179 ``UFFD_PAGEFAULT_FLAG_*`` flags.
180
181 - None of the page-delivering ioctls default t
182 registered with. You must fill in all field
183 ioctl struct including the range.
184
185 - You get the address of the access that trigg
186 event out of a struct uffd_msg that you read
187 uffd. You can supply as many pages as you w
188 Keep in mind that unless you used DONTWAKE t
189 those IOCTLs wakes up the faulting thread.
190
191 - Be sure to test for all errors including
192 (``pollfd[0].revents & POLLERR``). This can
193 supplied were incorrect.
194
195 Write Protect Notifications
196 ---------------------------
197
198 This is equivalent to (but faster than) using
199 signal handler.
200
201 Firstly you need to register a range with ``UF
202 Instead of using mprotect(2) you use
203 ``ioctl(uffd, UFFDIO_WRITEPROTECT, struct *uff
204 while ``mode = UFFDIO_WRITEPROTECT_MODE_WP``
205 in the struct passed in. The range does not d
206 have to be identical to the range you register
207 protect as many ranges as you like (inside the
208 Then, in the thread reading from uffd the stru
209 ``msg.arg.pagefault.flags & UFFD_PAGEFAULT_FLA
210 ``ioctl(uffd, UFFDIO_WRITEPROTECT, struct *uff
211 again while ``pagefault.mode`` does not have `
212 set. This wakes up the thread which will conti
213 allows you to do the bookkeeping about the wri
214 thread before the ioctl.
215
216 If you registered with both ``UFFDIO_REGISTER_
217 ``UFFDIO_REGISTER_MODE_WP`` then you need to t
218 which you supply a page and undo write protect
219 difference between writes into a WP area and i
220 former will have ``UFFD_PAGEFAULT_FLAG_WP`` se
221 ``UFFD_PAGEFAULT_FLAG_WRITE``. The latter did
222 you still need to supply a page when ``UFFDIO_
223 used.
224
225 Userfaultfd write-protect mode currently behav
226 (when e.g. page is missing) over different typ
227
228 For anonymous memory, ``ioctl(UFFDIO_WRITEPROT
229 (e.g. when pages are missing and not populated
230 like shmem and hugetlbfs, none ptes will be wr
231 present pte. In other words, there will be a
232 message generated when writing to a missing pa
233 as long as the page range was write-protected
234 not be generated on anonymous memories by defa
235
236 If the application wants to be able to write p
237 memory, one can pre-populate the memory with e
238 newer kernels, one can also detect the feature
239 and set the feature bit in advance to make sur
240 write protected even upon anonymous memory.
241
242 When using ``UFFDIO_REGISTER_MODE_WP`` in comb
243 ``UFFDIO_REGISTER_MODE_MISSING`` or ``UFFDIO_R
244 resolving missing / minor faults with ``UFFDIO
245 respectively, it may be desirable for the new
246 write-protected (so future writes will also re
247 support a mode flag (``UFFDIO_COPY_MODE_WP`` o
248 respectively) to configure the mapping this wa
249
250 If the userfaultfd context has ``UFFD_FEATURE_
251 any vma registered with write-protection will
252 than the default sync mode.
253
254 In async mode, there will be no message genera
255 happens, meanwhile the write-protection will b
256 the kernel. It can be seen as a more accurate
257 tracking and it can be different in a few ways
258
259 - The dirty result will not be affected by v
260 merging) because the dirty is only tracked
261
262 - It supports range operations by default, s
263 any range of memory as long as page aligne
264
265 - Dirty information will not get lost if the
266 various reasons (e.g. during split of a sh
267
268 - Due to a reverted meaning of soft-dirty (p
269 set; dirty when uffd-wp bit cleared), it h
270 some of the memory operations. For exampl
271 anonymous (or ``MADV_REMOVE`` on a file ma
272 dirtying of memory by dropping uffd-wp bit
273
274 The user app can collect the "written/dirty" s
275 uffd-wp bit for the pages being interested in
276
277 The page will not be under track of uffd-wp as
278 explicitly write-protected by ``ioctl(UFFDIO_W
279 flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Try
280 that was tracked by async mode userfaultfd-wp
281
282 When userfaultfd-wp async mode is used alone,
283 kinds of memory.
284
285 Memory Poisioning Emulation
286 ---------------------------
287
288 In response to a fault (either missing or mino
289 take to "resolve" it is to issue a ``UFFDIO_PO
290 future faulters to either get a SIGBUS, or in
291 receive an MCE as if there were hardware memor
292
293 This is used to emulate hardware memory poison
294 machine which experiences a real hardware memo
295 the VM to another physical machine. Since we w
296 transparent to the guest, we want that same ad
297 still poisoned, even though it's on a new phys
298 doesn't have a memory error in the exact same
299
300 QEMU/KVM
301 ========
302
303 QEMU/KVM is using the ``userfaultfd`` syscall
304 migration. Postcopy live migration is one form
305 externalization consisting of a virtual machin
306 all of its memory residing on a different node
307 ``userfaultfd`` abstraction is generic enough
308 KVM kernel code had to be modified in order to
309 migration to QEMU.
310
311 Guest async page faults, ``FOLL_NOWAIT`` and a
312 just fine in combination with userfaults. User
313 page faults in the guest scheduler so those gu
314 aren't waiting for userfaults (i.e. network bo
315 the guest vcpus.
316
317 It is generally beneficial to run one pass of
318 just before starting postcopy live migration,
319 generating userfaults for readonly guest regio
320
321 The implementation of postcopy live migration
322 single bidirectional socket but in the future
323 will be used (to reduce the latency of the use
324 possible without having to decrease ``/proc/sy
325
326 The QEMU in the source node writes all pages t
327 in the destination node, into the socket, and
328 the QEMU running in the destination node runs
329 ioctls on the ``userfaultfd`` in order to map
330 guest (``UFFDIO_ZEROCOPY`` is used if the sour
331
332 A different postcopy thread in the destination
333 poll() to the ``userfaultfd`` in parallel. Whe
334 generated after a userfault triggers, the post
335 the ``userfaultfd`` and receives the fault add
336 userfault was already resolved and waken by a
337 by the parallel QEMU migration thread).
338
339 After the QEMU postcopy thread (running in the
340 the userfault address it writes the informatio
341 into the socket. The QEMU source node receives
342 roughly "seeks" to that page address and conti
343 remaining missing pages from that new page off
344 (just the time to flush the tcp_wmem queue thr
345 migration thread in the QEMU running in the de
346 receive the page that triggered the userfault
347 usual with the ``UFFDIO_COPY|ZEROPAGE`` (witho
348 was spontaneously sent by the source or if it
349 requested through a userfault).
350
351 By the time the userfaults start, the QEMU in
352 doesn't need to keep any per-page state bitmap
353 migration around and a single per-page bitmap
354 the QEMU running in the source node to know wh
355 missing in the destination node. The bitmap in
356 checked to find which missing pages to send in
357 over it when receiving incoming userfaults. Af
358 course the bitmap is updated accordingly. It's
359 sending the same page twice (in case the userf
360 postcopy thread just before ``UFFDIO_COPY|ZERO
361 thread).
362
363 Non-cooperative userfaultfd
364 ===========================
365
366 When the ``userfaultfd`` is monitored by an ex
367 must be able to track changes in the process v
368 layout. Userfaultfd can notify the manager abo
369 the same read(2) protocol as for the page faul
370 manager has to explicitly enable these events
371 bits in ``uffdio_api.features`` passed to ``UF
372
373 ``UFFD_FEATURE_EVENT_FORK``
374 enable ``userfaultfd`` hooks for fork(
375 enabled, the ``userfaultfd`` context o
376 duplicated into the newly created proc
377 receives ``UFFD_EVENT_FORK`` with file
378 ``userfaultfd`` context in the ``uffd_
379
380 ``UFFD_FEATURE_EVENT_REMAP``
381 enable notifications about mremap() ca
382 non-cooperative process moves a virtua
383 different location, the manager will r
384 ``UFFD_EVENT_REMAP``. The ``uffd_msg.r
385 new addresses of the area and its orig
386
387 ``UFFD_FEATURE_EVENT_REMOVE``
388 enable notifications about madvise(MAD
389 madvise(MADV_DONTNEED) calls. The even
390 be generated upon these calls to madvi
391 will contain start and end addresses o
392
393 ``UFFD_FEATURE_EVENT_UNMAP``
394 enable notifications about memory unma
395 get ``UFFD_EVENT_UNMAP`` with ``uffd_m
396 end addresses of the unmapped area.
397
398 Although the ``UFFD_FEATURE_EVENT_REMOVE`` and
399 are pretty similar, they quite differ in the a
400 ``userfaultfd`` manager. In the former case, t
401 removed, but the area is not, the area remains
402 ``userfaultfd``, and if a page fault occurs in
403 delivered to the manager. The proper resolutio
404 to zeromap the faulting address. However, in t
405 area is unmapped, either explicitly (with munm
406 implicitly (e.g. during mremap()), the area is
407 ``userfaultfd`` context for such area disappea
408 not get further userland page faults from the
409 notification is required in order to prevent m
410 ``UFFDIO_COPY`` on the unmapped area.
411
412 Unlike userland page faults which have to be s
413 explicit or implicit wakeup, all the events ar
414 asynchronously and the non-cooperative process
415 soon as manager executes read(). The ``userfau
416 carefully synchronize calls to ``UFFDIO_COPY``
417 processing. To aid the synchronization, the ``
418 return ``-ENOSPC`` when the monitored process
419 ``UFFDIO_COPY``, and ``-ENOENT``, when the non
420 its virtual memory layout simultaneously with
421 operation.
422
423 The current asynchronous model of the event de
424 single threaded non-cooperative ``userfaultfd`
425 synchronous event delivery model can be added
426 ``userfaultfd`` feature to facilitate multithr
427 non cooperative manager, for example to allow
428 run in parallel to the event reception. Single
429 implementations should continue to use the cur
430 delivery model instead.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.