1 .. SPDX-License-Identifier: GPL-2.0
2 .. _iomap_operations:
3
4 ..
5 Dumb style notes to maintain the autho
6 Please try to start sentences on separ
7 sentence changes don't bleed colors in
8 Heading decorations are documented in
9
10 =========================
11 Supported File Operations
12 =========================
13
14 .. contents:: Table of Contents
15 :local:
16
17 Below are a discussion of the high level file
18 implements.
19
20 Buffered I/O
21 ============
22
23 Buffered I/O is the default file I/O path in L
24 File contents are cached in memory ("pagecache
25 writes.
26 Dirty cache will be written back to disk at so
27 forced via ``fsync`` and variants.
28
29 iomap implements nearly all the folio and page
30 filesystems have to implement themselves under
31 This means that the filesystem need not know t
32 mapping, managing uptodate and dirty state, or
33 folios.
34 Under the legacy I/O model, this was managed v
35 linked lists of buffer heads instead of the pe
36 uses.
37 Unless the filesystem explicitly opts in to bu
38 be used, which makes buffered I/O much more ef
39 maintainer much happier.
40
41 ``struct address_space_operations``
42 -----------------------------------
43
44 The following iomap functions can be reference
45 address space operations structure:
46
47 * ``iomap_dirty_folio``
48 * ``iomap_release_folio``
49 * ``iomap_invalidate_folio``
50 * ``iomap_is_partially_uptodate``
51
52 The following address space operations can be
53
54 * ``read_folio``
55 * ``readahead``
56 * ``writepages``
57 * ``bmap``
58 * ``swap_activate``
59
60 ``struct iomap_folio_ops``
61 --------------------------
62
63 The ``->iomap_begin`` function for pagecache o
64 ``struct iomap::folio_ops`` field to an ops st
65 default behaviors of iomap:
66
67 .. code-block:: c
68
69 struct iomap_folio_ops {
70 struct folio *(*get_folio)(struct iomap_i
71 unsigned len);
72 void (*put_folio)(struct inode *inode, lo
73 struct folio *folio);
74 bool (*iomap_valid)(struct inode *inode,
75 };
76
77 iomap calls these functions:
78
79 - ``get_folio``: Called to allocate and retu
80 a locked folio prior to starting a write.
81 If this function is not provided, iomap wi
82 ``iomap_get_folio``.
83 This could be used to `set up per-folio fi
84 <https://lore.kernel.org/all/20190429220934
85 for a write.
86
87 - ``put_folio``: Called to unlock and put a
88 operation completes.
89 If this function is not provided, iomap wi
90 ``folio_put`` on its own.
91 This could be used to `commit per-folio fi
92 <https://lore.kernel.org/all/20180619164137
93 that was set up by ``->get_folio``.
94
95 - ``iomap_valid``: The filesystem may not ho
96 ``->iomap_begin`` and ``->iomap_end`` beca
97 can take folio locks, fault on userspace p
98 for memory reclamation, or engage in other
99 If a file's space mapping data are mutable
100 mapping for a particular pagecache folio c
101 takes
102 <https://lore.kernel.org/all/20221123055812
103 to allocate, install, and lock that folio.
104
105 For the pagecache, races can happen if wri
106 ``i_rwsem`` or ``invalidate_lock`` and upd
107 Races can also happen if the filesytem all
108 For such files, the mapping *must* be reva
109 lock has been taken so that iomap can mana
110
111 fsdax does not need this revalidation beca
112 and no support for unwritten extents.
113
114 Filesystems subject to this kind of race m
115 ``->iomap_valid`` function to decide if th
116 If the mapping is not valid, the mapping w
117
118 To support making the validity decision, t
119 ``->iomap_begin`` function may set ``struc
120 at the same time that it populates the oth
121 A simple validation cookie implementation
122 If the filesystem bumps the sequence count
123 the inode's extent map, it can be placed i
124 iomap::validity_cookie`` during ``->iomap_
125 If the value in the cookie is found to be
126 the filesystem holds when the mapping is p
127 ``->iomap_valid``, then the iomap should c
128 validation failed.
129
130 These ``struct kiocb`` flags are significant f
131
132 * ``IOCB_NOWAIT``: Turns on ``IOMAP_NOWAIT``.
133
134 Internal per-Folio State
135 ------------------------
136
137 If the fsblock size matches the size of a page
138 that all disk I/O operations will operate on t
139 The uptodate (memory contents are at least as
140 dirty (memory contents are newer than what's o
141 folio are all that's needed for this case.
142
143 If the fsblock size is less than the size of a
144 tracks the per-fsblock uptodate and dirty stat
145 This enables iomap to handle both "bs < ps" `f
146 <https://lore.kernel.org/all/20230725122932.144
147 and large folios in the pagecache.
148
149 iomap internally tracks two state bits per fsb
150
151 * ``uptodate``: iomap will try to keep folios
152 If there are read(ahead) errors, those fsbl
153 uptodate.
154 The folio itself will be marked uptodate wh
155 folio are uptodate.
156
157 * ``dirty``: iomap will set the per-block dir
158 write to the file.
159 The folio itself will be marked dirty when
160 folio is dirty.
161
162 iomap also tracks the amount of read and write
163 flight.
164 This structure is much lighter weight than ``s
165 because there is only one per folio, and the p
166 bits vs. 104 bytes.
167
168 Filesystems wishing to turn on large folios in
169 ``mapping_set_large_folios`` when initializing
170
171 Buffered Readahead and Reads
172 ----------------------------
173
174 The ``iomap_readahead`` function initiates rea
175 The ``iomap_read_folio`` function reads one fo
176 the pagecache.
177 The ``flags`` argument to ``->iomap_begin`` wi
178 The pagecache takes whatever locks it needs be
179 filesystem.
180
181 Buffered Writes
182 ---------------
183
184 The ``iomap_file_buffered_write`` function wri
185 pagecache.
186 ``IOMAP_WRITE`` or ``IOMAP_WRITE`` | ``IOMAP_N
187 the ``flags`` argument to ``->iomap_begin``.
188 Callers commonly take ``i_rwsem`` in either sh
189 before calling this function.
190
191 mmap Write Faults
192 ~~~~~~~~~~~~~~~~~
193
194 The ``iomap_page_mkwrite`` function handles a
195 the pagecache.
196 ``IOMAP_WRITE | IOMAP_FAULT`` will be passed a
197 to ``->iomap_begin``.
198 Callers commonly take the mmap ``invalidate_lo
199 exclusive mode before calling this function.
200
201 Buffered Write Failures
202 ~~~~~~~~~~~~~~~~~~~~~~~
203
204 After a short write to the pagecache, the area
205 become marked dirty.
206 The filesystem must arrange to `cancel
207 <https://lore.kernel.org/all/20221123055812.747
208 such `reservations
209 <https://lore.kernel.org/linux-xfs/202208170936
210 because writeback will not consume the reserva
211 The ``iomap_write_delalloc_release`` can be ca
212 ``->iomap_end`` function to find all the clean
213 caching a fresh (``IOMAP_F_NEW``) delalloc map
214 It takes the ``invalidate_lock``.
215
216 The filesystem must supply a function ``punch`
217 each file range in this state.
218 This function must *only* remove delayed alloc
219 case another thread racing with the current th
220 to the same region and triggers writeback to f
221 disk.
222
223 Zeroing for File Operations
224 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
225
226 Filesystems can call ``iomap_zero_range`` to p
227 pagecache for non-truncation file operations t
228 the fsblock size.
229 ``IOMAP_ZERO`` will be passed as the ``flags``
230 ``->iomap_begin``.
231 Callers typically hold ``i_rwsem`` and ``inval
232 mode before calling this function.
233
234 Unsharing Reflinked File Data
235 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
236
237 Filesystems can call ``iomap_file_unshare`` to
238 storage with another file to preemptively copy
239 allocate storage.
240 ``IOMAP_WRITE | IOMAP_UNSHARE`` will be passed
241 to ``->iomap_begin``.
242 Callers typically hold ``i_rwsem`` and ``inval
243 mode before calling this function.
244
245 Truncation
246 ----------
247
248 Filesystems can call ``iomap_truncate_page`` t
249 pagecache from EOF to the end of the fsblock d
250 operation.
251 ``truncate_setsize`` or ``truncate_pagecache``
252 everything after the EOF block.
253 ``IOMAP_ZERO`` will be passed as the ``flags``
254 ``->iomap_begin``.
255 Callers typically hold ``i_rwsem`` and ``inval
256 mode before calling this function.
257
258 Pagecache Writeback
259 -------------------
260
261 Filesystems can call ``iomap_writepages`` to r
262 write dirty pagecache folios to disk.
263 The ``mapping`` and ``wbc`` parameters should
264 The ``wpc`` pointer should be allocated by the
265 be initialized to zero.
266
267 The pagecache will lock each folio before tryi
268 writeback.
269 It does not lock ``i_rwsem`` or ``invalidate_l
270
271 The dirty bit will be cleared for all folios r
272 ``->map_blocks`` machinery described below eve
273 This is to prevent dirty folio clots when stor
274 ``-EIO`` is recorded for userspace to collect
275
276 The ``ops`` structure must be specified and is
277
278 ``struct iomap_writeback_ops``
279 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
280
281 .. code-block:: c
282
283 struct iomap_writeback_ops {
284 int (*map_blocks)(struct iomap_writepage_
285 loff_t offset, unsigned
286 int (*prepare_ioend)(struct iomap_ioend *
287 void (*discard_folio)(struct folio *folio
288 };
289
290 The fields are as follows:
291
292 - ``map_blocks``: Sets ``wpc->iomap`` to the
293 range (in bytes) given by ``offset`` and `
294 iomap calls this function for each dirty f
295 though it will `reuse mappings
296 <https://lore.kernel.org/all/20231207072710
297 for runs of contiguous dirty fsblocks with
298 Do not return ``IOMAP_INLINE`` mappings he
299 function must deal with persisting written
300 Do not return ``IOMAP_DELALLOC`` mappings
301 requires mapping to allocated space.
302 Filesystems can skip a potentially expensi
303 mappings have not changed.
304 This revalidation must be open-coded by th
305 unclear if ``iomap::validity_cookie`` can
306 purpose.
307 This function must be supplied by the file
308
309 - ``prepare_ioend``: Enables filesystems to
310 ioend or perform any other preparatory wor
311 is submitted.
312 This might include pre-write space account
313 a custom ``->bi_end_io`` function for inte
314 deferring the ioend completion to a workqu
315 transactions from process context.
316 This function is optional.
317
318 - ``discard_folio``: iomap calls this functi
319 fails to schedule I/O for any part of a di
320 The function should throw away any reserva
321 made for the write.
322 The folio will be marked clean and an ``-E
323 pagecache.
324 Filesystems can use this callback to `remo
325 <https://lore.kernel.org/all/20201029163313
326 delalloc reservations to avoid having dela
327 clean pagecache.
328 This function is optional.
329
330 Pagecache Writeback Completion
331 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
332
333 To handle the bookkeeping that must happen aft
334 completes, iomap creates chains of ``struct io
335 wrap the ``bio`` that is used to write pagecac
336 By default, iomap finishes writeback ioends by
337 bit on the folios attached to the ``ioend``.
338 If the write failed, it will also set the erro
339 the address space.
340 This can happen in interrupt or process contex
341 storage device.
342
343 Filesystems that need to update internal bookk
344 extent conversions) should provide a ``->prepa
345 set ``struct iomap_end::bio::bi_end_io`` to it
346 This function should call ``iomap_finish_ioend
347 own work (e.g. unwritten extent conversion).
348
349 Some filesystems may wish to `amortize the cos
350 transactions
351 <https://lore.kernel.org/all/20220120034733.221
352 for post-writeback updates by batching them.
353 They may also require transactions to run from
354 implies punting batches to a workqueue.
355 iomap ioends contain a ``list_head`` to enable
356
357 Given a batch of ioends, iomap has a few helpe
358 amortization:
359
360 * ``iomap_sort_ioends``: Sort all the ioends
361 offset.
362
363 * ``iomap_ioend_try_merge``: Given an ioend t
364 a separate list of sorted ioends, merge as
365 the head of the list into the given ioend.
366 ioends can only be merged if the file range
367 contiguous; the unwritten and shared status
368 write I/O outcome is the same.
369 The merged ioends become their own list.
370
371 * ``iomap_finish_ioends``: Finish an ioend th
372 ioends linked to it.
373
374 Direct I/O
375 ==========
376
377 In Linux, direct I/O is defined as file I/O th
378 storage, bypassing the pagecache.
379 The ``iomap_dio_rw`` function implements O_DIR
380 writes for files.
381
382 .. code-block:: c
383
384 ssize_t iomap_dio_rw(struct kiocb *iocb, stru
385 const struct iomap_ops *
386 const struct iomap_dio_o
387 unsigned int dio_flags,
388 size_t done_before);
389
390 The filesystem can provide the ``dops`` parame
391 extra work before or after the I/O is issued t
392 The ``done_before`` parameter tells the how mu
393 already been transferred.
394 It is used to continue a request asynchronousl
395 request
396 <https://git.kernel.org/pub/scm/linux/kernel/g
397 has already been completed synchronously.
398
399 The ``done_before`` parameter should be set if
400 have been initiated prior to the call.
401 The direction of the I/O is determined from th
402
403 The ``dio_flags`` argument can be set to any c
404 following values:
405
406 * ``IOMAP_DIO_FORCE_WAIT``: Wait for the I/O
407 kiocb is not synchronous.
408
409 * ``IOMAP_DIO_OVERWRITE_ONLY``: Perform a pur
410 or fail with ``-EAGAIN``.
411 This can be used by filesystems with comple
412 write paths to provide an optimised fast pa
413 If a pure overwrite can be performed, then
414 other I/Os to the same filesystem block(s)
415 no risk of stale data exposure or data loss
416 If a pure overwrite cannot be performed, th
417 perform the serialisation steps needed to p
418 to the unaligned I/O range so that it can p
419 sub-block zeroing safely.
420 Filesystems can use this flag to try to red
421 but a lot of `detailed checking
422 <https://lore.kernel.org/linux-ext4/20230314
423 is required to do it `correctly
424 <https://lore.kernel.org/linux-ext4/20230810
425
426 * ``IOMAP_DIO_PARTIAL``: If a page fault occu
427 progress has already been made.
428 The caller may deal with the page fault and
429 If the caller decides to retry the operatio
430 accumulated return values of all previous c
431 ``done_before`` parameter to the next call.
432
433 These ``struct kiocb`` flags are significant f
434
435 * ``IOCB_NOWAIT``: Turns on ``IOMAP_NOWAIT``.
436
437 * ``IOCB_SYNC``: Ensure that the device has p
438 before completing the call.
439 In the case of pure overwrites, the I/O may
440 enabled.
441
442 * ``IOCB_HIPRI``: Poll for I/O completion ins
443 interrupt.
444 Only meaningful for asynchronous I/O, and o
445 be issued as a single ``struct bio``.
446
447 * ``IOCB_DIO_CALLER_COMP``: Try to run I/O co
448 process context.
449 See ``linux/fs.h`` for more details.
450
451 Filesystems should call ``iomap_dio_rw`` from
452 ``->write_iter``, and set ``FMODE_CAN_ODIRECT`
453 function for the file.
454 They should not set ``->direct_IO``, which is
455
456 If a filesystem wishes to perform its own work
457 completion, it should call ``__iomap_dio_rw``.
458 If its return value is not an error pointer or
459 filesystem should pass the return value to ``i
460 finishing its internal work.
461
462 Return Values
463 -------------
464
465 ``iomap_dio_rw`` can return one of the followi
466
467 * A non-negative number of bytes transferred.
468
469 * ``-ENOTBLK``: Fall back to buffered I/O.
470 iomap itself will return this value if it c
471 cache before issuing the I/O to storage.
472 The ``->iomap_begin`` or ``->iomap_end`` fu
473 this value.
474
475 * ``-EIOCBQUEUED``: The asynchronous direct I
476 queued and will be completed separately.
477
478 * Any of the other negative error codes.
479
480 Direct Reads
481 ------------
482
483 A direct I/O read initiates a read I/O from th
484 caller's buffer.
485 Dirty parts of the pagecache are flushed to st
486 the read io.
487 The ``flags`` value for ``->iomap_begin`` will
488 any combination of the following enhancements:
489
490 * ``IOMAP_NOWAIT``, as defined previously.
491
492 Callers commonly hold ``i_rwsem`` in shared mo
493 function.
494
495 Direct Writes
496 -------------
497
498 A direct I/O write initiates a write I/O to th
499 caller's buffer.
500 Dirty parts of the pagecache are flushed to st
501 the write io.
502 The pagecache is invalidated both before and a
503 The ``flags`` value for ``->iomap_begin`` will
504 IOMAP_WRITE`` with any combination of the foll
505
506 * ``IOMAP_NOWAIT``, as defined previously.
507
508 * ``IOMAP_OVERWRITE_ONLY``: Allocating blocks
509 blocks is not allowed.
510 The entire file range must map to a single
511 extent.
512 The file I/O range must be aligned to the f
513 if the mapping is unwritten and the filesys
514 the unaligned regions without exposing stal
515
516 Callers commonly hold ``i_rwsem`` in shared or
517 calling this function.
518
519 ``struct iomap_dio_ops:``
520 -------------------------
521 .. code-block:: c
522
523 struct iomap_dio_ops {
524 void (*submit_io)(const struct iomap_iter
525 loff_t file_offset);
526 int (*end_io)(struct kiocb *iocb, ssize_t
527 unsigned flags);
528 struct bio_set *bio_set;
529 };
530
531 The fields of this structure are as follows:
532
533 - ``submit_io``: iomap calls this function w
534 ``struct bio`` object for the I/O requeste
535 to the block device.
536 If no function is provided, ``submit_bio``
537 Filesystems that would like to perform add
538 data replication for btrfs) should impleme
539
540 - ``end_io``: This is called after the ``str
541 This function should perform post-write co
542 extent mappings, handle write failures, et
543 The ``flags`` argument may be set to a com
544
545 * ``IOMAP_DIO_UNWRITTEN``: The mapping was
546 should mark the extent as written.
547
548 * ``IOMAP_DIO_COW``: Writing to the space
549 copy on write operation, so the ioend sh
550
551 - ``bio_set``: This allows the filesystem to
552 for allocating direct I/O bios.
553 This enables filesystems to `stash additio
554 <https://lore.kernel.org/all/20220505201115
555 for private use.
556 If this field is NULL, generic ``struct bi
557
558 Filesystems that want to perform extra work af
559 should set a custom ``->bi_end_io`` function v
560 Afterwards, the custom endio function must cal
561 ``iomap_dio_bio_end_io`` to finish the direct
562
563 DAX I/O
564 =======
565
566 Some storage devices can be directly mapped as
567 These devices support a new access mode known
568 loads and stores through the CPU and memory co
569
570 fsdax Reads
571 -----------
572
573 A fsdax read performs a memcpy from storage de
574 buffer.
575 The ``flags`` value for ``->iomap_begin`` will
576 combination of the following enhancements:
577
578 * ``IOMAP_NOWAIT``, as defined previously.
579
580 Callers commonly hold ``i_rwsem`` in shared mo
581 function.
582
583 fsdax Writes
584 ------------
585
586 A fsdax write initiates a memcpy to the storag
587 buffer.
588 The ``flags`` value for ``->iomap_begin`` will
589 IOMAP_WRITE`` with any combination of the foll
590
591 * ``IOMAP_NOWAIT``, as defined previously.
592
593 * ``IOMAP_OVERWRITE_ONLY``: The caller requir
594 performed from this mapping.
595 This requires the filesystem extent mapping
596 ``IOMAP_MAPPED`` type and span the entire r
597 request.
598 If the filesystem cannot map this request i
599 iomap infrastructure to perform a pure over
600 mapping operation with ``-EAGAIN``.
601
602 Callers commonly hold ``i_rwsem`` in exclusive
603 function.
604
605 fsdax mmap Faults
606 ~~~~~~~~~~~~~~~~~
607
608 The ``dax_iomap_fault`` function handles read
609 storage.
610 For a read fault, ``IOMAP_DAX | IOMAP_FAULT``
611 ``flags`` argument to ``->iomap_begin``.
612 For a write fault, ``IOMAP_DAX | IOMAP_FAULT |
613 passed as the ``flags`` argument to ``->iomap_
614
615 Callers commonly hold the same locks as they d
616 pagecache counterparts.
617
618 fsdax Truncation, fallocate, and Unsharing
619 ------------------------------------------
620
621 For fsdax files, the following functions are p
622 iomap pagecache I/O counterparts.
623 The ``flags`` argument to ``->iomap_begin`` ar
624 pagecache counterparts, with ``IOMAP_DAX`` add
625
626 * ``dax_file_unshare``
627 * ``dax_zero_range``
628 * ``dax_truncate_page``
629
630 Callers commonly hold the same locks as they d
631 pagecache counterparts.
632
633 fsdax Deduplication
634 -------------------
635
636 Filesystems implementing the ``FIDEDUPERANGE``
637 ``dax_remap_file_range_prep`` function with th
638
639 Seeking Files
640 =============
641
642 iomap implements the two iterating whence mode
643 call.
644
645 SEEK_DATA
646 ---------
647
648 The ``iomap_seek_data`` function implements th
649 for llseek.
650 ``IOMAP_REPORT`` will be passed as the ``flags
651 ``->iomap_begin``.
652
653 For unwritten mappings, the pagecache will be
654 Regions of the pagecache with a folio mapped a
655 within those folios will be reported as data a
656
657 Callers commonly hold ``i_rwsem`` in shared mo
658 function.
659
660 SEEK_HOLE
661 ---------
662
663 The ``iomap_seek_hole`` function implements th
664 for llseek.
665 ``IOMAP_REPORT`` will be passed as the ``flags
666 ``->iomap_begin``.
667
668 For unwritten mappings, the pagecache will be
669 Regions of the pagecache with no folio mapped,
670 within a folio will be reported as sparse hole
671
672 Callers commonly hold ``i_rwsem`` in shared mo
673 function.
674
675 Swap File Activation
676 ====================
677
678 The ``iomap_swapfile_activate`` function finds
679 regions in a file and sets them up as swap spa
680 The file will be ``fsync()``'d before activati
681 ``IOMAP_REPORT`` will be passed as the ``flags
682 ``->iomap_begin``.
683 All mappings must be mapped or unwritten; cann
684 cannot span multiple block devices.
685 Callers must hold ``i_rwsem`` in exclusive mod
686 provided by ``swapon``.
687
688 File Space Mapping Reporting
689 ============================
690
691 iomap implements two of the file space mapping
692
693 FS_IOC_FIEMAP
694 -------------
695
696 The ``iomap_fiemap`` function exports file ext
697 in the format specified by the ``FS_IOC_FIEMAP
698 ``IOMAP_REPORT`` will be passed as the ``flags
699 ``->iomap_begin``.
700 Callers commonly hold ``i_rwsem`` in shared mo
701 function.
702
703 FIBMAP (deprecated)
704 -------------------
705
706 ``iomap_bmap`` implements FIBMAP.
707 The calling conventions are the same as for FI
708 This function is only provided to maintain com
709 that implemented FIBMAP prior to conversion.
710 This ioctl is deprecated; do **not** add a FIB
711 filesystems that do not have it.
712 Callers should probably hold ``i_rwsem`` in sh
713 this function, but this is unclear.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.