1 ========================== 1 ==========================
2 Extensible Scheduler Class 2 Extensible Scheduler Class
3 ========================== 3 ==========================
4 4
5 sched_ext is a scheduler class whose behavior 5 sched_ext is a scheduler class whose behavior can be defined by a set of BPF
6 programs - the BPF scheduler. 6 programs - the BPF scheduler.
7 7
8 * sched_ext exports a full scheduling interfac 8 * sched_ext exports a full scheduling interface so that any scheduling
9 algorithm can be implemented on top. 9 algorithm can be implemented on top.
10 10
11 * The BPF scheduler can group CPUs however it 11 * The BPF scheduler can group CPUs however it sees fit and schedule them
12 together, as tasks aren't tied to specific C 12 together, as tasks aren't tied to specific CPUs at the time of wakeup.
13 13
14 * The BPF scheduler can be turned on and off d 14 * The BPF scheduler can be turned on and off dynamically anytime.
15 15
16 * The system integrity is maintained no matter 16 * The system integrity is maintained no matter what the BPF scheduler does.
17 The default scheduling behavior is restored 17 The default scheduling behavior is restored anytime an error is detected,
18 a runnable task stalls, or on invoking the S 18 a runnable task stalls, or on invoking the SysRq key sequence
19 :kbd:`SysRq-S`. 19 :kbd:`SysRq-S`.
20 20
21 * When the BPF scheduler triggers an error, de 21 * When the BPF scheduler triggers an error, debug information is dumped to
22 aid debugging. The debug dump is passed to a 22 aid debugging. The debug dump is passed to and printed out by the
23 scheduler binary. The debug dump can also be 23 scheduler binary. The debug dump can also be accessed through the
24 `sched_ext_dump` tracepoint. The SysRq key s 24 `sched_ext_dump` tracepoint. The SysRq key sequence :kbd:`SysRq-D`
25 triggers a debug dump. This doesn't terminat 25 triggers a debug dump. This doesn't terminate the BPF scheduler and can
26 only be read through the tracepoint. 26 only be read through the tracepoint.
27 27
28 Switching to and from sched_ext 28 Switching to and from sched_ext
29 =============================== 29 ===============================
30 30
31 ``CONFIG_SCHED_CLASS_EXT`` is the config optio 31 ``CONFIG_SCHED_CLASS_EXT`` is the config option to enable sched_ext and
32 ``tools/sched_ext`` contains the example sched 32 ``tools/sched_ext`` contains the example schedulers. The following config
33 options should be enabled to use sched_ext: 33 options should be enabled to use sched_ext:
34 34
35 .. code-block:: none 35 .. code-block:: none
36 36
37 CONFIG_BPF=y 37 CONFIG_BPF=y
38 CONFIG_SCHED_CLASS_EXT=y 38 CONFIG_SCHED_CLASS_EXT=y
39 CONFIG_BPF_SYSCALL=y 39 CONFIG_BPF_SYSCALL=y
40 CONFIG_BPF_JIT=y 40 CONFIG_BPF_JIT=y
41 CONFIG_DEBUG_INFO_BTF=y 41 CONFIG_DEBUG_INFO_BTF=y
42 CONFIG_BPF_JIT_ALWAYS_ON=y 42 CONFIG_BPF_JIT_ALWAYS_ON=y
43 CONFIG_BPF_JIT_DEFAULT_ON=y 43 CONFIG_BPF_JIT_DEFAULT_ON=y
44 CONFIG_PAHOLE_HAS_SPLIT_BTF=y 44 CONFIG_PAHOLE_HAS_SPLIT_BTF=y
45 CONFIG_PAHOLE_HAS_BTF_TAG=y 45 CONFIG_PAHOLE_HAS_BTF_TAG=y
46 46
47 sched_ext is used only when the BPF scheduler 47 sched_ext is used only when the BPF scheduler is loaded and running.
48 48
49 If a task explicitly sets its scheduling polic 49 If a task explicitly sets its scheduling policy to ``SCHED_EXT``, it will be
50 treated as ``SCHED_NORMAL`` and scheduled by C 50 treated as ``SCHED_NORMAL`` and scheduled by CFS until the BPF scheduler is
51 loaded. 51 loaded.
52 52
53 When the BPF scheduler is loaded and ``SCX_OPS 53 When the BPF scheduler is loaded and ``SCX_OPS_SWITCH_PARTIAL`` is not set
54 in ``ops->flags``, all ``SCHED_NORMAL``, ``SCH 54 in ``ops->flags``, all ``SCHED_NORMAL``, ``SCHED_BATCH``, ``SCHED_IDLE``, and
55 ``SCHED_EXT`` tasks are scheduled by sched_ext 55 ``SCHED_EXT`` tasks are scheduled by sched_ext.
56 56
57 However, when the BPF scheduler is loaded and 57 However, when the BPF scheduler is loaded and ``SCX_OPS_SWITCH_PARTIAL`` is
58 set in ``ops->flags``, only tasks with the ``S 58 set in ``ops->flags``, only tasks with the ``SCHED_EXT`` policy are scheduled
59 by sched_ext, while tasks with ``SCHED_NORMAL` 59 by sched_ext, while tasks with ``SCHED_NORMAL``, ``SCHED_BATCH`` and
60 ``SCHED_IDLE`` policies are scheduled by CFS. 60 ``SCHED_IDLE`` policies are scheduled by CFS.
61 61
62 Terminating the sched_ext scheduler program, t 62 Terminating the sched_ext scheduler program, triggering :kbd:`SysRq-S`, or
63 detection of any internal error including stal 63 detection of any internal error including stalled runnable tasks aborts the
64 BPF scheduler and reverts all tasks back to CF 64 BPF scheduler and reverts all tasks back to CFS.
65 65
66 .. code-block:: none 66 .. code-block:: none
67 67
68 # make -j16 -C tools/sched_ext 68 # make -j16 -C tools/sched_ext
69 # tools/sched_ext/build/bin/scx_simple 69 # tools/sched_ext/build/bin/scx_simple
70 local=0 global=3 70 local=0 global=3
71 local=5 global=24 71 local=5 global=24
72 local=9 global=44 72 local=9 global=44
73 local=13 global=56 73 local=13 global=56
74 local=17 global=72 74 local=17 global=72
75 ^CEXIT: BPF scheduler unregistered 75 ^CEXIT: BPF scheduler unregistered
76 76
77 The current status of the BPF scheduler can be 77 The current status of the BPF scheduler can be determined as follows:
78 78
79 .. code-block:: none 79 .. code-block:: none
80 80
81 # cat /sys/kernel/sched_ext/state 81 # cat /sys/kernel/sched_ext/state
82 enabled 82 enabled
83 # cat /sys/kernel/sched_ext/root/ops 83 # cat /sys/kernel/sched_ext/root/ops
84 simple 84 simple
85 85
86 You can check if any BPF scheduler has ever be 86 You can check if any BPF scheduler has ever been loaded since boot by examining
87 this monotonically incrementing counter (a val 87 this monotonically incrementing counter (a value of zero indicates that no BPF
88 scheduler has been loaded): 88 scheduler has been loaded):
89 89
90 .. code-block:: none 90 .. code-block:: none
91 91
92 # cat /sys/kernel/sched_ext/enable_seq 92 # cat /sys/kernel/sched_ext/enable_seq
93 1 93 1
94 94
95 ``tools/sched_ext/scx_show_state.py`` is a drg 95 ``tools/sched_ext/scx_show_state.py`` is a drgn script which shows more
96 detailed information: 96 detailed information:
97 97
98 .. code-block:: none 98 .. code-block:: none
99 99
100 # tools/sched_ext/scx_show_state.py 100 # tools/sched_ext/scx_show_state.py
101 ops : simple 101 ops : simple
102 enabled : 1 102 enabled : 1
103 switching_all : 1 103 switching_all : 1
104 switched_all : 1 104 switched_all : 1
105 enable_state : enabled (2) 105 enable_state : enabled (2)
106 bypass_depth : 0 106 bypass_depth : 0
107 nr_rejected : 0 107 nr_rejected : 0
108 enable_seq : 1 108 enable_seq : 1
109 109
110 If ``CONFIG_SCHED_DEBUG`` is set, whether a gi 110 If ``CONFIG_SCHED_DEBUG`` is set, whether a given task is on sched_ext can
111 be determined as follows: 111 be determined as follows:
112 112
113 .. code-block:: none 113 .. code-block:: none
114 114
115 # grep ext /proc/self/sched 115 # grep ext /proc/self/sched
116 ext.enabled 116 ext.enabled : 1
117 117
118 The Basics 118 The Basics
119 ========== 119 ==========
120 120
121 Userspace can implement an arbitrary BPF sched 121 Userspace can implement an arbitrary BPF scheduler by loading a set of BPF
122 programs that implement ``struct sched_ext_ops 122 programs that implement ``struct sched_ext_ops``. The only mandatory field
123 is ``ops.name`` which must be a valid BPF obje 123 is ``ops.name`` which must be a valid BPF object name. All operations are
124 optional. The following modified excerpt is fr 124 optional. The following modified excerpt is from
125 ``tools/sched_ext/scx_simple.bpf.c`` showing a 125 ``tools/sched_ext/scx_simple.bpf.c`` showing a minimal global FIFO scheduler.
126 126
127 .. code-block:: c 127 .. code-block:: c
128 128
129 /* 129 /*
130 * Decide which CPU a task should be migra 130 * Decide which CPU a task should be migrated to before being
131 * enqueued (either at wakeup, fork time, 131 * enqueued (either at wakeup, fork time, or exec time). If an
132 * idle core is found by the default ops.s 132 * idle core is found by the default ops.select_cpu() implementation,
133 * then dispatch the task directly to SCX_ 133 * then dispatch the task directly to SCX_DSQ_LOCAL and skip the
134 * ops.enqueue() callback. 134 * ops.enqueue() callback.
135 * 135 *
136 * Note that this implementation has exact 136 * Note that this implementation has exactly the same behavior as the
137 * default ops.select_cpu implementation. 137 * default ops.select_cpu implementation. The behavior of the scheduler
138 * would be exactly same if the implementa 138 * would be exactly same if the implementation just didn't define the
139 * simple_select_cpu() struct_ops prog. 139 * simple_select_cpu() struct_ops prog.
140 */ 140 */
141 s32 BPF_STRUCT_OPS(simple_select_cpu, stru 141 s32 BPF_STRUCT_OPS(simple_select_cpu, struct task_struct *p,
142 s32 prev_cpu, u64 wake_ 142 s32 prev_cpu, u64 wake_flags)
143 { 143 {
144 s32 cpu; 144 s32 cpu;
145 /* Need to initialize or the BPF v 145 /* Need to initialize or the BPF verifier will reject the program */
146 bool direct = false; 146 bool direct = false;
147 147
148 cpu = scx_bpf_select_cpu_dfl(p, pr 148 cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, &direct);
149 149
150 if (direct) 150 if (direct)
151 scx_bpf_dispatch(p, SCX_DS 151 scx_bpf_dispatch(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, 0);
152 152
153 return cpu; 153 return cpu;
154 } 154 }
155 155
156 /* 156 /*
157 * Do a direct dispatch of a task to the g 157 * Do a direct dispatch of a task to the global DSQ. This ops.enqueue()
158 * callback will only be invoked if we fai 158 * callback will only be invoked if we failed to find a core to dispatch
159 * to in ops.select_cpu() above. 159 * to in ops.select_cpu() above.
160 * 160 *
161 * Note that this implementation has exact 161 * Note that this implementation has exactly the same behavior as the
162 * default ops.enqueue implementation, whi 162 * default ops.enqueue implementation, which just dispatches the task
163 * to SCX_DSQ_GLOBAL. The behavior of the 163 * to SCX_DSQ_GLOBAL. The behavior of the scheduler would be exactly same
164 * if the implementation just didn't defin 164 * if the implementation just didn't define the simple_enqueue struct_ops
165 * prog. 165 * prog.
166 */ 166 */
167 void BPF_STRUCT_OPS(simple_enqueue, struct 167 void BPF_STRUCT_OPS(simple_enqueue, struct task_struct *p, u64 enq_flags)
168 { 168 {
169 scx_bpf_dispatch(p, SCX_DSQ_GLOBAL 169 scx_bpf_dispatch(p, SCX_DSQ_GLOBAL, SCX_SLICE_DFL, enq_flags);
170 } 170 }
171 171
172 s32 BPF_STRUCT_OPS_SLEEPABLE(simple_init) 172 s32 BPF_STRUCT_OPS_SLEEPABLE(simple_init)
173 { 173 {
174 /* 174 /*
175 * By default, all SCHED_EXT, SCHE 175 * By default, all SCHED_EXT, SCHED_OTHER, SCHED_IDLE, and
176 * SCHED_BATCH tasks should use sc 176 * SCHED_BATCH tasks should use sched_ext.
177 */ 177 */
178 return 0; 178 return 0;
179 } 179 }
180 180
181 void BPF_STRUCT_OPS(simple_exit, struct sc 181 void BPF_STRUCT_OPS(simple_exit, struct scx_exit_info *ei)
182 { 182 {
183 exit_type = ei->type; 183 exit_type = ei->type;
184 } 184 }
185 185
186 SEC(".struct_ops") 186 SEC(".struct_ops")
187 struct sched_ext_ops simple_ops = { 187 struct sched_ext_ops simple_ops = {
188 .select_cpu = (void *) 188 .select_cpu = (void *)simple_select_cpu,
189 .enqueue = (void *) 189 .enqueue = (void *)simple_enqueue,
190 .init = (void *) 190 .init = (void *)simple_init,
191 .exit = (void *) 191 .exit = (void *)simple_exit,
192 .name = "simple" 192 .name = "simple",
193 }; 193 };
194 194
195 Dispatch Queues 195 Dispatch Queues
196 --------------- 196 ---------------
197 197
198 To match the impedance between the scheduler c 198 To match the impedance between the scheduler core and the BPF scheduler,
199 sched_ext uses DSQs (dispatch queues) which ca 199 sched_ext uses DSQs (dispatch queues) which can operate as both a FIFO and a
200 priority queue. By default, there is one globa 200 priority queue. By default, there is one global FIFO (``SCX_DSQ_GLOBAL``),
201 and one local dsq per CPU (``SCX_DSQ_LOCAL``). 201 and one local dsq per CPU (``SCX_DSQ_LOCAL``). The BPF scheduler can manage
202 an arbitrary number of dsq's using ``scx_bpf_c 202 an arbitrary number of dsq's using ``scx_bpf_create_dsq()`` and
203 ``scx_bpf_destroy_dsq()``. 203 ``scx_bpf_destroy_dsq()``.
204 204
205 A CPU always executes a task from its local DS 205 A CPU always executes a task from its local DSQ. A task is "dispatched" to a
206 DSQ. A non-local DSQ is "consumed" to transfer 206 DSQ. A non-local DSQ is "consumed" to transfer a task to the consuming CPU's
207 local DSQ. 207 local DSQ.
208 208
209 When a CPU is looking for the next task to run 209 When a CPU is looking for the next task to run, if the local DSQ is not
210 empty, the first task is picked. Otherwise, th 210 empty, the first task is picked. Otherwise, the CPU tries to consume the
211 global DSQ. If that doesn't yield a runnable t 211 global DSQ. If that doesn't yield a runnable task either, ``ops.dispatch()``
212 is invoked. 212 is invoked.
213 213
214 Scheduling Cycle 214 Scheduling Cycle
215 ---------------- 215 ----------------
216 216
217 The following briefly shows how a waking task 217 The following briefly shows how a waking task is scheduled and executed.
218 218
219 1. When a task is waking up, ``ops.select_cpu( 219 1. When a task is waking up, ``ops.select_cpu()`` is the first operation
220 invoked. This serves two purposes. First, C 220 invoked. This serves two purposes. First, CPU selection optimization
221 hint. Second, waking up the selected CPU if 221 hint. Second, waking up the selected CPU if idle.
222 222
223 The CPU selected by ``ops.select_cpu()`` is 223 The CPU selected by ``ops.select_cpu()`` is an optimization hint and not
224 binding. The actual decision is made at the 224 binding. The actual decision is made at the last step of scheduling.
225 However, there is a small performance gain 225 However, there is a small performance gain if the CPU
226 ``ops.select_cpu()`` returns matches the CP 226 ``ops.select_cpu()`` returns matches the CPU the task eventually runs on.
227 227
228 A side-effect of selecting a CPU is waking 228 A side-effect of selecting a CPU is waking it up from idle. While a BPF
229 scheduler can wake up any cpu using the ``s 229 scheduler can wake up any cpu using the ``scx_bpf_kick_cpu()`` helper,
230 using ``ops.select_cpu()`` judiciously can 230 using ``ops.select_cpu()`` judiciously can be simpler and more efficient.
231 231
232 A task can be immediately dispatched to a D 232 A task can be immediately dispatched to a DSQ from ``ops.select_cpu()`` by
233 calling ``scx_bpf_dispatch()``. If the task 233 calling ``scx_bpf_dispatch()``. If the task is dispatched to
234 ``SCX_DSQ_LOCAL`` from ``ops.select_cpu()`` 234 ``SCX_DSQ_LOCAL`` from ``ops.select_cpu()``, it will be dispatched to the
235 local DSQ of whichever CPU is returned from 235 local DSQ of whichever CPU is returned from ``ops.select_cpu()``.
236 Additionally, dispatching directly from ``o 236 Additionally, dispatching directly from ``ops.select_cpu()`` will cause the
237 ``ops.enqueue()`` callback to be skipped. 237 ``ops.enqueue()`` callback to be skipped.
238 238
239 Note that the scheduler core will ignore an 239 Note that the scheduler core will ignore an invalid CPU selection, for
240 example, if it's outside the allowed cpumas 240 example, if it's outside the allowed cpumask of the task.
241 241
242 2. Once the target CPU is selected, ``ops.enqu 242 2. Once the target CPU is selected, ``ops.enqueue()`` is invoked (unless the
243 task was dispatched directly from ``ops.sel 243 task was dispatched directly from ``ops.select_cpu()``). ``ops.enqueue()``
244 can make one of the following decisions: 244 can make one of the following decisions:
245 245
246 * Immediately dispatch the task to either t 246 * Immediately dispatch the task to either the global or local DSQ by
247 calling ``scx_bpf_dispatch()`` with ``SCX 247 calling ``scx_bpf_dispatch()`` with ``SCX_DSQ_GLOBAL`` or
248 ``SCX_DSQ_LOCAL``, respectively. 248 ``SCX_DSQ_LOCAL``, respectively.
249 249
250 * Immediately dispatch the task to a custom 250 * Immediately dispatch the task to a custom DSQ by calling
251 ``scx_bpf_dispatch()`` with a DSQ ID whic 251 ``scx_bpf_dispatch()`` with a DSQ ID which is smaller than 2^63.
252 252
253 * Queue the task on the BPF side. 253 * Queue the task on the BPF side.
254 254
255 3. When a CPU is ready to schedule, it first l 255 3. When a CPU is ready to schedule, it first looks at its local DSQ. If
256 empty, it then looks at the global DSQ. If 256 empty, it then looks at the global DSQ. If there still isn't a task to
257 run, ``ops.dispatch()`` is invoked which ca 257 run, ``ops.dispatch()`` is invoked which can use the following two
258 functions to populate the local DSQ. 258 functions to populate the local DSQ.
259 259
260 * ``scx_bpf_dispatch()`` dispatches a task 260 * ``scx_bpf_dispatch()`` dispatches a task to a DSQ. Any target DSQ can
261 be used - ``SCX_DSQ_LOCAL``, ``SCX_DSQ_LO 261 be used - ``SCX_DSQ_LOCAL``, ``SCX_DSQ_LOCAL_ON | cpu``,
262 ``SCX_DSQ_GLOBAL`` or a custom DSQ. While 262 ``SCX_DSQ_GLOBAL`` or a custom DSQ. While ``scx_bpf_dispatch()``
263 currently can't be called with BPF locks 263 currently can't be called with BPF locks held, this is being worked on
264 and will be supported. ``scx_bpf_dispatch 264 and will be supported. ``scx_bpf_dispatch()`` schedules dispatching
265 rather than performing them immediately. 265 rather than performing them immediately. There can be up to
266 ``ops.dispatch_max_batch`` pending tasks. 266 ``ops.dispatch_max_batch`` pending tasks.
267 267
268 * ``scx_bpf_consume()`` tranfers a task fro 268 * ``scx_bpf_consume()`` tranfers a task from the specified non-local DSQ
269 to the dispatching DSQ. This function can 269 to the dispatching DSQ. This function cannot be called with any BPF
270 locks held. ``scx_bpf_consume()`` flushes 270 locks held. ``scx_bpf_consume()`` flushes the pending dispatched tasks
271 before trying to consume the specified DS 271 before trying to consume the specified DSQ.
272 272
273 4. After ``ops.dispatch()`` returns, if there 273 4. After ``ops.dispatch()`` returns, if there are tasks in the local DSQ,
274 the CPU runs the first one. If empty, the f 274 the CPU runs the first one. If empty, the following steps are taken:
275 275
276 * Try to consume the global DSQ. If success 276 * Try to consume the global DSQ. If successful, run the task.
277 277
278 * If ``ops.dispatch()`` has dispatched any 278 * If ``ops.dispatch()`` has dispatched any tasks, retry #3.
279 279
280 * If the previous task is an SCX task and s 280 * If the previous task is an SCX task and still runnable, keep executing
281 it (see ``SCX_OPS_ENQ_LAST``). 281 it (see ``SCX_OPS_ENQ_LAST``).
282 282
283 * Go idle. 283 * Go idle.
284 284
285 Note that the BPF scheduler can always choose 285 Note that the BPF scheduler can always choose to dispatch tasks immediately
286 in ``ops.enqueue()`` as illustrated in the abo 286 in ``ops.enqueue()`` as illustrated in the above simple example. If only the
287 built-in DSQs are used, there is no need to im 287 built-in DSQs are used, there is no need to implement ``ops.dispatch()`` as
288 a task is never queued on the BPF scheduler an 288 a task is never queued on the BPF scheduler and both the local and global
289 DSQs are consumed automatically. 289 DSQs are consumed automatically.
290 290
291 ``scx_bpf_dispatch()`` queues the task on the 291 ``scx_bpf_dispatch()`` queues the task on the FIFO of the target DSQ. Use
292 ``scx_bpf_dispatch_vtime()`` for the priority 292 ``scx_bpf_dispatch_vtime()`` for the priority queue. Internal DSQs such as
293 ``SCX_DSQ_LOCAL`` and ``SCX_DSQ_GLOBAL`` do no 293 ``SCX_DSQ_LOCAL`` and ``SCX_DSQ_GLOBAL`` do not support priority-queue
294 dispatching, and must be dispatched to with `` 294 dispatching, and must be dispatched to with ``scx_bpf_dispatch()``. See the
295 function documentation and usage in ``tools/sc 295 function documentation and usage in ``tools/sched_ext/scx_simple.bpf.c`` for
296 more information. 296 more information.
297 297
298 Where to Look 298 Where to Look
299 ============= 299 =============
300 300
301 * ``include/linux/sched/ext.h`` defines the co 301 * ``include/linux/sched/ext.h`` defines the core data structures, ops table
302 and constants. 302 and constants.
303 303
304 * ``kernel/sched/ext.c`` contains sched_ext co 304 * ``kernel/sched/ext.c`` contains sched_ext core implementation and helpers.
305 The functions prefixed with ``scx_bpf_`` can 305 The functions prefixed with ``scx_bpf_`` can be called from the BPF
306 scheduler. 306 scheduler.
307 307
308 * ``tools/sched_ext/`` hosts example BPF sched 308 * ``tools/sched_ext/`` hosts example BPF scheduler implementations.
309 309
310 * ``scx_simple[.bpf].c``: Minimal global FIF 310 * ``scx_simple[.bpf].c``: Minimal global FIFO scheduler example using a
311 custom DSQ. 311 custom DSQ.
312 312
313 * ``scx_qmap[.bpf].c``: A multi-level FIFO s 313 * ``scx_qmap[.bpf].c``: A multi-level FIFO scheduler supporting five
314 levels of priority implemented with ``BPF_ 314 levels of priority implemented with ``BPF_MAP_TYPE_QUEUE``.
315 315
316 ABI Instability 316 ABI Instability
317 =============== 317 ===============
318 318
319 The APIs provided by sched_ext to BPF schedule 319 The APIs provided by sched_ext to BPF schedulers programs have no stability
320 guarantees. This includes the ops table callba 320 guarantees. This includes the ops table callbacks and constants defined in
321 ``include/linux/sched/ext.h``, as well as the 321 ``include/linux/sched/ext.h``, as well as the ``scx_bpf_`` kfuncs defined in
322 ``kernel/sched/ext.c``. 322 ``kernel/sched/ext.c``.
323 323
324 While we will attempt to provide a relatively 324 While we will attempt to provide a relatively stable API surface when
325 possible, they are subject to change without w 325 possible, they are subject to change without warning between kernel
326 versions. 326 versions.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.