1 ======================= 2 Kernel Probes (Kprobes) 3 ======================= 4 5 :Author: Jim Keniston <jkenisto@us.ibm.com> 6 :Author: Prasanna S Panchamukhi <prasanna.panch 7 :Author: Masami Hiramatsu <mhiramat@kernel.org> 8 9 .. CONTENTS 10 11 1. Concepts: Kprobes, and Return Probes 12 2. Architectures Supported 13 3. Configuring Kprobes 14 4. API Reference 15 5. Kprobes Features and Limitations 16 6. Probe Overhead 17 7. TODO 18 8. Kprobes Example 19 9. Kretprobes Example 20 10. Deprecated Features 21 Appendix A: The kprobes debugfs interface 22 Appendix B: The kprobes sysctl interface 23 Appendix C: References 24 25 Concepts: Kprobes and Return Probes 26 ========================================= 27 28 Kprobes enables you to dynamically break into 29 collect debugging and performance information 30 can trap at almost any kernel code address [1] 31 routine to be invoked when the breakpoint is h 32 33 .. [1] some parts of the kernel code can not b 34 :ref:`kprobes_blacklist`) 35 36 There are currently two types of probes: kprob 37 (also called return probes). A kprobe can be 38 any instruction in the kernel. A return probe 39 function returns. 40 41 In the typical case, Kprobes-based instrumenta 42 a kernel module. The module's init function i 43 one or more probes, and the exit function unre 44 registration function such as register_kprobe( 45 the probe is to be inserted and what handler i 46 the probe is hit. 47 48 There are also ``register_/unregister_*probes( 49 registration/unregistration of a group of ``*p 50 can speed up unregistration process when you h 51 a lot of probes at once. 52 53 The next four subsections explain how the diff 54 probes work and how jump optimization works. 55 things that you'll need to know in order to ma 56 Kprobes -- e.g., the difference between a pre_ 57 a post_handler, and how to use the maxactive a 58 a kretprobe. But if you're in a hurry to star 59 can skip ahead to :ref:`kprobes_archs_supporte 60 61 How Does a Kprobe Work? 62 ----------------------- 63 64 When a kprobe is registered, Kprobes makes a c 65 instruction and replaces the first byte(s) of 66 with a breakpoint instruction (e.g., int3 on i 67 68 When a CPU hits the breakpoint instruction, a 69 registers are saved, and control passes to Kpr 70 notifier_call_chain mechanism. Kprobes execut 71 associated with the kprobe, passing the handle 72 kprobe struct and the saved registers. 73 74 Next, Kprobes single-steps its copy of the pro 75 (It would be simpler to single-step the actual 76 but then Kprobes would have to temporarily rem 77 instruction. This would open a small time win 78 could sail right past the probepoint.) 79 80 After the instruction is single-stepped, Kprob 81 "post_handler," if any, that is associated wit 82 Execution then continues with the instruction 83 84 Changing Execution Path 85 ----------------------- 86 87 Since kprobes can probe into a running kernel 88 register set, including instruction pointer. T 89 maximum care, such as keeping the stack frame, 90 path etc. Since it operates on a running kerne 91 of computer architecture and concurrent comput 92 your foot. 93 94 If you change the instruction pointer (and set 95 registers) in pre_handler, you must return !0 96 single stepping and just returns to the given 97 This also means post_handler should not be cal 98 99 Note that this operation may be harder on some 100 TOC (Table of Contents) for function call, sin 101 TOC for your function in your module, and reco 102 returning from it. 103 104 Return Probes 105 ------------- 106 107 How Does a Return Probe Work? 108 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 109 110 When you call register_kretprobe(), Kprobes es 111 the entry to the function. When the probed fu 112 probe is hit, Kprobes saves a copy of the retu 113 the return address with the address of a "tram 114 is an arbitrary piece of code -- typically jus 115 At boot time, Kprobes registers a kprobe at th 116 117 When the probed function executes its return i 118 passes to the trampoline and that probe is hit 119 handler calls the user-specified return handle 120 kretprobe, then sets the saved instruction poi 121 address, and that's where execution resumes up 122 123 While the probed function is executing, its re 124 stored in an object of type kretprobe_instance 125 register_kretprobe(), the user sets the maxact 126 kretprobe struct to specify how many instances 127 function can be probed simultaneously. regist 128 pre-allocates the indicated number of kretprob 129 130 For example, if the function is non-recursive 131 spinlock held, maxactive = 1 should be enough. 132 non-recursive and can never relinquish the CPU 133 or preemption), NR_CPUS should be enough. If 134 set to a default value: max(10, 2*NR_CPUS). 135 136 It's not a disaster if you set maxactive too l 137 some probes. In the kretprobe struct, the nmi 138 zero when the return probe is registered, and 139 time the probed function is entered but there 140 object available for establishing the return p 141 142 Kretprobe entry-handler 143 ^^^^^^^^^^^^^^^^^^^^^^^ 144 145 Kretprobes also provides an optional user-spec 146 on function entry. This handler is specified b 147 field of the kretprobe struct. Whenever the kp 148 function entry is hit, the user-defined entry_ 149 If the entry_handler returns 0 (success) then 150 is guaranteed to be called upon function retur 151 returns a non-zero error then Kprobes leaves t 152 the kretprobe has no further effect for that p 153 154 Multiple entry and return handler invocations 155 kretprobe_instance object associated with them 156 may also specify per return-instance private d 157 kretprobe_instance object. This is especially 158 data between corresponding user entry and retu 159 private data object can be specified at kretpr 160 setting the data_size field of the kretprobe s 161 accessed through the data field of each kretpr 162 163 In case probed function is entered but there i 164 object available, then in addition to incremen 165 the user entry_handler invocation is also skip 166 167 .. _kprobes_jump_optimization: 168 169 How Does Jump Optimization Work? 170 -------------------------------- 171 172 If your kernel is built with CONFIG_OPTPROBES= 173 is automatically set 'y' on x86/x86-64, non-pr 174 the "debug.kprobes_optimization" kernel parame 175 sysctl(8)), Kprobes tries to reduce probe-hit 176 instruction instead of a breakpoint instructio 177 178 Init a Kprobe 179 ^^^^^^^^^^^^^ 180 181 When a probe is registered, before attempting 182 Kprobes inserts an ordinary, breakpoint-based 183 address. So, even if it's not possible to opti 184 probepoint, there'll be a probe there. 185 186 Safety Check 187 ^^^^^^^^^^^^ 188 189 Before optimizing a probe, Kprobes performs th 190 191 - Kprobes verifies that the region that will b 192 instruction (the "optimized region") lies en 193 (A jump instruction is multiple bytes, and s 194 instructions.) 195 196 - Kprobes analyzes the entire function and ver 197 jump into the optimized region. Specificall 198 199 - the function contains no indirect jump; 200 - the function contains no instruction that 201 the fixup code triggered by the exception 202 optimized region -- Kprobes checks the exc 203 - there is no near jump to the optimized reg 204 byte). 205 206 - For each instruction in the optimized region 207 the instruction can be executed out of line. 208 209 Preparing Detour Buffer 210 ^^^^^^^^^^^^^^^^^^^^^^^ 211 212 Next, Kprobes prepares a "detour" buffer, whic 213 instruction sequence: 214 215 - code to push the CPU's registers (emulating 216 - a call to the trampoline code which calls us 217 - code to restore registers 218 - the instructions from the optimized region 219 - a jump back to the original execution path. 220 221 Pre-optimization 222 ^^^^^^^^^^^^^^^^ 223 224 After preparing the detour buffer, Kprobes ver 225 following situations exist: 226 227 - The probe has a post_handler. 228 - Other instructions in the optimized region a 229 - The probe is disabled. 230 231 In any of the above cases, Kprobes won't start 232 Since these are temporary situations, Kprobes 233 optimizing it again if the situation is change 234 235 If the kprobe can be optimized, Kprobes enqueu 236 optimizing list, and kicks the kprobe-optimize 237 it. If the to-be-optimized probepoint is hit 238 Kprobes returns control to the original instru 239 the CPU's instruction pointer to the copied co 240 -- thus at least avoiding the single-step. 241 242 Optimization 243 ^^^^^^^^^^^^ 244 245 The Kprobe-optimizer doesn't insert the jump i 246 rather, it calls synchronize_rcu() for safety 247 possible for a CPU to be interrupted in the mi 248 optimized region [3]_. As you know, synchroni 249 that all interruptions that were active when s 250 was called are done, but only if CONFIG_PREEMP 251 of kprobe optimization supports only kernels w 252 253 After that, the Kprobe-optimizer calls stop_ma 254 the optimized region with a jump instruction t 255 using text_poke_smp(). 256 257 Unoptimization 258 ^^^^^^^^^^^^^^ 259 260 When an optimized kprobe is unregistered, disa 261 another kprobe, it will be unoptimized. If th 262 the optimization is complete, the kprobe is ju 263 optimized list. If the optimization has been 264 replaced with the original code (except for an 265 the first byte) by using text_poke_smp(). 266 267 .. [3] Please imagine that the 2nd instruction 268 the optimizer replaces the 2nd instruction 269 while the interrupt handler is running. Whe 270 returns to original address, there is no va 271 and it causes an unexpected result. 272 273 .. [4] This optimization-safety checking may b 274 stop-machine method that ksplice uses for s 275 kernel. 276 277 NOTE for geeks: 278 The jump optimization changes the kprobe's pre 279 Without optimization, the pre_handler can chan 280 path by changing regs->ip and returning 1. Ho 281 is optimized, that modification is ignored. T 282 tweak the kernel's execution path, you need to 283 using one of the following techniques: 284 285 - Specify an empty function for the kprobe's p 286 287 or 288 289 - Execute 'sysctl -w debug.kprobes_optimizatio 290 291 .. _kprobes_blacklist: 292 293 Blacklist 294 --------- 295 296 Kprobes can probe most of the kernel except it 297 that there are some functions where kprobes ca 298 (trapping) such functions can cause a recursiv 299 fault) or the nested probe handler may never b 300 Kprobes manages such functions as a blacklist. 301 If you want to add a function into the blackli 302 to (1) include linux/kprobes.h and (2) use NOK 303 to specify a blacklisted function. 304 Kprobes checks the given probe address against 305 rejects registering it, if the given address i 306 307 .. _kprobes_archs_supported: 308 309 Architectures Supported 310 ======================= 311 312 Kprobes and return probes are implemented on t 313 architectures: 314 315 - i386 (Supports jump optimization) 316 - x86_64 (AMD-64, EM64T) (Supports jump optimi 317 - ppc64 318 - sparc64 (Return probes not yet implemented.) 319 - arm 320 - ppc 321 - mips 322 - s390 323 - parisc 324 - loongarch 325 - riscv 326 327 Configuring Kprobes 328 =================== 329 330 When configuring the kernel using make menucon 331 ensure that CONFIG_KPROBES is set to "y", look 332 "General architecture-dependent options". 333 334 So that you can load and unload Kprobes-based 335 make sure "Loadable module support" (CONFIG_MO 336 unloading" (CONFIG_MODULE_UNLOAD) are set to " 337 338 Also make sure that CONFIG_KALLSYMS and perhap 339 are set to "y", since kallsyms_lookup_name() i 340 kprobe address resolution code. 341 342 If you need to insert a probe in the middle of 343 it useful to "Compile the kernel with debug in 344 so you can use "objdump -d -l vmlinux" to see 345 code mapping. 346 347 API Reference 348 ============= 349 350 The Kprobes API includes a "register" function 351 function for each type of probe. The API also 352 and "unregister_*probes" functions for (un)reg 353 Here are terse, mini-man-page specifications f 354 the associated probe handlers that you'll writ 355 samples/kprobes/ sub-directory for examples. 356 357 register_kprobe 358 --------------- 359 360 :: 361 362 #include <linux/kprobes.h> 363 int register_kprobe(struct kprobe *kp) 364 365 Sets a breakpoint at the address kp->addr. Wh 366 calls kp->pre_handler. After the probed instr 367 calls kp->post_handler. Any or all handlers c 368 KPROBE_FLAG_DISABLED, that kp will be register 369 aren't hit until calling enable_kprobe(kp). 370 371 .. note:: 372 373 1. With the introduction of the "symbol_nam 374 the probepoint address resolution will n 375 The following will now work:: 376 377 kp.symbol_name = "symbol_name"; 378 379 (64-bit powerpc intricacies such as func 380 transparently) 381 382 2. Use the "offset" field of struct kprobe 383 to install a probepoint is known. This f 384 probepoint. 385 386 3. Specify either the kprobe "symbol_name" 387 specified, kprobe registration will fail 388 389 4. With CISC architectures (such as i386 an 390 does not validate if the kprobe.addr is 391 Use "offset" with caution. 392 393 register_kprobe() returns 0 on success, or a n 394 395 User's pre-handler (kp->pre_handler):: 396 397 #include <linux/kprobes.h> 398 #include <linux/ptrace.h> 399 int pre_handler(struct kprobe *p, stru 400 401 Called with p pointing to the kprobe associate 402 and regs pointing to the struct containing the 403 the breakpoint was hit. Return 0 here unless 404 405 User's post-handler (kp->post_handler):: 406 407 #include <linux/kprobes.h> 408 #include <linux/ptrace.h> 409 void post_handler(struct kprobe *p, st 410 unsigned long flags) 411 412 p and regs are as described for the pre_handle 413 to be zero. 414 415 register_kretprobe 416 ------------------ 417 418 :: 419 420 #include <linux/kprobes.h> 421 int register_kretprobe(struct kretprob 422 423 Establishes a return probe for the function wh 424 rp->kp.addr. When that function returns, Kpro 425 You must set rp->maxactive appropriately befor 426 register_kretprobe(); see "How Does a Return P 427 428 register_kretprobe() returns 0 on success, or 429 otherwise. 430 431 User's return-probe handler (rp->handler):: 432 433 #include <linux/kprobes.h> 434 #include <linux/ptrace.h> 435 int kretprobe_handler(struct kretprobe 436 struct pt_regs * 437 438 regs is as described for kprobe.pre_handler. 439 kretprobe_instance object, of which the follow 440 of interest: 441 442 - ret_addr: the return address 443 - rp: points to the corresponding kretprobe ob 444 - task: points to the corresponding task struc 445 - data: points to per return-instance private 446 entry-handler" for details. 447 448 The regs_return_value(regs) macro provides a s 449 extract the return value from the appropriate 450 the architecture's ABI. 451 452 The handler's return value is currently ignore 453 454 unregister_*probe 455 ------------------ 456 457 :: 458 459 #include <linux/kprobes.h> 460 void unregister_kprobe(struct kprobe * 461 void unregister_kretprobe(struct kretp 462 463 Removes the specified probe. The unregister f 464 at any time after the probe has been registere 465 466 .. note:: 467 468 If the functions find an incorrect probe (e 469 they clear the addr field of the probe. 470 471 register_*probes 472 ---------------- 473 474 :: 475 476 #include <linux/kprobes.h> 477 int register_kprobes(struct kprobe **k 478 int register_kretprobes(struct kretpro 479 480 Registers each of the num probes in the specif 481 error occurs during registration, all probes i 482 the bad probe, are safely unregistered before 483 function returns. 484 485 - kps/rps: an array of pointers to ``*probe`` 486 - num: the number of the array entries. 487 488 .. note:: 489 490 You have to allocate(or define) an array of 491 of the array entries before using these fun 492 493 unregister_*probes 494 ------------------ 495 496 :: 497 498 #include <linux/kprobes.h> 499 void unregister_kprobes(struct kprobe 500 void unregister_kretprobes(struct kret 501 502 Removes each of the num probes in the specifie 503 504 .. note:: 505 506 If the functions find some incorrect probes 507 probes) in the specified array, they clear 508 incorrect probes. However, other probes in 509 unregistered correctly. 510 511 disable_*probe 512 -------------- 513 514 :: 515 516 #include <linux/kprobes.h> 517 int disable_kprobe(struct kprobe *kp); 518 int disable_kretprobe(struct kretprobe 519 520 Temporarily disables the specified ``*probe``. 521 enable_*probe(). You must specify the probe wh 522 523 enable_*probe 524 ------------- 525 526 :: 527 528 #include <linux/kprobes.h> 529 int enable_kprobe(struct kprobe *kp); 530 int enable_kretprobe(struct kretprobe 531 532 Enables ``*probe`` which has been disabled by 533 the probe which has been registered. 534 535 Kprobes Features and Limitations 536 ================================ 537 538 Kprobes allows multiple probes at the same add 539 a probepoint for which there is a post_handler 540 So if you install a kprobe with a post_handler 541 probepoint, the probepoint will be unoptimized 542 543 In general, you can install a probe anywhere i 544 In particular, you can probe interrupt handler 545 are discussed in this section. 546 547 The register_*probe functions will return -EIN 548 to install a probe in the code that implements 549 kernel/kprobes.c and ``arch/*/kernel/kprobes.c 550 as do_page_fault and notifier_call_chain). 551 552 If you install a probe in an inline-able funct 553 no attempt to chase down all inline instances 554 install probes there. gcc may inline a functi 555 so keep this in mind if you're not seeing the 556 557 A probe handler can modify the environment of 558 -- e.g., by modifying kernel data structures, 559 contents of the pt_regs struct (which are rest 560 upon return from the breakpoint). So Kprobes 561 to install a bug fix or to inject faults for t 562 course, has no way to distinguish the delibera 563 from the accidental ones. Don't drink and pro 564 565 Kprobes makes no attempt to prevent probe hand 566 each other -- e.g., probing printk() and then 567 probe handler. If a probe handler hits a prob 568 handlers won't be run in that instance, and th 569 of the second probe will be incremented. 570 571 As of Linux v2.6.15-rc1, multiple handlers (or 572 the same handler) may run concurrently on diff 573 574 Kprobes does not use mutexes or allocate memor 575 registration and unregistration. 576 577 Probe handlers are run with preemption disable 578 which depends on the architecture and optimiza 579 kretprobe handlers and optimized kprobe handle 580 disabled on x86/x86-64). In any case, your ha 581 the CPU (e.g., by attempting to acquire a sema 582 583 Since a return probe is implemented by replaci 584 address with the trampoline's address, stack b 585 to __builtin_return_address() will typically y 586 address instead of the real return address for 587 (As far as we can tell, __builtin_return_addre 588 for instrumentation and error reporting.) 589 590 If the number of times a function is called do 591 of times it returns, registering a return prob 592 produce undesirable results. In such a case, a 593 kretprobe BUG!: Processing kretprobe d00000000 594 gets printed. With this information, one will 595 exact instance of the kretprobe that caused th 596 do_exit() case covered. do_execve() and do_for 597 We're unaware of other specific cases where th 598 599 If, upon entry to or exit from a function, the 600 a stack other than that of the current task, r 601 probe on that function may produce undesirable 602 reason, Kprobes doesn't support return probes 603 on the x86_64 version of __switch_to(); the re 604 return -EINVAL. 605 606 On x86/x86-64, since the Jump Optimization of 607 instructions widely, there are some limitation 608 explain it, we introduce some terminology. Ima 609 sequence consisting of a two 2-byte instructio 610 instruction. 611 612 :: 613 614 IA 615 | 616 [-2][-1][0][1][2][3][4][5][6][7] 617 [ins1][ins2][ ins3 ] 618 [<- DCR ->] 619 [<- JTPR ->] 620 621 ins1: 1st Instruction 622 ins2: 2nd Instruction 623 ins3: 3rd Instruction 624 IA: Insertion Address 625 JTPR: Jump Target Prohibition Region 626 DCR: Detoured Code Region 627 628 The instructions in DCR are copied to the out- 629 of the kprobe, because the bytes in DCR are re 630 a 5-byte jump instruction. So there are severa 631 632 a) The instructions in DCR must be relocatable 633 b) The instructions in DCR must not include a 634 c) JTPR must not be targeted by any jump or ca 635 d) DCR must not straddle the border between fu 636 637 Anyway, these limitations are checked by the i 638 decoder, so you don't need to worry about that 639 640 Probe Overhead 641 ============== 642 643 On a typical CPU in use in 2005, a kprobe hit 644 microseconds to process. Specifically, a benc 645 probepoint repeatedly, firing a simple handler 646 million hits per second, depending on the arch 647 hit typically takes 50-75% longer than a kprob 648 When you have a return probe set on a function 649 the entry to that function adds essentially no 650 651 Here are sample overhead figures (in usec) for 652 653 k = kprobe; r = return probe; kr = kprobe + 654 on same function 655 656 i386: Intel Pentium M, 1495 MHz, 2957.31 bog 657 k = 0.57 usec; r = 0.92; kr = 0.99 658 659 x86_64: AMD Opteron 246, 1994 MHz, 3971.48 b 660 k = 0.49 usec; r = 0.80; kr = 0.82 661 662 ppc64: POWER5 (gr), 1656 MHz (SMT disabled, 663 k = 0.77 usec; r = 1.26; kr = 1.45 664 665 Optimized Probe Overhead 666 ------------------------ 667 668 Typically, an optimized kprobe hit takes 0.07 669 process. Here are sample overhead figures (in 670 671 k = unoptimized kprobe, b = boosted (single- 672 r = unoptimized kretprobe, rb = boosted kret 673 674 i386: Intel(R) Xeon(R) E5410, 2.33GHz, 4656. 675 k = 0.80 usec; b = 0.33; o = 0.05; r = 1.10; 676 677 x86-64: Intel(R) Xeon(R) E5410, 2.33GHz, 465 678 k = 0.99 usec; b = 0.43; o = 0.06; r = 1.24; 679 680 TODO 681 ==== 682 683 a. SystemTap (http://sourceware.org/systemtap) 684 programming interface for probe-based instr 685 b. Kernel return probes for sparc64. 686 c. Support for other architectures. 687 d. User-space probes. 688 e. Watchpoint probes (which fire on data refer 689 690 Kprobes Example 691 =============== 692 693 See samples/kprobes/kprobe_example.c 694 695 Kretprobes Example 696 ================== 697 698 See samples/kprobes/kretprobe_example.c 699 700 Deprecated Features 701 =================== 702 703 Jprobes is now a deprecated feature. People wh 704 migrate to other tracing features or use older 705 migrate your tool to one of the following opti 706 707 - Use trace-event to trace target function wit 708 709 trace-event is a low-overhead (and almost no 710 is off) statically defined event interface. 711 and trace it via ftrace or any other tracing 712 713 See the following urls: 714 715 - https://lwn.net/Articles/379903/ 716 - https://lwn.net/Articles/381064/ 717 - https://lwn.net/Articles/383362/ 718 719 - Use ftrace dynamic events (kprobe event) wit 720 721 If you build your kernel with debug info (CO 722 find which register/stack is assigned to whi 723 by using perf-probe and set up new event to 724 725 See following documents: 726 727 - Documentation/trace/kprobetrace.rst 728 - Documentation/trace/events.rst 729 - tools/perf/Documentation/perf-probe.txt 730 731 732 The kprobes debugfs interface 733 ============================= 734 735 736 With recent kernels (> 2.6.20) the list of reg 737 under the /sys/kernel/debug/kprobes/ directory 738 739 /sys/kernel/debug/kprobes/list: Lists all regi 740 741 c015d71a k vfs_read+0x0 742 c03dedc5 r tcp_v4_rcv+0x0 743 744 The first column provides the kernel address w 745 The second column identifies the type of probe 746 while the third column specifies the symbol+of 747 If the probed function belongs to a module, th 748 specified. Following columns show probe status 749 a virtual address that is no longer valid (mod 750 virtual addresses that correspond to modules t 751 such probes are marked with [GONE]. If the pro 752 such probes are marked with [DISABLED]. If the 753 marked with [OPTIMIZED]. If the probe is ftrac 754 [FTRACE]. 755 756 /sys/kernel/debug/kprobes/enabled: Turn kprobe 757 758 Provides a knob to globally and forcibly turn 759 By default, all kprobes are enabled. By echoin 760 registered probes will be disarmed, till such 761 file. Note that this knob just disarms and arm 762 change each probe's disabling state. This mean 763 [DISABLED]) will be not enabled if you turn ON 764 765 766 The kprobes sysctl interface 767 ============================ 768 769 /proc/sys/debug/kprobes-optimization: Turn kpr 770 771 When CONFIG_OPTPROBES=y, this sysctl interface 772 a knob to globally and forcibly turn jump opti 773 :ref:`kprobes_jump_optimization`) ON or OFF. B 774 is allowed (ON). If you echo "0" to this file 775 "debug.kprobes_optimization" to 0 via sysctl, 776 unoptimized, and any new probes registered aft 777 778 Note that this knob *changes* the optimized st 779 probes (marked [OPTIMIZED]) will be unoptimize 780 removed). If the knob is turned on, they will 781 782 References 783 ========== 784 785 For additional information on Kprobes, refer t 786 787 - https://lwn.net/Articles/132196/ 788 - https://www.kernel.org/doc/ols/2006/ols2006v 789
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.