1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY 1 .. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
2 2
3 ============================================== 3 ======================================================
4 Discovering Linux kernel subsystems used by a 4 Discovering Linux kernel subsystems used by a workload
5 ============================================== 5 ======================================================
6 6
7 :Authors: - Shuah Khan <skhan@linuxfoundation.o 7 :Authors: - Shuah Khan <skhan@linuxfoundation.org>
8 - Shefali Sharma <sshefali021@gmail.c 8 - Shefali Sharma <sshefali021@gmail.com>
9 :maintained-by: Shuah Khan <skhan@linuxfoundati 9 :maintained-by: Shuah Khan <skhan@linuxfoundation.org>
10 10
11 Key Points 11 Key Points
12 ========== 12 ==========
13 13
14 * Understanding system resources necessary to 14 * Understanding system resources necessary to build and run a workload
15 is important. 15 is important.
16 * Linux tracing and strace can be used to dis 16 * Linux tracing and strace can be used to discover the system resources
17 in use by a workload. The completeness of t 17 in use by a workload. The completeness of the system usage information
18 depends on the completeness of coverage of 18 depends on the completeness of coverage of a workload.
19 * Performance and security of the operating s 19 * Performance and security of the operating system can be analyzed with
20 the help of tools such as: 20 the help of tools such as:
21 `perf <https://man7.org/linux/man-pages/man 21 `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_,
22 `stress-ng <https://www.mankier.com/1/stres 22 `stress-ng <https://www.mankier.com/1/stress-ng>`_,
23 `paxtest <https://github.com/opntr/paxtest- 23 `paxtest <https://github.com/opntr/paxtest-freebsd>`_.
24 * Once we discover and understand the workloa 24 * Once we discover and understand the workload needs, we can focus on them
25 to avoid regressions and use it to evaluate 25 to avoid regressions and use it to evaluate safety considerations.
26 26
27 Methodology 27 Methodology
28 =========== 28 ===========
29 29
30 `strace <https://man7.org/linux/man-pages/man1 30 `strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_ is a
31 diagnostic, instructional, and debugging tool 31 diagnostic, instructional, and debugging tool and can be used to discover
32 the system resources in use by a workload. Onc 32 the system resources in use by a workload. Once we discover and understand
33 the workload needs, we can focus on them to av 33 the workload needs, we can focus on them to avoid regressions and use it
34 to evaluate safety considerations. We use stra 34 to evaluate safety considerations. We use strace tool to trace workloads.
35 35
36 This method of tracing using strace tells us t 36 This method of tracing using strace tells us the system calls invoked by
37 the workload and doesn't include all the syste 37 the workload and doesn't include all the system calls that can be invoked
38 by it. In addition, this tracing method tells 38 by it. In addition, this tracing method tells us just the code paths within
39 these system calls that are invoked. As an exa 39 these system calls that are invoked. As an example, if a workload opens a
40 file and reads from it successfully, then the 40 file and reads from it successfully, then the success path is the one that
41 is traced. Any error paths in that system call 41 is traced. Any error paths in that system call will not be traced. If there
42 is a workload that provides full coverage of a 42 is a workload that provides full coverage of a workload then the method
43 outlined here will trace and find all possible 43 outlined here will trace and find all possible code paths. The completeness
44 of the system usage information depends on the 44 of the system usage information depends on the completeness of coverage of a
45 workload. 45 workload.
46 46
47 The goal is tracing a workload on a system run 47 The goal is tracing a workload on a system running a default kernel without
48 requiring custom kernel installs. 48 requiring custom kernel installs.
49 49
50 How do we gather fine-grained system informati 50 How do we gather fine-grained system information?
51 ============================================== 51 =================================================
52 52
53 strace tool can be used to trace system calls 53 strace tool can be used to trace system calls made by a process and signals
54 it receives. System calls are the fundamental 54 it receives. System calls are the fundamental interface between an
55 application and the operating system kernel. T 55 application and the operating system kernel. They enable a program to
56 request services from the kernel. For instance 56 request services from the kernel. For instance, the open() system call in
57 Linux is used to provide access to a file in t 57 Linux is used to provide access to a file in the file system. strace enables
58 us to track all the system calls made by an ap 58 us to track all the system calls made by an application. It lists all the
59 system calls made by a process and their resul 59 system calls made by a process and their resulting output.
60 60
61 You can generate profiling data combining stra 61 You can generate profiling data combining strace and perf record tools to
62 record the events and information associated w 62 record the events and information associated with a process. This provides
63 insight into the process. "perf annotate" tool 63 insight into the process. "perf annotate" tool generates the statistics of
64 each instruction of the program. This document 64 each instruction of the program. This document goes over the details of how
65 to gather fine-grained information on a worklo 65 to gather fine-grained information on a workload's usage of system resources.
66 66
67 We used strace to trace the perf, stress-ng, p 67 We used strace to trace the perf, stress-ng, paxtest workloads to illustrate
68 our methodology to discover resources used by 68 our methodology to discover resources used by a workload. This process can
69 be applied to trace other workloads. 69 be applied to trace other workloads.
70 70
71 Getting the system ready for tracing 71 Getting the system ready for tracing
72 ==================================== 72 ====================================
73 73
74 Before we can get started we will show you how 74 Before we can get started we will show you how to get your system ready.
75 We assume that you have a Linux distribution r 75 We assume that you have a Linux distribution running on a physical system
76 or a virtual machine. Most distributions will 76 or a virtual machine. Most distributions will include strace command. Let’s
77 install other tools that aren’t usually incl 77 install other tools that aren’t usually included to build Linux kernel.
78 Please note that the following works on Debian 78 Please note that the following works on Debian based distributions. You
79 might have to find equivalent packages on othe 79 might have to find equivalent packages on other Linux distributions.
80 80
81 Install tools to build Linux kernel and tools 81 Install tools to build Linux kernel and tools in kernel repository.
82 scripts/ver_linux is a good way to check if yo 82 scripts/ver_linux is a good way to check if your system already has
83 the necessary tools:: 83 the necessary tools::
84 84
85 sudo apt-get build-essentials flex bison yac 85 sudo apt-get build-essentials flex bison yacc
86 sudo apt install libelf-dev systemtap-sdt-de 86 sudo apt install libelf-dev systemtap-sdt-dev libaudit-dev libslang2-dev libperl-dev libdw-dev
87 87
88 cscope is a good tool to browse kernel sources 88 cscope is a good tool to browse kernel sources. Let's install it now::
89 89
90 sudo apt-get install cscope 90 sudo apt-get install cscope
91 91
92 Install stress-ng and paxtest:: 92 Install stress-ng and paxtest::
93 93
94 apt-get install stress-ng 94 apt-get install stress-ng
95 apt-get install paxtest 95 apt-get install paxtest
96 96
97 Workload overview 97 Workload overview
98 ================= 98 =================
99 99
100 As mentioned earlier, we used strace to trace 100 As mentioned earlier, we used strace to trace perf bench, stress-ng and
101 paxtest workloads to show how to analyze a wor 101 paxtest workloads to show how to analyze a workload and identify Linux
102 subsystems used by these workloads. Let's star 102 subsystems used by these workloads. Let's start with an overview of these
103 three workloads to get a better understanding 103 three workloads to get a better understanding of what they do and how to
104 use them. 104 use them.
105 105
106 perf bench (all) workload 106 perf bench (all) workload
107 ------------------------- 107 -------------------------
108 108
109 The perf bench command contains multiple multi 109 The perf bench command contains multiple multi-threaded microkernel
110 benchmarks for executing different subsystems 110 benchmarks for executing different subsystems in the Linux kernel and
111 system calls. This allows us to easily measure 111 system calls. This allows us to easily measure the impact of changes,
112 which can help mitigate performance regression 112 which can help mitigate performance regressions. It also acts as a common
113 benchmarking framework, enabling developers to 113 benchmarking framework, enabling developers to easily create test cases,
114 integrate transparently, and use performance-r 114 integrate transparently, and use performance-rich tooling subsystems.
115 115
116 Stress-ng netdev stressor workload 116 Stress-ng netdev stressor workload
117 ---------------------------------- 117 ----------------------------------
118 118
119 stress-ng is used for performing stress testin 119 stress-ng is used for performing stress testing on the kernel. It allows
120 you to exercise various physical subsystems of 120 you to exercise various physical subsystems of the computer, as well as
121 interfaces of the OS kernel, using "stressor-s 121 interfaces of the OS kernel, using "stressor-s". They are available for
122 CPU, CPU cache, devices, I/O, interrupts, file 122 CPU, CPU cache, devices, I/O, interrupts, file system, memory, network,
123 operating system, pipelines, schedulers, and v 123 operating system, pipelines, schedulers, and virtual machines. Please refer
124 to the `stress-ng man-page <https://www.mankie 124 to the `stress-ng man-page <https://www.mankier.com/1/stress-ng>`_ to
125 find the description of all the available stre 125 find the description of all the available stressor-s. The netdev stressor
126 starts specified number (N) of workers that ex 126 starts specified number (N) of workers that exercise various netdevice
127 ioctl commands across all the available networ 127 ioctl commands across all the available network devices.
128 128
129 paxtest kiddie workload 129 paxtest kiddie workload
130 ----------------------- 130 -----------------------
131 131
132 paxtest is a program that tests buffer overflo 132 paxtest is a program that tests buffer overflows in the kernel. It tests
133 kernel enforcements over memory usage. General 133 kernel enforcements over memory usage. Generally, execution in some memory
134 segments makes buffer overflows possible. It r 134 segments makes buffer overflows possible. It runs a set of programs that
135 attempt to subvert memory usage. It is used as 135 attempt to subvert memory usage. It is used as a regression test suite for
136 PaX, but might be useful to test other memory 136 PaX, but might be useful to test other memory protection patches for the
137 kernel. We used paxtest kiddie mode which look 137 kernel. We used paxtest kiddie mode which looks for simple vulnerabilities.
138 138
139 What is strace and how do we use it? 139 What is strace and how do we use it?
140 ==================================== 140 ====================================
141 141
142 As mentioned earlier, strace which is a useful 142 As mentioned earlier, strace which is a useful diagnostic, instructional,
143 and debugging tool and can be used to discover 143 and debugging tool and can be used to discover the system resources in use
144 by a workload. It can be used: 144 by a workload. It can be used:
145 145
146 * To see how a process interacts with the ker 146 * To see how a process interacts with the kernel.
147 * To see why a process is failing or hanging. 147 * To see why a process is failing or hanging.
148 * For reverse engineering a process. 148 * For reverse engineering a process.
149 * To find the files on which a program depend 149 * To find the files on which a program depends.
150 * For analyzing the performance of an applica 150 * For analyzing the performance of an application.
151 * For troubleshooting various problems relate 151 * For troubleshooting various problems related to the operating system.
152 152
153 In addition, strace can generate run-time stat 153 In addition, strace can generate run-time statistics on times, calls, and
154 errors for each system call and report a summa 154 errors for each system call and report a summary when program exits,
155 suppressing the regular output. This attempts 155 suppressing the regular output. This attempts to show system time (CPU time
156 spent running in the kernel) independent of wa 156 spent running in the kernel) independent of wall clock time. We plan to use
157 these features to get information on workload 157 these features to get information on workload system usage.
158 158
159 strace command supports basic, verbose, and st 159 strace command supports basic, verbose, and stats modes. strace command when
160 run in verbose mode gives more detailed inform 160 run in verbose mode gives more detailed information about the system calls
161 invoked by a process. 161 invoked by a process.
162 162
163 Running strace -c generates a report of the pe 163 Running strace -c generates a report of the percentage of time spent in each
164 system call, the total time in seconds, the mi 164 system call, the total time in seconds, the microseconds per call, the total
165 number of calls, the count of each system call 165 number of calls, the count of each system call that has failed with an error
166 and the type of system call made. 166 and the type of system call made.
167 167
168 * Usage: strace <command we want to trace> 168 * Usage: strace <command we want to trace>
169 * Verbose mode usage: strace -v <command> 169 * Verbose mode usage: strace -v <command>
170 * Gather statistics: strace -c <command> 170 * Gather statistics: strace -c <command>
171 171
172 We used the “-c” option to gather fine-gra 172 We used the “-c” option to gather fine-grained run-time statistics in use
173 by three workloads we have chose for this anal 173 by three workloads we have chose for this analysis.
174 174
175 * perf 175 * perf
176 * stress-ng 176 * stress-ng
177 * paxtest 177 * paxtest
178 178
179 What is cscope and how do we use it? 179 What is cscope and how do we use it?
180 ==================================== 180 ====================================
181 181
182 Now let’s look at `cscope <https://cscope.so 182 Now let’s look at `cscope <https://cscope.sourceforge.net/>`_, a command
183 line tool for browsing C, C++ or Java code-bas 183 line tool for browsing C, C++ or Java code-bases. We can use it to find
184 all the references to a symbol, global definit 184 all the references to a symbol, global definitions, functions called by a
185 function, functions calling a function, text s 185 function, functions calling a function, text strings, regular expression
186 patterns, files including a file. 186 patterns, files including a file.
187 187
188 We can use cscope to find which system call be 188 We can use cscope to find which system call belongs to which subsystem.
189 This way we can find the kernel subsystems use 189 This way we can find the kernel subsystems used by a process when it is
190 executed. 190 executed.
191 191
192 Let’s checkout the latest Linux repository a 192 Let’s checkout the latest Linux repository and build cscope database::
193 193
194 git clone git://git.kernel.org/pub/scm/linux 194 git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux
195 cd linux 195 cd linux
196 cscope -R -p10 # builds cscope.out database 196 cscope -R -p10 # builds cscope.out database before starting browse session
197 cscope -d -p10 # starts browse session on c 197 cscope -d -p10 # starts browse session on cscope.out database
198 198
199 Note: Run "cscope -R -p10" to build the databa 199 Note: Run "cscope -R -p10" to build the database and c"scope -d -p10" to
200 enter into the browsing session. cscope by def 200 enter into the browsing session. cscope by default cscope.out database.
201 To get out of this mode press ctrl+d. -p optio 201 To get out of this mode press ctrl+d. -p option is used to specify the
202 number of file path components to display. -p1 202 number of file path components to display. -p10 is optimal for browsing
203 kernel sources. 203 kernel sources.
204 204
205 What is perf and how do we use it? 205 What is perf and how do we use it?
206 ================================== 206 ==================================
207 207
208 Perf is an analysis tool based on Linux 2.6+ s 208 Perf is an analysis tool based on Linux 2.6+ systems, which abstracts the
209 CPU hardware difference in performance measure 209 CPU hardware difference in performance measurement in Linux, and provides
210 a simple command line interface. Perf is based 210 a simple command line interface. Perf is based on the perf_events interface
211 exported by the kernel. It is very useful for 211 exported by the kernel. It is very useful for profiling the system and
212 finding performance bottlenecks in an applicat 212 finding performance bottlenecks in an application.
213 213
214 If you haven't already checked out the Linux m 214 If you haven't already checked out the Linux mainline repository, you can do
215 so and then build kernel and perf tool:: 215 so and then build kernel and perf tool::
216 216
217 git clone git://git.kernel.org/pub/scm/linux 217 git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux
218 cd linux 218 cd linux
219 make -j3 all 219 make -j3 all
220 cd tools/perf 220 cd tools/perf
221 make 221 make
222 222
223 Note: The perf command can be built without bu 223 Note: The perf command can be built without building the kernel in the
224 repository and can be run on older kernels. Ho 224 repository and can be run on older kernels. However matching the kernel
225 and perf revisions gives more accurate informa 225 and perf revisions gives more accurate information on the subsystem usage.
226 226
227 We used "perf stat" and "perf bench" options. 227 We used "perf stat" and "perf bench" options. For a detailed information on
228 the perf tool, run "perf -h". 228 the perf tool, run "perf -h".
229 229
230 perf stat 230 perf stat
231 --------- 231 ---------
232 The perf stat command generates a report of va 232 The perf stat command generates a report of various hardware and software
233 events. It does so with the help of hardware c 233 events. It does so with the help of hardware counter registers found in
234 modern CPUs that keep the count of these activ 234 modern CPUs that keep the count of these activities. "perf stat cal" shows
235 stats for cal command. 235 stats for cal command.
236 236
237 Perf bench 237 Perf bench
238 ---------- 238 ----------
239 The perf bench command contains multiple multi 239 The perf bench command contains multiple multi-threaded microkernel
240 benchmarks for executing different subsystems 240 benchmarks for executing different subsystems in the Linux kernel and
241 system calls. This allows us to easily measure 241 system calls. This allows us to easily measure the impact of changes,
242 which can help mitigate performance regression 242 which can help mitigate performance regressions. It also acts as a common
243 benchmarking framework, enabling developers to 243 benchmarking framework, enabling developers to easily create test cases,
244 integrate transparently, and use performance-r 244 integrate transparently, and use performance-rich tooling.
245 245
246 "perf bench all" command runs the following be 246 "perf bench all" command runs the following benchmarks:
247 247
248 * sched/messaging 248 * sched/messaging
249 * sched/pipe 249 * sched/pipe
250 * syscall/basic 250 * syscall/basic
251 * mem/memcpy 251 * mem/memcpy
252 * mem/memset 252 * mem/memset
253 253
254 What is stress-ng and how do we use it? 254 What is stress-ng and how do we use it?
255 ======================================= 255 =======================================
256 256
257 As mentioned earlier, stress-ng is used for pe 257 As mentioned earlier, stress-ng is used for performing stress testing on
258 the kernel. It allows you to exercise various 258 the kernel. It allows you to exercise various physical subsystems of the
259 computer, as well as interfaces of the OS kern 259 computer, as well as interfaces of the OS kernel, using stressor-s. They
260 are available for CPU, CPU cache, devices, I/O 260 are available for CPU, CPU cache, devices, I/O, interrupts, file system,
261 memory, network, operating system, pipelines, 261 memory, network, operating system, pipelines, schedulers, and virtual
262 machines. 262 machines.
263 263
264 The netdev stressor starts N workers that exer 264 The netdev stressor starts N workers that exercise various netdevice ioctl
265 commands across all the available network devi 265 commands across all the available network devices. The following ioctls are
266 exercised: 266 exercised:
267 267
268 * SIOCGIFCONF, SIOCGIFINDEX, SIOCGIFNAME, SIO 268 * SIOCGIFCONF, SIOCGIFINDEX, SIOCGIFNAME, SIOCGIFFLAGS
269 * SIOCGIFADDR, SIOCGIFNETMASK, SIOCGIFMETRIC, 269 * SIOCGIFADDR, SIOCGIFNETMASK, SIOCGIFMETRIC, SIOCGIFMTU
270 * SIOCGIFHWADDR, SIOCGIFMAP, SIOCGIFTXQLEN 270 * SIOCGIFHWADDR, SIOCGIFMAP, SIOCGIFTXQLEN
271 271
272 The following command runs the stressor:: 272 The following command runs the stressor::
273 273
274 stress-ng --netdev 1 -t 60 --metrics command 274 stress-ng --netdev 1 -t 60 --metrics command.
275 275
276 We can use the perf record command to record t 276 We can use the perf record command to record the events and information
277 associated with a process. This command record 277 associated with a process. This command records the profiling data in the
278 perf.data file in the same directory. 278 perf.data file in the same directory.
279 279
280 Using the following commands you can record th 280 Using the following commands you can record the events associated with the
281 netdev stressor, view the generated report per 281 netdev stressor, view the generated report perf.data and annotate the to
282 view the statistics of each instruction of the 282 view the statistics of each instruction of the program::
283 283
284 perf record stress-ng --netdev 1 -t 60 --met 284 perf record stress-ng --netdev 1 -t 60 --metrics command.
285 perf report 285 perf report
286 perf annotate 286 perf annotate
287 287
288 What is paxtest and how do we use it? 288 What is paxtest and how do we use it?
289 ===================================== 289 =====================================
290 290
291 paxtest is a program that tests buffer overflo 291 paxtest is a program that tests buffer overflows in the kernel. It tests
292 kernel enforcements over memory usage. General 292 kernel enforcements over memory usage. Generally, execution in some memory
293 segments makes buffer overflows possible. It r 293 segments makes buffer overflows possible. It runs a set of programs that
294 attempt to subvert memory usage. It is used as 294 attempt to subvert memory usage. It is used as a regression test suite for
295 PaX, and will be useful to test other memory p 295 PaX, and will be useful to test other memory protection patches for the
296 kernel. 296 kernel.
297 297
298 paxtest provides kiddie and blackhat modes. Th 298 paxtest provides kiddie and blackhat modes. The paxtest kiddie mode runs
299 in normal mode, whereas the blackhat mode trie 299 in normal mode, whereas the blackhat mode tries to get around the protection
300 of the kernel testing for vulnerabilities. We 300 of the kernel testing for vulnerabilities. We focus on the kiddie mode here
301 and combine "paxtest kiddie" run with "perf re 301 and combine "paxtest kiddie" run with "perf record" to collect CPU stack
302 traces for the paxtest kiddie run to see which 302 traces for the paxtest kiddie run to see which function is calling other
303 functions in the performance profile. Then the 303 functions in the performance profile. Then the "dwarf" (DWARF's Call Frame
304 Information) mode can be used to unwind the st 304 Information) mode can be used to unwind the stack.
305 305
306 The following command can be used to view resu 306 The following command can be used to view resulting report in call-graph
307 format:: 307 format::
308 308
309 perf record --call-graph dwarf paxtest kiddi 309 perf record --call-graph dwarf paxtest kiddie
310 perf report --stdio 310 perf report --stdio
311 311
312 Tracing workloads 312 Tracing workloads
313 ================= 313 =================
314 314
315 Now that we understand the workloads, let's st 315 Now that we understand the workloads, let's start tracing them.
316 316
317 Tracing perf bench all workload 317 Tracing perf bench all workload
318 ------------------------------- 318 -------------------------------
319 319
320 Run the following command to trace perf bench 320 Run the following command to trace perf bench all workload::
321 321
322 strace -c perf bench all 322 strace -c perf bench all
323 323
324 **System Calls made by the workload** 324 **System Calls made by the workload**
325 325
326 The below table shows the system calls invoked 326 The below table shows the system calls invoked by the workload, number of
327 times each system call is invoked, and the cor 327 times each system call is invoked, and the corresponding Linux subsystem.
328 328
329 +-------------------+-----------+------------- 329 +-------------------+-----------+-----------------+-------------------------+
330 | System Call | # calls | Linux Subsys 330 | System Call | # calls | Linux Subsystem | System Call (API) |
331 +===================+===========+============= 331 +===================+===========+=================+=========================+
332 | getppid | 10000001 | Process Mgmt 332 | getppid | 10000001 | Process Mgmt | sys_getpid() |
333 +-------------------+-----------+------------- 333 +-------------------+-----------+-----------------+-------------------------+
334 | clone | 1077 | Process Mgmt 334 | clone | 1077 | Process Mgmt. | sys_clone() |
335 +-------------------+-----------+------------- 335 +-------------------+-----------+-----------------+-------------------------+
336 | prctl | 23 | Process Mgmt 336 | prctl | 23 | Process Mgmt. | sys_prctl() |
337 +-------------------+-----------+------------- 337 +-------------------+-----------+-----------------+-------------------------+
338 | prlimit64 | 7 | Process Mgmt 338 | prlimit64 | 7 | Process Mgmt. | sys_prlimit64() |
339 +-------------------+-----------+------------- 339 +-------------------+-----------+-----------------+-------------------------+
340 | getpid | 10 | Process Mgmt 340 | getpid | 10 | Process Mgmt. | sys_getpid() |
341 +-------------------+-----------+------------- 341 +-------------------+-----------+-----------------+-------------------------+
342 | uname | 3 | Process Mgmt 342 | uname | 3 | Process Mgmt. | sys_uname() |
343 +-------------------+-----------+------------- 343 +-------------------+-----------+-----------------+-------------------------+
344 | sysinfo | 1 | Process Mgmt 344 | sysinfo | 1 | Process Mgmt. | sys_sysinfo() |
345 +-------------------+-----------+------------- 345 +-------------------+-----------+-----------------+-------------------------+
346 | getuid | 1 | Process Mgmt 346 | getuid | 1 | Process Mgmt. | sys_getuid() |
347 +-------------------+-----------+------------- 347 +-------------------+-----------+-----------------+-------------------------+
348 | getgid | 1 | Process Mgmt 348 | getgid | 1 | Process Mgmt. | sys_getgid() |
349 +-------------------+-----------+------------- 349 +-------------------+-----------+-----------------+-------------------------+
350 | geteuid | 1 | Process Mgmt 350 | geteuid | 1 | Process Mgmt. | sys_geteuid() |
351 +-------------------+-----------+------------- 351 +-------------------+-----------+-----------------+-------------------------+
352 | getegid | 1 | Process Mgmt 352 | getegid | 1 | Process Mgmt. | sys_getegid |
353 +-------------------+-----------+------------- 353 +-------------------+-----------+-----------------+-------------------------+
354 | close | 49951 | Filesystem 354 | close | 49951 | Filesystem | sys_close() |
355 +-------------------+-----------+------------- 355 +-------------------+-----------+-----------------+-------------------------+
356 | pipe | 604 | Filesystem 356 | pipe | 604 | Filesystem | sys_pipe() |
357 +-------------------+-----------+------------- 357 +-------------------+-----------+-----------------+-------------------------+
358 | openat | 48560 | Filesystem 358 | openat | 48560 | Filesystem | sys_opennat() |
359 +-------------------+-----------+------------- 359 +-------------------+-----------+-----------------+-------------------------+
360 | fstat | 8338 | Filesystem 360 | fstat | 8338 | Filesystem | sys_fstat() |
361 +-------------------+-----------+------------- 361 +-------------------+-----------+-----------------+-------------------------+
362 | stat | 1573 | Filesystem 362 | stat | 1573 | Filesystem | sys_stat() |
363 +-------------------+-----------+------------- 363 +-------------------+-----------+-----------------+-------------------------+
364 | pread64 | 9646 | Filesystem 364 | pread64 | 9646 | Filesystem | sys_pread64() |
365 +-------------------+-----------+------------- 365 +-------------------+-----------+-----------------+-------------------------+
366 | getdents64 | 1873 | Filesystem 366 | getdents64 | 1873 | Filesystem | sys_getdents64() |
367 +-------------------+-----------+------------- 367 +-------------------+-----------+-----------------+-------------------------+
368 | access | 3 | Filesystem 368 | access | 3 | Filesystem | sys_access() |
369 +-------------------+-----------+------------- 369 +-------------------+-----------+-----------------+-------------------------+
370 | lstat | 1880 | Filesystem 370 | lstat | 1880 | Filesystem | sys_lstat() |
371 +-------------------+-----------+------------- 371 +-------------------+-----------+-----------------+-------------------------+
372 | lseek | 6 | Filesystem 372 | lseek | 6 | Filesystem | sys_lseek() |
373 +-------------------+-----------+------------- 373 +-------------------+-----------+-----------------+-------------------------+
374 | ioctl | 3 | Filesystem 374 | ioctl | 3 | Filesystem | sys_ioctl() |
375 +-------------------+-----------+------------- 375 +-------------------+-----------+-----------------+-------------------------+
376 | dup2 | 1 | Filesystem 376 | dup2 | 1 | Filesystem | sys_dup2() |
377 +-------------------+-----------+------------- 377 +-------------------+-----------+-----------------+-------------------------+
378 | execve | 2 | Filesystem 378 | execve | 2 | Filesystem | sys_execve() |
379 +-------------------+-----------+------------- 379 +-------------------+-----------+-----------------+-------------------------+
380 | fcntl | 8779 | Filesystem 380 | fcntl | 8779 | Filesystem | sys_fcntl() |
381 +-------------------+-----------+------------- 381 +-------------------+-----------+-----------------+-------------------------+
382 | statfs | 1 | Filesystem 382 | statfs | 1 | Filesystem | sys_statfs() |
383 +-------------------+-----------+------------- 383 +-------------------+-----------+-----------------+-------------------------+
384 | epoll_create | 2 | Filesystem 384 | epoll_create | 2 | Filesystem | sys_epoll_create() |
385 +-------------------+-----------+------------- 385 +-------------------+-----------+-----------------+-------------------------+
386 | epoll_ctl | 64 | Filesystem 386 | epoll_ctl | 64 | Filesystem | sys_epoll_ctl() |
387 +-------------------+-----------+------------- 387 +-------------------+-----------+-----------------+-------------------------+
388 | newfstatat | 8318 | Filesystem 388 | newfstatat | 8318 | Filesystem | sys_newfstatat() |
389 +-------------------+-----------+------------- 389 +-------------------+-----------+-----------------+-------------------------+
390 | eventfd2 | 192 | Filesystem 390 | eventfd2 | 192 | Filesystem | sys_eventfd2() |
391 +-------------------+-----------+------------- 391 +-------------------+-----------+-----------------+-------------------------+
392 | mmap | 243 | Memory Mgmt. 392 | mmap | 243 | Memory Mgmt. | sys_mmap() |
393 +-------------------+-----------+------------- 393 +-------------------+-----------+-----------------+-------------------------+
394 | mprotect | 32 | Memory Mgmt. 394 | mprotect | 32 | Memory Mgmt. | sys_mprotect() |
395 +-------------------+-----------+------------- 395 +-------------------+-----------+-----------------+-------------------------+
396 | brk | 21 | Memory Mgmt. 396 | brk | 21 | Memory Mgmt. | sys_brk() |
397 +-------------------+-----------+------------- 397 +-------------------+-----------+-----------------+-------------------------+
398 | munmap | 128 | Memory Mgmt. 398 | munmap | 128 | Memory Mgmt. | sys_munmap() |
399 +-------------------+-----------+------------- 399 +-------------------+-----------+-----------------+-------------------------+
400 | set_mempolicy | 156 | Memory Mgmt. 400 | set_mempolicy | 156 | Memory Mgmt. | sys_set_mempolicy() |
401 +-------------------+-----------+------------- 401 +-------------------+-----------+-----------------+-------------------------+
402 | set_tid_address | 1 | Process Mgmt 402 | set_tid_address | 1 | Process Mgmt. | sys_set_tid_address() |
403 +-------------------+-----------+------------- 403 +-------------------+-----------+-----------------+-------------------------+
404 | set_robust_list | 1 | Futex 404 | set_robust_list | 1 | Futex | sys_set_robust_list() |
405 +-------------------+-----------+------------- 405 +-------------------+-----------+-----------------+-------------------------+
406 | futex | 341 | Futex 406 | futex | 341 | Futex | sys_futex() |
407 +-------------------+-----------+------------- 407 +-------------------+-----------+-----------------+-------------------------+
408 | sched_getaffinity | 79 | Scheduler 408 | sched_getaffinity | 79 | Scheduler | sys_sched_getaffinity() |
409 +-------------------+-----------+------------- 409 +-------------------+-----------+-----------------+-------------------------+
410 | sched_setaffinity | 223 | Scheduler 410 | sched_setaffinity | 223 | Scheduler | sys_sched_setaffinity() |
411 +-------------------+-----------+------------- 411 +-------------------+-----------+-----------------+-------------------------+
412 | socketpair | 202 | Network 412 | socketpair | 202 | Network | sys_socketpair() |
413 +-------------------+-----------+------------- 413 +-------------------+-----------+-----------------+-------------------------+
414 | rt_sigprocmask | 21 | Signal 414 | rt_sigprocmask | 21 | Signal | sys_rt_sigprocmask() |
415 +-------------------+-----------+------------- 415 +-------------------+-----------+-----------------+-------------------------+
416 | rt_sigaction | 36 | Signal 416 | rt_sigaction | 36 | Signal | sys_rt_sigaction() |
417 +-------------------+-----------+------------- 417 +-------------------+-----------+-----------------+-------------------------+
418 | rt_sigreturn | 2 | Signal 418 | rt_sigreturn | 2 | Signal | sys_rt_sigreturn() |
419 +-------------------+-----------+------------- 419 +-------------------+-----------+-----------------+-------------------------+
420 | wait4 | 889 | Time 420 | wait4 | 889 | Time | sys_wait4() |
421 +-------------------+-----------+------------- 421 +-------------------+-----------+-----------------+-------------------------+
422 | clock_nanosleep | 37 | Time 422 | clock_nanosleep | 37 | Time | sys_clock_nanosleep() |
423 +-------------------+-----------+------------- 423 +-------------------+-----------+-----------------+-------------------------+
424 | capget | 4 | Capability 424 | capget | 4 | Capability | sys_capget() |
425 +-------------------+-----------+------------- 425 +-------------------+-----------+-----------------+-------------------------+
426 426
427 Tracing stress-ng netdev stressor workload 427 Tracing stress-ng netdev stressor workload
428 ------------------------------------------ 428 ------------------------------------------
429 429
430 Run the following command to trace stress-ng n 430 Run the following command to trace stress-ng netdev stressor workload::
431 431
432 strace -c stress-ng --netdev 1 -t 60 --metr 432 strace -c stress-ng --netdev 1 -t 60 --metrics
433 433
434 **System Calls made by the workload** 434 **System Calls made by the workload**
435 435
436 The below table shows the system calls invoked 436 The below table shows the system calls invoked by the workload, number of
437 times each system call is invoked, and the cor 437 times each system call is invoked, and the corresponding Linux subsystem.
438 438
439 +-------------------+-----------+------------- 439 +-------------------+-----------+-----------------+-------------------------+
440 | System Call | # calls | Linux Subsys 440 | System Call | # calls | Linux Subsystem | System Call (API) |
441 +===================+===========+============= 441 +===================+===========+=================+=========================+
442 | openat | 74 | Filesystem 442 | openat | 74 | Filesystem | sys_openat() |
443 +-------------------+-----------+------------- 443 +-------------------+-----------+-----------------+-------------------------+
444 | close | 75 | Filesystem 444 | close | 75 | Filesystem | sys_close() |
445 +-------------------+-----------+------------- 445 +-------------------+-----------+-----------------+-------------------------+
446 | read | 58 | Filesystem 446 | read | 58 | Filesystem | sys_read() |
447 +-------------------+-----------+------------- 447 +-------------------+-----------+-----------------+-------------------------+
448 | fstat | 20 | Filesystem 448 | fstat | 20 | Filesystem | sys_fstat() |
449 +-------------------+-----------+------------- 449 +-------------------+-----------+-----------------+-------------------------+
450 | flock | 10 | Filesystem 450 | flock | 10 | Filesystem | sys_flock() |
451 +-------------------+-----------+------------- 451 +-------------------+-----------+-----------------+-------------------------+
452 | write | 7 | Filesystem 452 | write | 7 | Filesystem | sys_write() |
453 +-------------------+-----------+------------- 453 +-------------------+-----------+-----------------+-------------------------+
454 | getdents64 | 8 | Filesystem 454 | getdents64 | 8 | Filesystem | sys_getdents64() |
455 +-------------------+-----------+------------- 455 +-------------------+-----------+-----------------+-------------------------+
456 | pread64 | 8 | Filesystem 456 | pread64 | 8 | Filesystem | sys_pread64() |
457 +-------------------+-----------+------------- 457 +-------------------+-----------+-----------------+-------------------------+
458 | lseek | 1 | Filesystem 458 | lseek | 1 | Filesystem | sys_lseek() |
459 +-------------------+-----------+------------- 459 +-------------------+-----------+-----------------+-------------------------+
460 | access | 2 | Filesystem 460 | access | 2 | Filesystem | sys_access() |
461 +-------------------+-----------+------------- 461 +-------------------+-----------+-----------------+-------------------------+
462 | getcwd | 1 | Filesystem 462 | getcwd | 1 | Filesystem | sys_getcwd() |
463 +-------------------+-----------+------------- 463 +-------------------+-----------+-----------------+-------------------------+
464 | execve | 1 | Filesystem 464 | execve | 1 | Filesystem | sys_execve() |
465 +-------------------+-----------+------------- 465 +-------------------+-----------+-----------------+-------------------------+
466 | mmap | 61 | Memory Mgmt. 466 | mmap | 61 | Memory Mgmt. | sys_mmap() |
467 +-------------------+-----------+------------- 467 +-------------------+-----------+-----------------+-------------------------+
468 | munmap | 3 | Memory Mgmt. 468 | munmap | 3 | Memory Mgmt. | sys_munmap() |
469 +-------------------+-----------+------------- 469 +-------------------+-----------+-----------------+-------------------------+
470 | mprotect | 20 | Memory Mgmt. 470 | mprotect | 20 | Memory Mgmt. | sys_mprotect() |
471 +-------------------+-----------+------------- 471 +-------------------+-----------+-----------------+-------------------------+
472 | mlock | 2 | Memory Mgmt. 472 | mlock | 2 | Memory Mgmt. | sys_mlock() |
473 +-------------------+-----------+------------- 473 +-------------------+-----------+-----------------+-------------------------+
474 | brk | 3 | Memory Mgmt. 474 | brk | 3 | Memory Mgmt. | sys_brk() |
475 +-------------------+-----------+------------- 475 +-------------------+-----------+-----------------+-------------------------+
476 | rt_sigaction | 21 | Signal 476 | rt_sigaction | 21 | Signal | sys_rt_sigaction() |
477 +-------------------+-----------+------------- 477 +-------------------+-----------+-----------------+-------------------------+
478 | rt_sigprocmask | 1 | Signal 478 | rt_sigprocmask | 1 | Signal | sys_rt_sigprocmask() |
479 +-------------------+-----------+------------- 479 +-------------------+-----------+-----------------+-------------------------+
480 | sigaltstack | 1 | Signal 480 | sigaltstack | 1 | Signal | sys_sigaltstack() |
481 +-------------------+-----------+------------- 481 +-------------------+-----------+-----------------+-------------------------+
482 | rt_sigreturn | 1 | Signal 482 | rt_sigreturn | 1 | Signal | sys_rt_sigreturn() |
483 +-------------------+-----------+------------- 483 +-------------------+-----------+-----------------+-------------------------+
484 | getpid | 8 | Process Mgmt 484 | getpid | 8 | Process Mgmt. | sys_getpid() |
485 +-------------------+-----------+------------- 485 +-------------------+-----------+-----------------+-------------------------+
486 | prlimit64 | 5 | Process Mgmt 486 | prlimit64 | 5 | Process Mgmt. | sys_prlimit64() |
487 +-------------------+-----------+------------- 487 +-------------------+-----------+-----------------+-------------------------+
488 | arch_prctl | 2 | Process Mgmt 488 | arch_prctl | 2 | Process Mgmt. | sys_arch_prctl() |
489 +-------------------+-----------+------------- 489 +-------------------+-----------+-----------------+-------------------------+
490 | sysinfo | 2 | Process Mgmt 490 | sysinfo | 2 | Process Mgmt. | sys_sysinfo() |
491 +-------------------+-----------+------------- 491 +-------------------+-----------+-----------------+-------------------------+
492 | getuid | 2 | Process Mgmt 492 | getuid | 2 | Process Mgmt. | sys_getuid() |
493 +-------------------+-----------+------------- 493 +-------------------+-----------+-----------------+-------------------------+
494 | uname | 1 | Process Mgmt 494 | uname | 1 | Process Mgmt. | sys_uname() |
495 +-------------------+-----------+------------- 495 +-------------------+-----------+-----------------+-------------------------+
496 | setpgid | 1 | Process Mgmt 496 | setpgid | 1 | Process Mgmt. | sys_setpgid() |
497 +-------------------+-----------+------------- 497 +-------------------+-----------+-----------------+-------------------------+
498 | getrusage | 1 | Process Mgmt 498 | getrusage | 1 | Process Mgmt. | sys_getrusage() |
499 +-------------------+-----------+------------- 499 +-------------------+-----------+-----------------+-------------------------+
500 | geteuid | 1 | Process Mgmt 500 | geteuid | 1 | Process Mgmt. | sys_geteuid() |
501 +-------------------+-----------+------------- 501 +-------------------+-----------+-----------------+-------------------------+
502 | getppid | 1 | Process Mgmt 502 | getppid | 1 | Process Mgmt. | sys_getppid() |
503 +-------------------+-----------+------------- 503 +-------------------+-----------+-----------------+-------------------------+
504 | sendto | 3 | Network 504 | sendto | 3 | Network | sys_sendto() |
505 +-------------------+-----------+------------- 505 +-------------------+-----------+-----------------+-------------------------+
506 | connect | 1 | Network 506 | connect | 1 | Network | sys_connect() |
507 +-------------------+-----------+------------- 507 +-------------------+-----------+-----------------+-------------------------+
508 | socket | 1 | Network 508 | socket | 1 | Network | sys_socket() |
509 +-------------------+-----------+------------- 509 +-------------------+-----------+-----------------+-------------------------+
510 | clone | 1 | Process Mgmt 510 | clone | 1 | Process Mgmt. | sys_clone() |
511 +-------------------+-----------+------------- 511 +-------------------+-----------+-----------------+-------------------------+
512 | set_tid_address | 1 | Process Mgmt 512 | set_tid_address | 1 | Process Mgmt. | sys_set_tid_address() |
513 +-------------------+-----------+------------- 513 +-------------------+-----------+-----------------+-------------------------+
514 | wait4 | 2 | Time 514 | wait4 | 2 | Time | sys_wait4() |
515 +-------------------+-----------+------------- 515 +-------------------+-----------+-----------------+-------------------------+
516 | alarm | 1 | Time 516 | alarm | 1 | Time | sys_alarm() |
517 +-------------------+-----------+------------- 517 +-------------------+-----------+-----------------+-------------------------+
518 | set_robust_list | 1 | Futex 518 | set_robust_list | 1 | Futex | sys_set_robust_list() |
519 +-------------------+-----------+------------- 519 +-------------------+-----------+-----------------+-------------------------+
520 520
521 Tracing paxtest kiddie workload 521 Tracing paxtest kiddie workload
522 ------------------------------- 522 -------------------------------
523 523
524 Run the following command to trace paxtest kid 524 Run the following command to trace paxtest kiddie workload::
525 525
526 strace -c paxtest kiddie 526 strace -c paxtest kiddie
527 527
528 **System Calls made by the workload** 528 **System Calls made by the workload**
529 529
530 The below table shows the system calls invoked 530 The below table shows the system calls invoked by the workload, number of
531 times each system call is invoked, and the cor 531 times each system call is invoked, and the corresponding Linux subsystem.
532 532
533 +-------------------+-----------+------------- 533 +-------------------+-----------+-----------------+----------------------+
534 | System Call | # calls | Linux Subsys 534 | System Call | # calls | Linux Subsystem | System Call (API) |
535 +===================+===========+============= 535 +===================+===========+=================+======================+
536 | read | 3 | Filesystem 536 | read | 3 | Filesystem | sys_read() |
537 +-------------------+-----------+------------- 537 +-------------------+-----------+-----------------+----------------------+
538 | write | 11 | Filesystem 538 | write | 11 | Filesystem | sys_write() |
539 +-------------------+-----------+------------- 539 +-------------------+-----------+-----------------+----------------------+
540 | close | 41 | Filesystem 540 | close | 41 | Filesystem | sys_close() |
541 +-------------------+-----------+------------- 541 +-------------------+-----------+-----------------+----------------------+
542 | stat | 24 | Filesystem 542 | stat | 24 | Filesystem | sys_stat() |
543 +-------------------+-----------+------------- 543 +-------------------+-----------+-----------------+----------------------+
544 | fstat | 2 | Filesystem 544 | fstat | 2 | Filesystem | sys_fstat() |
545 +-------------------+-----------+------------- 545 +-------------------+-----------+-----------------+----------------------+
546 | pread64 | 6 | Filesystem 546 | pread64 | 6 | Filesystem | sys_pread64() |
547 +-------------------+-----------+------------- 547 +-------------------+-----------+-----------------+----------------------+
548 | access | 1 | Filesystem 548 | access | 1 | Filesystem | sys_access() |
549 +-------------------+-----------+------------- 549 +-------------------+-----------+-----------------+----------------------+
550 | pipe | 1 | Filesystem 550 | pipe | 1 | Filesystem | sys_pipe() |
551 +-------------------+-----------+------------- 551 +-------------------+-----------+-----------------+----------------------+
552 | dup2 | 24 | Filesystem 552 | dup2 | 24 | Filesystem | sys_dup2() |
553 +-------------------+-----------+------------- 553 +-------------------+-----------+-----------------+----------------------+
554 | execve | 1 | Filesystem 554 | execve | 1 | Filesystem | sys_execve() |
555 +-------------------+-----------+------------- 555 +-------------------+-----------+-----------------+----------------------+
556 | fcntl | 26 | Filesystem 556 | fcntl | 26 | Filesystem | sys_fcntl() |
557 +-------------------+-----------+------------- 557 +-------------------+-----------+-----------------+----------------------+
558 | openat | 14 | Filesystem 558 | openat | 14 | Filesystem | sys_openat() |
559 +-------------------+-----------+------------- 559 +-------------------+-----------+-----------------+----------------------+
560 | rt_sigaction | 7 | Signal 560 | rt_sigaction | 7 | Signal | sys_rt_sigaction() |
561 +-------------------+-----------+------------- 561 +-------------------+-----------+-----------------+----------------------+
562 | rt_sigreturn | 38 | Signal 562 | rt_sigreturn | 38 | Signal | sys_rt_sigreturn() |
563 +-------------------+-----------+------------- 563 +-------------------+-----------+-----------------+----------------------+
564 | clone | 38 | Process Mgmt 564 | clone | 38 | Process Mgmt. | sys_clone() |
565 +-------------------+-----------+------------- 565 +-------------------+-----------+-----------------+----------------------+
566 | wait4 | 44 | Time 566 | wait4 | 44 | Time | sys_wait4() |
567 +-------------------+-----------+------------- 567 +-------------------+-----------+-----------------+----------------------+
568 | mmap | 7 | Memory Mgmt. 568 | mmap | 7 | Memory Mgmt. | sys_mmap() |
569 +-------------------+-----------+------------- 569 +-------------------+-----------+-----------------+----------------------+
570 | mprotect | 3 | Memory Mgmt. 570 | mprotect | 3 | Memory Mgmt. | sys_mprotect() |
571 +-------------------+-----------+------------- 571 +-------------------+-----------+-----------------+----------------------+
572 | munmap | 1 | Memory Mgmt. 572 | munmap | 1 | Memory Mgmt. | sys_munmap() |
573 +-------------------+-----------+------------- 573 +-------------------+-----------+-----------------+----------------------+
574 | brk | 3 | Memory Mgmt. 574 | brk | 3 | Memory Mgmt. | sys_brk() |
575 +-------------------+-----------+------------- 575 +-------------------+-----------+-----------------+----------------------+
576 | getpid | 1 | Process Mgmt 576 | getpid | 1 | Process Mgmt. | sys_getpid() |
577 +-------------------+-----------+------------- 577 +-------------------+-----------+-----------------+----------------------+
578 | getuid | 1 | Process Mgmt 578 | getuid | 1 | Process Mgmt. | sys_getuid() |
579 +-------------------+-----------+------------- 579 +-------------------+-----------+-----------------+----------------------+
580 | getgid | 1 | Process Mgmt 580 | getgid | 1 | Process Mgmt. | sys_getgid() |
581 +-------------------+-----------+------------- 581 +-------------------+-----------+-----------------+----------------------+
582 | geteuid | 2 | Process Mgmt 582 | geteuid | 2 | Process Mgmt. | sys_geteuid() |
583 +-------------------+-----------+------------- 583 +-------------------+-----------+-----------------+----------------------+
584 | getegid | 1 | Process Mgmt 584 | getegid | 1 | Process Mgmt. | sys_getegid() |
585 +-------------------+-----------+------------- 585 +-------------------+-----------+-----------------+----------------------+
586 | getppid | 1 | Process Mgmt 586 | getppid | 1 | Process Mgmt. | sys_getppid() |
587 +-------------------+-----------+------------- 587 +-------------------+-----------+-----------------+----------------------+
588 | arch_prctl | 2 | Process Mgmt 588 | arch_prctl | 2 | Process Mgmt. | sys_arch_prctl() |
589 +-------------------+-----------+------------- 589 +-------------------+-----------+-----------------+----------------------+
590 590
591 Conclusion 591 Conclusion
592 ========== 592 ==========
593 593
594 This document is intended to be used as a guid 594 This document is intended to be used as a guide on how to gather fine-grained
595 information on the resources in use by workloa 595 information on the resources in use by workloads using strace.
596 596
597 References 597 References
598 ========== 598 ==========
599 599
600 * `Discovery Linux Kernel Subsystems used by 600 * `Discovery Linux Kernel Subsystems used by OpenAPS <https://elisa.tech/blog/2022/02/02/discovery-linux-kernel-subsystems-used-by-openaps>`_
601 * `ELISA-White-Papers-Discovering Linux kerne 601 * `ELISA-White-Papers-Discovering Linux kernel subsystems used by a workload <https://github.com/elisa-tech/ELISA-White-Papers/blob/master/Processes/Discovering_Linux_kernel_subsystems_used_by_a_workload.md>`_
602 * `strace <https://man7.org/linux/man-pages/m 602 * `strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_
603 * `perf <https://man7.org/linux/man-pages/man 603 * `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_
604 * `paxtest README <https://github.com/opntr/p 604 * `paxtest README <https://github.com/opntr/paxtest-freebsd/blob/hardenedbsd/0.9.14-hbsd/README>`_
605 * `stress-ng <https://www.mankier.com/1/stres 605 * `stress-ng <https://www.mankier.com/1/stress-ng>`_
606 * `Monitoring and managing system status and 606 * `Monitoring and managing system status and performance <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/index>`_
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.