1 .. _perf_security: 1 .. _perf_security:
2 2
3 Perf events and tool security !! 3 Perf Events and tool security
4 ============================= 4 =============================
5 5
6 Overview 6 Overview
7 -------- 7 --------
8 8
9 Usage of Performance Counters for Linux (perf_ 9 Usage of Performance Counters for Linux (perf_events) [1]_ , [2]_ , [3]_
10 can impose a considerable risk of leaking sens 10 can impose a considerable risk of leaking sensitive data accessed by
11 monitored processes. The data leakage is possi 11 monitored processes. The data leakage is possible both in scenarios of
12 direct usage of perf_events system call API [2 12 direct usage of perf_events system call API [2]_ and over data files
13 generated by Perf tool user mode utility (Perf 13 generated by Perf tool user mode utility (Perf) [3]_ , [4]_ . The risk
14 depends on the nature of data that perf_events 14 depends on the nature of data that perf_events performance monitoring
15 units (PMU) [2]_ and Perf collect and expose f 15 units (PMU) [2]_ and Perf collect and expose for performance analysis.
16 Collected system and performance data may be s 16 Collected system and performance data may be split into several
17 categories: 17 categories:
18 18
19 1. System hardware and software configuration 19 1. System hardware and software configuration data, for example: a CPU
20 model and its cache configuration, an amoun 20 model and its cache configuration, an amount of available memory and
21 its topology, used kernel and Perf versions 21 its topology, used kernel and Perf versions, performance monitoring
22 setup including experiment time, events con 22 setup including experiment time, events configuration, Perf command
23 line parameters, etc. 23 line parameters, etc.
24 24
25 2. User and kernel module paths and their load 25 2. User and kernel module paths and their load addresses with sizes,
26 process and thread names with their PIDs an 26 process and thread names with their PIDs and TIDs, timestamps for
27 captured hardware and software events. 27 captured hardware and software events.
28 28
29 3. Content of kernel software counters (e.g., 29 3. Content of kernel software counters (e.g., for context switches, page
30 faults, CPU migrations), architectural hard 30 faults, CPU migrations), architectural hardware performance counters
31 (PMC) [8]_ and machine specific registers ( 31 (PMC) [8]_ and machine specific registers (MSR) [9]_ that provide
32 execution metrics for various monitored par 32 execution metrics for various monitored parts of the system (e.g.,
33 memory controller (IMC), interconnect (QPI/ 33 memory controller (IMC), interconnect (QPI/UPI) or peripheral (PCIe)
34 uncore counters) without direct attribution 34 uncore counters) without direct attribution to any execution context
35 state. 35 state.
36 36
37 4. Content of architectural execution context 37 4. Content of architectural execution context registers (e.g., RIP, RSP,
38 RBP on x86_64), process user and kernel spa 38 RBP on x86_64), process user and kernel space memory addresses and
39 data, content of various architectural MSRs 39 data, content of various architectural MSRs that capture data from
40 this category. 40 this category.
41 41
42 Data that belong to the fourth category can po 42 Data that belong to the fourth category can potentially contain
43 sensitive process data. If PMUs in some monito 43 sensitive process data. If PMUs in some monitoring modes capture values
44 of execution context registers or data from pr 44 of execution context registers or data from process memory then access
45 to such monitoring modes requires to be ordere !! 45 to such monitoring capabilities requires to be ordered and secured
46 So, perf_events performance monitoring and obs !! 46 properly. So, perf_events/Perf performance monitoring is the subject for
47 the subject for security access control manage !! 47 security access control management [5]_ .
48 48
49 perf_events access control !! 49 perf_events/Perf access control
50 ------------------------------- 50 -------------------------------
51 51
52 To perform security checks, the Linux implemen 52 To perform security checks, the Linux implementation splits processes
53 into two categories [6]_ : a) privileged proce 53 into two categories [6]_ : a) privileged processes (whose effective user
54 ID is 0, referred to as superuser or root), an 54 ID is 0, referred to as superuser or root), and b) unprivileged
55 processes (whose effective UID is nonzero). Pr 55 processes (whose effective UID is nonzero). Privileged processes bypass
56 all kernel security permission checks so perf_ 56 all kernel security permission checks so perf_events performance
57 monitoring is fully available to privileged pr 57 monitoring is fully available to privileged processes without access,
58 scope and resource restrictions. 58 scope and resource restrictions.
59 59
60 Unprivileged processes are subject to a full s 60 Unprivileged processes are subject to a full security permission check
61 based on the process's credentials [5]_ (usual 61 based on the process's credentials [5]_ (usually: effective UID,
62 effective GID, and supplementary group list). 62 effective GID, and supplementary group list).
63 63
64 Linux divides the privileges traditionally ass 64 Linux divides the privileges traditionally associated with superuser
65 into distinct units, known as capabilities [6] 65 into distinct units, known as capabilities [6]_ , which can be
66 independently enabled and disabled on per-thre 66 independently enabled and disabled on per-thread basis for processes and
67 files of unprivileged users. 67 files of unprivileged users.
68 68
69 Unprivileged processes with enabled CAP_PERFMO !! 69 Unprivileged processes with enabled CAP_SYS_ADMIN capability are treated
70 as privileged processes with respect to perf_e 70 as privileged processes with respect to perf_events performance
71 monitoring and observability operations, thus, !! 71 monitoring and bypass *scope* permissions checks in the kernel.
72 checks in the kernel. CAP_PERFMON implements t !! 72
73 privilege [13]_ (POSIX 1003.1e: 2.2.2.39) for !! 73 Unprivileged processes using perf_events system call API is also subject
74 observability operations in the kernel and pro !! 74 for PTRACE_MODE_READ_REALCREDS ptrace access mode check [7]_ , whose
75 performance monitoring and observability in th !! 75 outcome determines whether monitoring is permitted. So unprivileged
76 !! 76 processes provided with CAP_SYS_PTRACE capability are effectively
77 For backward compatibility reasons the access !! 77 permitted to pass the check.
78 observability operations is also open for CAP_ <<
79 processes but CAP_SYS_ADMIN usage for secure m <<
80 use cases is discouraged with respect to the C <<
81 If system audit records [14]_ for a process us <<
82 API contain denial records of acquiring both C <<
83 capabilities then providing the process with C <<
84 is recommended as the preferred secure approac <<
85 denial logging related to usage of performance <<
86 <<
87 Prior Linux v5.9 unprivileged processes using <<
88 are also subject for PTRACE_MODE_READ_REALCRED <<
89 [7]_ , whose outcome determines whether monito <<
90 So unprivileged processes provided with CAP_SY <<
91 effectively permitted to pass the check. Start <<
92 CAP_SYS_PTRACE capability is not required and <<
93 be provided for processes to make performance <<
94 operations. <<
95 78
96 Other capabilities being granted to unprivileg 79 Other capabilities being granted to unprivileged processes can
97 effectively enable capturing of additional dat 80 effectively enable capturing of additional data required for later
98 performance analysis of monitored processes or 81 performance analysis of monitored processes or a system. For example,
99 CAP_SYSLOG capability permits reading kernel s 82 CAP_SYSLOG capability permits reading kernel space memory addresses from
100 /proc/kallsyms file. 83 /proc/kallsyms file.
101 84
102 Privileged Perf users groups !! 85 perf_events/Perf privileged users
103 --------------------------------- 86 ---------------------------------
104 87
105 Mechanisms of capabilities, privileged capabil !! 88 Mechanisms of capabilities, privileged capability-dumb files [6]_ and
106 file system ACLs [10]_ and sudo [15]_ utility !! 89 file system ACLs [10]_ can be used to create a dedicated group of
107 dedicated groups of privileged Perf users who !! 90 perf_events/Perf privileged users who are permitted to execute
108 performance monitoring and observability witho !! 91 performance monitoring without scope limits. The following steps can be
109 steps can be taken to create such groups of pr !! 92 taken to create such a group of privileged Perf users.
110 93
111 1. Create perf_users group of privileged Perf 94 1. Create perf_users group of privileged Perf users, assign perf_users
112 group to Perf tool executable and limit acc 95 group to Perf tool executable and limit access to the executable for
113 other users in the system who are not in th 96 other users in the system who are not in the perf_users group:
114 97
115 :: 98 ::
116 99
117 # groupadd perf_users 100 # groupadd perf_users
118 # ls -alhF 101 # ls -alhF
119 -rwxr-xr-x 2 root root 11M Oct 19 15:12 p 102 -rwxr-xr-x 2 root root 11M Oct 19 15:12 perf
120 # chgrp perf_users perf 103 # chgrp perf_users perf
121 # ls -alhF 104 # ls -alhF
122 -rwxr-xr-x 2 root perf_users 11M Oct 19 1 105 -rwxr-xr-x 2 root perf_users 11M Oct 19 15:12 perf
123 # chmod o-rwx perf 106 # chmod o-rwx perf
124 # ls -alhF 107 # ls -alhF
125 -rwxr-x--- 2 root perf_users 11M Oct 19 1 108 -rwxr-x--- 2 root perf_users 11M Oct 19 15:12 perf
126 109
127 2. Assign the required capabilities to the Per 110 2. Assign the required capabilities to the Perf tool executable file and
128 enable members of perf_users group with mon !! 111 enable members of perf_users group with performance monitoring
129 privileges [6]_ : 112 privileges [6]_ :
130 113
131 :: 114 ::
132 115
133 # setcap "cap_perfmon,cap_sys_ptrace,cap_sy !! 116 # setcap "cap_sys_admin,cap_sys_ptrace,cap_syslog=ep" perf
134 # setcap -v "cap_perfmon,cap_sys_ptrace,cap !! 117 # setcap -v "cap_sys_admin,cap_sys_ptrace,cap_syslog=ep" perf
135 perf: OK 118 perf: OK
136 # getcap perf 119 # getcap perf
137 perf = cap_sys_ptrace,cap_syslog,cap_perfmo !! 120 perf = cap_sys_ptrace,cap_sys_admin,cap_syslog+ep
138 <<
139 If the libcap [16]_ installed doesn't yet supp <<
140 i.e.: <<
141 <<
142 :: <<
143 <<
144 # setcap "38,cap_ipc_lock,cap_sys_ptrace,ca <<
145 <<
146 Note that you may need to have 'cap_ipc_lock' <<
147 'perf top', alternatively use 'perf top -m N', <<
148 it uses for the perf ring buffer, see the memo <<
149 <<
150 Using a libcap without support for CAP_PERFMON <<
151 CAP_EFFECTIVE, &val) fail, which will lead the <<
152 so as a workaround explicitly ask for the 'cyc <<
153 <<
154 :: <<
155 <<
156 # perf top -e cycles <<
157 <<
158 To get kernel and user samples with a perf bin <<
159 121
160 As a result, members of perf_users group are c 122 As a result, members of perf_users group are capable of conducting
161 performance monitoring and observability by us !! 123 performance monitoring by using functionality of the configured Perf
162 configured Perf tool executable that, when exe !! 124 tool executable that, when executes, passes perf_events subsystem scope
163 subsystem scope checks. !! 125 checks.
164 <<
165 In case Perf tool executable can't be assigned <<
166 file system is mounted with nosuid option or e <<
167 not supported by the file system) then creatio <<
168 privileged environment, naturally shell, is po <<
169 inherent processes with CAP_PERFMON and other <<
170 performance monitoring and observability opera <<
171 environment without limits. Access to the envi <<
172 utility for members of perf_users group only. <<
173 environment: <<
174 <<
175 1. Create shell script that uses capsh utility <<
176 and other required capabilities into ambien <<
177 process, lock the process security bits aft <<
178 SECBIT_NOROOT and SECBIT_NO_CAP_AMBIENT_RAI <<
179 the process identity to sudo caller of the <<
180 be a member of perf_users group: <<
181 <<
182 :: <<
183 <<
184 # ls -alh /usr/local/bin/perf.shell <<
185 -rwxr-xr-x. 1 root root 83 Oct 13 23:57 /us <<
186 # cat /usr/local/bin/perf.shell <<
187 exec /usr/sbin/capsh --iab=^cap_perfmon --s <<
188 <<
189 2. Extend sudo policy at /etc/sudoers file wit <<
190 <<
191 :: <<
192 <<
193 # grep perf_users /etc/sudoers <<
194 %perf_users ALL=/usr/local/bin/perf.shel <<
195 <<
196 3. Check that members of perf_users group have <<
197 shell and have CAP_PERFMON and other requir <<
198 in permitted, effective and ambient capabil <<
199 <<
200 :: <<
201 <<
202 $ id <<
203 uid=1003(capsh_test) gid=1004(capsh_test) gr <<
204 $ sudo perf.shell <<
205 [sudo] password for capsh_test: <<
206 $ grep Cap /proc/self/status <<
207 CapInh: 0000004000000000 <<
208 CapPrm: 0000004000000000 <<
209 CapEff: 0000004000000000 <<
210 CapBnd: 000000ffffffffff <<
211 CapAmb: 0000004000000000 <<
212 $ capsh --decode=0000004000000000 <<
213 0x0000004000000000=cap_perfmon <<
214 <<
215 As a result, members of perf_users group have <<
216 environment where they can use tools employing <<
217 governed by CAP_PERFMON Linux capability. <<
218 126
219 This specific access control management is onl 127 This specific access control management is only available to superuser
220 or root running processes with CAP_SETPCAP, CA 128 or root running processes with CAP_SETPCAP, CAP_SETFCAP [6]_
221 capabilities. 129 capabilities.
222 130
223 Unprivileged users !! 131 perf_events/Perf unprivileged users
224 ----------------------------------- 132 -----------------------------------
225 133
226 perf_events *scope* and *access* control for u !! 134 perf_events/Perf *scope* and *access* control for unprivileged processes
227 is governed by perf_event_paranoid [2]_ settin 135 is governed by perf_event_paranoid [2]_ setting:
228 136
229 -1: 137 -1:
230 Impose no *scope* and *access* restrictio 138 Impose no *scope* and *access* restrictions on using perf_events
231 performance monitoring. Per-user per-cpu 139 performance monitoring. Per-user per-cpu perf_event_mlock_kb [2]_
232 locking limit is ignored when allocating 140 locking limit is ignored when allocating memory buffers for storing
233 performance data. This is the least secur 141 performance data. This is the least secure mode since allowed
234 monitored *scope* is maximized and no per 142 monitored *scope* is maximized and no perf_events specific limits
235 are imposed on *resources* allocated for 143 are imposed on *resources* allocated for performance monitoring.
236 144
237 >=0: 145 >=0:
238 *scope* includes per-process and system w 146 *scope* includes per-process and system wide performance monitoring
239 but excludes raw tracepoints and ftrace f 147 but excludes raw tracepoints and ftrace function tracepoints
240 monitoring. CPU and system events happene 148 monitoring. CPU and system events happened when executing either in
241 user or in kernel space can be monitored 149 user or in kernel space can be monitored and captured for later
242 analysis. Per-user per-cpu perf_event_mlo 150 analysis. Per-user per-cpu perf_event_mlock_kb locking limit is
243 imposed but ignored for unprivileged proc 151 imposed but ignored for unprivileged processes with CAP_IPC_LOCK
244 [6]_ capability. 152 [6]_ capability.
245 153
246 >=1: 154 >=1:
247 *scope* includes per-process performance 155 *scope* includes per-process performance monitoring only and
248 excludes system wide performance monitori 156 excludes system wide performance monitoring. CPU and system events
249 happened when executing either in user or 157 happened when executing either in user or in kernel space can be
250 monitored and captured for later analysis 158 monitored and captured for later analysis. Per-user per-cpu
251 perf_event_mlock_kb locking limit is impo 159 perf_event_mlock_kb locking limit is imposed but ignored for
252 unprivileged processes with CAP_IPC_LOCK 160 unprivileged processes with CAP_IPC_LOCK capability.
253 161
254 >=2: 162 >=2:
255 *scope* includes per-process performance 163 *scope* includes per-process performance monitoring only. CPU and
256 system events happened when executing in 164 system events happened when executing in user space only can be
257 monitored and captured for later analysis 165 monitored and captured for later analysis. Per-user per-cpu
258 perf_event_mlock_kb locking limit is impo 166 perf_event_mlock_kb locking limit is imposed but ignored for
259 unprivileged processes with CAP_IPC_LOCK 167 unprivileged processes with CAP_IPC_LOCK capability.
260 168
261 Resource control !! 169 perf_events/Perf resource control
262 --------------------------------- 170 ---------------------------------
263 171
264 Open file descriptors 172 Open file descriptors
265 +++++++++++++++++++++ 173 +++++++++++++++++++++
266 174
267 The perf_events system call API [2]_ allocates 175 The perf_events system call API [2]_ allocates file descriptors for
268 every configured PMU event. Open file descript 176 every configured PMU event. Open file descriptors are a per-process
269 accountable resource governed by the RLIMIT_NO 177 accountable resource governed by the RLIMIT_NOFILE [11]_ limit
270 (ulimit -n), which is usually derived from the 178 (ulimit -n), which is usually derived from the login shell process. When
271 configuring Perf collection for a long list of 179 configuring Perf collection for a long list of events on a large server
272 system, this limit can be easily hit preventin 180 system, this limit can be easily hit preventing required monitoring
273 configuration. RLIMIT_NOFILE limit can be incr 181 configuration. RLIMIT_NOFILE limit can be increased on per-user basis
274 modifying content of the limits.conf file [12] 182 modifying content of the limits.conf file [12]_ . Ordinarily, a Perf
275 sampling session (perf record) requires an amo 183 sampling session (perf record) requires an amount of open perf_event
276 file descriptors that is not less than the num 184 file descriptors that is not less than the number of monitored events
277 multiplied by the number of monitored CPUs. 185 multiplied by the number of monitored CPUs.
278 186
279 Memory allocation 187 Memory allocation
280 +++++++++++++++++ 188 +++++++++++++++++
281 189
282 The amount of memory available to user process 190 The amount of memory available to user processes for capturing
283 performance monitoring data is governed by the 191 performance monitoring data is governed by the perf_event_mlock_kb [2]_
284 setting. This perf_event specific resource set 192 setting. This perf_event specific resource setting defines overall
285 per-cpu limits of memory allowed for mapping b 193 per-cpu limits of memory allowed for mapping by the user processes to
286 execute performance monitoring. The setting es 194 execute performance monitoring. The setting essentially extends the
287 RLIMIT_MEMLOCK [11]_ limit, but only for memor 195 RLIMIT_MEMLOCK [11]_ limit, but only for memory regions mapped
288 specifically for capturing monitored performan 196 specifically for capturing monitored performance events and related data.
289 197
290 For example, if a machine has eight cores and 198 For example, if a machine has eight cores and perf_event_mlock_kb limit
291 is set to 516 KiB, then a user process is prov 199 is set to 516 KiB, then a user process is provided with 516 KiB * 8 =
292 4128 KiB of memory above the RLIMIT_MEMLOCK li 200 4128 KiB of memory above the RLIMIT_MEMLOCK limit (ulimit -l) for
293 perf_event mmap buffers. In particular, this m 201 perf_event mmap buffers. In particular, this means that, if the user
294 wants to start two or more performance monitor 202 wants to start two or more performance monitoring processes, the user is
295 required to manually distribute the available 203 required to manually distribute the available 4128 KiB between the
296 monitoring processes, for example, using the - 204 monitoring processes, for example, using the --mmap-pages Perf record
297 mode option. Otherwise, the first started perf 205 mode option. Otherwise, the first started performance monitoring process
298 allocates all available 4128 KiB and the other 206 allocates all available 4128 KiB and the other processes will fail to
299 proceed due to the lack of memory. 207 proceed due to the lack of memory.
300 208
301 RLIMIT_MEMLOCK and perf_event_mlock_kb resourc 209 RLIMIT_MEMLOCK and perf_event_mlock_kb resource constraints are ignored
302 for processes with the CAP_IPC_LOCK capability 210 for processes with the CAP_IPC_LOCK capability. Thus, perf_events/Perf
303 privileged users can be provided with memory a 211 privileged users can be provided with memory above the constraints for
304 perf_events/Perf performance monitoring purpos 212 perf_events/Perf performance monitoring purpose by providing the Perf
305 executable with CAP_IPC_LOCK capability. 213 executable with CAP_IPC_LOCK capability.
306 214
307 Bibliography 215 Bibliography
308 ------------ 216 ------------
309 217
310 .. [1] `<https://lwn.net/Articles/337493/>`_ 218 .. [1] `<https://lwn.net/Articles/337493/>`_
311 .. [2] `<http://man7.org/linux/man-pages/man2/ 219 .. [2] `<http://man7.org/linux/man-pages/man2/perf_event_open.2.html>`_
312 .. [3] `<http://web.eece.maine.edu/~vweaver/pr 220 .. [3] `<http://web.eece.maine.edu/~vweaver/projects/perf_events/>`_
313 .. [4] `<https://perf.wiki.kernel.org/index.ph 221 .. [4] `<https://perf.wiki.kernel.org/index.php/Main_Page>`_
314 .. [5] `<https://www.kernel.org/doc/html/lates 222 .. [5] `<https://www.kernel.org/doc/html/latest/security/credentials.html>`_
315 .. [6] `<http://man7.org/linux/man-pages/man7/ 223 .. [6] `<http://man7.org/linux/man-pages/man7/capabilities.7.html>`_
316 .. [7] `<http://man7.org/linux/man-pages/man2/ 224 .. [7] `<http://man7.org/linux/man-pages/man2/ptrace.2.html>`_
317 .. [8] `<https://en.wikipedia.org/wiki/Hardwar 225 .. [8] `<https://en.wikipedia.org/wiki/Hardware_performance_counter>`_
318 .. [9] `<https://en.wikipedia.org/wiki/Model-s 226 .. [9] `<https://en.wikipedia.org/wiki/Model-specific_register>`_
319 .. [10] `<http://man7.org/linux/man-pages/man5 227 .. [10] `<http://man7.org/linux/man-pages/man5/acl.5.html>`_
320 .. [11] `<http://man7.org/linux/man-pages/man2 228 .. [11] `<http://man7.org/linux/man-pages/man2/getrlimit.2.html>`_
321 .. [12] `<http://man7.org/linux/man-pages/man5 229 .. [12] `<http://man7.org/linux/man-pages/man5/limits.conf.5.html>`_
322 .. [13] `<https://sites.google.com/site/fullyc !! 230
323 .. [14] `<http://man7.org/linux/man-pages/man8 <<
324 .. [15] `<https://man7.org/linux/man-pages/man <<
325 .. [16] `<https://git.kernel.org/pub/scm/libs/ <<
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.