1 ==============
2 OSNOISE Tracer
3 ==============
4
5 In the context of high-performance computing (
6 Noise (*osnoise*) refers to the interference e
7 due to activities inside the operating system.
8 NMIs, IRQs, SoftIRQs, and any other system thr
9 system. Moreover, hardware-related jobs can al
10 via SMIs.
11
12 hwlat_detector is one of the tools used to ide
13 source of noise: *hardware noise*.
14
15 In a nutshell, the hwlat_detector creates a th
16 periodically for a given period. At the beginn
17 disables interrupt and starts sampling. While
18 thread reads the time in a loop. As interrupts
19 IRQs, and SoftIRQs cannot interfere with the h
20 cause of any gap between two different reads o
21 NMI or in the hardware itself. At the end of t
22 interrupts and reports the max observed gap be
23 prints a NMI occurrence counter. If the output
24 executions, the user can conclude that the har
25 the latency. The hwlat detects the NMI executi
26 the entry and exit of a NMI.
27
28 The osnoise tracer leverages the hwlat_detecto
29 similar loop with preemption, SoftIRQs and IRQ
30 all the sources of *osnoise* during its execut
31 of hwlat, osnoise takes note of the entry and
32 source of interferences, increasing a per-cpu
33 osnoise tracer also saves an interference coun
34 interference. The interference counter for NMI
35 threads is increased anytime the tool observes
36 events. When a noise happens without any inter
37 system level, the hardware noise counter incre
38 hardware-related noise. In this way, osnoise c
39 source of interference. At the end of the peri
40 prints the sum of all noise, the max single no
41 available for the thread, and the counters for
42
43 Usage
44 -----
45
46 Write the ASCII text "osnoise" into the curren
47 tracing system (generally mounted at /sys/kern
48
49 For example::
50
51 [root@f32 ~]# cd /sys/kernel/tracing/
52 [root@f32 tracing]# echo osnoise > cur
53
54 It is possible to follow the trace by reading
55
56 [root@f32 tracing]# cat trace
57 # tracer: osnoise
58 #
59 # _----
60 # / _---
61 # | / _--
62 # || / _-
63 # || /
64 # ||||
65 # TASK-PID CPU# ||||
66 # | | | ||||
67 <...>-859 [000] ....
68 <...>-860 [001] ....
69 <...>-861 [002] ....
70 <...>-862 [003] ....
71 <...>-863 [004] ....
72 <...>-864 [005] ....
73 <...>-865 [006] ....
74 <...>-866 [007] ....
75
76 In addition to the regular trace fields (from
77 tracer prints a message at the end of each per
78 running an osnoise/ thread. The osnoise specif
79
80 - The RUNTIME IN US reports the amount of tim
81 the osnoise thread kept looping reading the
82 - The NOISE IN US reports the sum of noise in
83 by the osnoise tracer during the associated
84 - The % OF CPU AVAILABLE reports the percenta
85 the osnoise thread during the runtime windo
86 - The MAX SINGLE NOISE IN US reports the maxi
87 during the runtime window.
88 - The Interference counters display how many
89 interference happened during the runtime wi
90
91 Note that the example above shows a high numbe
92 The reason being is that this sample was taken
93 and the host interference is detected as a har
94
95 Tracer Configuration
96 --------------------
97
98 The tracer has a set of options inside the osn
99
100 - osnoise/cpus: CPUs at which a osnoise threa
101 - osnoise/period_us: the period of the osnois
102 - osnoise/runtime_us: how long an osnoise thr
103 - osnoise/stop_tracing_us: stop the system tr
104 higher than the configured value happens. W
105 option.
106 - osnoise/stop_tracing_total_us: stop the sys
107 higher than the configured value happens. W
108 option.
109 - tracing_threshold: the minimum delta betwee
110 considered as noise, in us. When set to 0,
111 be used, which is currently 1 us.
112 - osnoise/options: a set of on/off options th
113 writing the option name to the file or disa
114 name preceded with the 'NO\_' prefix. For e
115 NO_OSNOISE_WORKLOAD disables the OSNOISE_WO
116 special DEAFAULTS option resets all options
117
118 Tracer Options
119 --------------
120
121 The osnoise/options file exposes a set of on/o
122 the osnoise tracer. These options are:
123
124 - DEFAULTS: reset the options to the default
125 - OSNOISE_WORKLOAD: do not dispatch osnoise w
126 section below).
127 - PANIC_ON_STOP: call panic() if the tracer s
128 capture a vmcore.
129 - OSNOISE_PREEMPT_DISABLE: disable preemption
130 workload, allowing only IRQ and hardware-re
131 - OSNOISE_IRQ_DISABLE: disable IRQs while run
132 allowing only NMIs and hardware-related noi
133
134 Additional Tracing
135 ------------------
136
137 In addition to the tracer, a set of tracepoint
138 facilitate the identification of the osnoise s
139
140 - osnoise:sample_threshold: printed anytime a
141 the configurable tolerance_ns.
142 - osnoise:nmi_noise: noise from NMI, includin
143 - osnoise:irq_noise: noise from an IRQ, inclu
144 - osnoise:softirq_noise: noise from a SoftIRQ
145 duration.
146 - osnoise:thread_noise: noise from a thread,
147
148 Note that all the values are *net values*. For
149 is running, another thread preempts the osnois
150 thread_noise duration at the start. Then, an I
151 the thread_noise, starting a irq_noise. When t
152 it will compute its duration, and this duratio
153 the thread_noise, in such a way as to avoid th
154 IRQ execution. This logic is valid for all sou
155
156 Here is one example of the usage of these trac
157
158 osnoise/8-961 [008] d.h. 5789.8575
159 osnoise/8-961 [008] dNh. 5789.8584
160 migration/8-54 [008] d... 5789.8584
161 osnoise/8-961 [008] .... 5789.8584
162
163 In this example, a noise sample of 8 microseco
164 line, pointing to two interferences. Looking b
165 two previous entries were about the migration
166 timer IRQ execution. The first event is not pa
167 it took place one millisecond before.
168
169 It is worth noticing that the sum of the durat
170 tracepoints is smaller than eight us reported
171 The reason roots in the overhead of the entry
172 before and after any interference execution. T
173 approach: measuring thread and tracing.
174
175 Running osnoise tracer without workload
176 ---------------------------------------
177
178 By enabling the osnoise tracer with the NO_OSN
179 the osnoise: tracepoints serve to measure the
180 any type of Linux task, free from the interfer
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.