1 .. SPDX-License-Identifier: GPL-2.0
2
3 =======================
4 Energy Model of devices
5 =======================
6
7 1. Overview
8 -----------
9
10 The Energy Model (EM) framework serves as an interface between drivers knowing
11 the power consumed by devices at various performance levels, and the kernel
12 subsystems willing to use that information to make energy-aware decisions.
13
14 The source of the information about the power consumed by devices can vary greatly
15 from one platform to another. These power costs can be estimated using
16 devicetree data in some cases. In others, the firmware will know better.
17 Alternatively, userspace might be best positioned. And so on. In order to avoid
18 each and every client subsystem to re-implement support for each and every
19 possible source of information on its own, the EM framework intervenes as an
20 abstraction layer which standardizes the format of power cost tables in the
21 kernel, hence enabling to avoid redundant work.
22
23 The power values might be expressed in micro-Watts or in an 'abstract scale'.
24 Multiple subsystems might use the EM and it is up to the system integrator to
25 check that the requirements for the power value scale types are met. An example
26 can be found in the Energy-Aware Scheduler documentation
27 Documentation/scheduler/sched-energy.rst. For some subsystems like thermal or
28 powercap power values expressed in an 'abstract scale' might cause issues.
29 These subsystems are more interested in estimation of power used in the past,
30 thus the real micro-Watts might be needed. An example of these requirements can
31 be found in the Intelligent Power Allocation in
32 Documentation/driver-api/thermal/power_allocator.rst.
33 Kernel subsystems might implement automatic detection to check whether EM
34 registered devices have inconsistent scale (based on EM internal flag).
35 Important thing to keep in mind is that when the power values are expressed in
36 an 'abstract scale' deriving real energy in micro-Joules would not be possible.
37
38 The figure below depicts an example of drivers (Arm-specific here, but the
39 approach is applicable to any architecture) providing power costs to the EM
40 framework, and interested clients reading the data from it::
41
42 +---------------+ +-----------------+ +---------------+
43 | Thermal (IPA) | | Scheduler (EAS) | | Other |
44 +---------------+ +-----------------+ +---------------+
45 | | em_cpu_energy() |
46 | | em_cpu_get() |
47 +---------+ | +---------+
48 | | |
49 v v v
50 +---------------------+
51 | Energy Model |
52 | Framework |
53 +---------------------+
54 ^ ^ ^
55 | | | em_dev_register_perf_domain()
56 +----------+ | +---------+
57 | | |
58 +---------------+ +---------------+ +--------------+
59 | cpufreq-dt | | arm_scmi | | Other |
60 +---------------+ +---------------+ +--------------+
61 ^ ^ ^
62 | | |
63 +--------------+ +---------------+ +--------------+
64 | Device Tree | | Firmware | | ? |
65 +--------------+ +---------------+ +--------------+
66
67 In case of CPU devices the EM framework manages power cost tables per
68 'performance domain' in the system. A performance domain is a group of CPUs
69 whose performance is scaled together. Performance domains generally have a
70 1-to-1 mapping with CPUFreq policies. All CPUs in a performance domain are
71 required to have the same micro-architecture. CPUs in different performance
72 domains can have different micro-architectures.
73
74 To better reflect power variation due to static power (leakage) the EM
75 supports runtime modifications of the power values. The mechanism relies on
76 RCU to free the modifiable EM perf_state table memory. Its user, the task
77 scheduler, also uses RCU to access this memory. The EM framework provides
78 API for allocating/freeing the new memory for the modifiable EM table.
79 The old memory is freed automatically using RCU callback mechanism when there
80 are no owners anymore for the given EM runtime table instance. This is tracked
81 using kref mechanism. The device driver which provided the new EM at runtime,
82 should call EM API to free it safely when it's no longer needed. The EM
83 framework will handle the clean-up when it's possible.
84
85 The kernel code which want to modify the EM values is protected from concurrent
86 access using a mutex. Therefore, the device driver code must run in sleeping
87 context when it tries to modify the EM.
88
89 With the runtime modifiable EM we switch from a 'single and during the entire
90 runtime static EM' (system property) design to a 'single EM which can be
91 changed during runtime according e.g. to the workload' (system and workload
92 property) design.
93
94 It is possible also to modify the CPU performance values for each EM's
95 performance state. Thus, the full power and performance profile (which
96 is an exponential curve) can be changed according e.g. to the workload
97 or system property.
98
99
100 2. Core APIs
101 ------------
102
103 2.1 Config options
104 ^^^^^^^^^^^^^^^^^^
105
106 CONFIG_ENERGY_MODEL must be enabled to use the EM framework.
107
108
109 2.2 Registration of performance domains
110 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
111
112 Registration of 'advanced' EM
113 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
114
115 The 'advanced' EM gets its name due to the fact that the driver is allowed
116 to provide more precised power model. It's not limited to some implemented math
117 formula in the framework (like it is in 'simple' EM case). It can better reflect
118 the real power measurements performed for each performance state. Thus, this
119 registration method should be preferred in case considering EM static power
120 (leakage) is important.
121
122 Drivers are expected to register performance domains into the EM framework by
123 calling the following API::
124
125 int em_dev_register_perf_domain(struct device *dev, unsigned int nr_states,
126 struct em_data_callback *cb, cpumask_t *cpus, bool microwatts);
127
128 Drivers must provide a callback function returning <frequency, power> tuples
129 for each performance state. The callback function provided by the driver is free
130 to fetch data from any relevant location (DT, firmware, ...), and by any mean
131 deemed necessary. Only for CPU devices, drivers must specify the CPUs of the
132 performance domains using cpumask. For other devices than CPUs the last
133 argument must be set to NULL.
134 The last argument 'microwatts' is important to set with correct value. Kernel
135 subsystems which use EM might rely on this flag to check if all EM devices use
136 the same scale. If there are different scales, these subsystems might decide
137 to return warning/error, stop working or panic.
138 See Section 3. for an example of driver implementing this
139 callback, or Section 2.4 for further documentation on this API
140
141 Registration of EM using DT
142 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
143
144 The EM can also be registered using OPP framework and information in DT
145 "operating-points-v2". Each OPP entry in DT can be extended with a property
146 "opp-microwatt" containing micro-Watts power value. This OPP DT property
147 allows a platform to register EM power values which are reflecting total power
148 (static + dynamic). These power values might be coming directly from
149 experiments and measurements.
150
151 Registration of 'artificial' EM
152 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
153
154 There is an option to provide a custom callback for drivers missing detailed
155 knowledge about power value for each performance state. The callback
156 .get_cost() is optional and provides the 'cost' values used by the EAS.
157 This is useful for platforms that only provide information on relative
158 efficiency between CPU types, where one could use the information to
159 create an abstract power model. But even an abstract power model can
160 sometimes be hard to fit in, given the input power value size restrictions.
161 The .get_cost() allows to provide the 'cost' values which reflect the
162 efficiency of the CPUs. This would allow to provide EAS information which
163 has different relation than what would be forced by the EM internal
164 formulas calculating 'cost' values. To register an EM for such platform, the
165 driver must set the flag 'microwatts' to 0, provide .get_power() callback
166 and provide .get_cost() callback. The EM framework would handle such platform
167 properly during registration. A flag EM_PERF_DOMAIN_ARTIFICIAL is set for such
168 platform. Special care should be taken by other frameworks which are using EM
169 to test and treat this flag properly.
170
171 Registration of 'simple' EM
172 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
173
174 The 'simple' EM is registered using the framework helper function
175 cpufreq_register_em_with_opp(). It implements a power model which is tight to
176 math formula::
177
178 Power = C * V^2 * f
179
180 The EM which is registered using this method might not reflect correctly the
181 physics of a real device, e.g. when static power (leakage) is important.
182
183
184 2.3 Accessing performance domains
185 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
186
187 There are two API functions which provide the access to the energy model:
188 em_cpu_get() which takes CPU id as an argument and em_pd_get() with device
189 pointer as an argument. It depends on the subsystem which interface it is
190 going to use, but in case of CPU devices both functions return the same
191 performance domain.
192
193 Subsystems interested in the energy model of a CPU can retrieve it using the
194 em_cpu_get() API. The energy model tables are allocated once upon creation of
195 the performance domains, and kept in memory untouched.
196
197 The energy consumed by a performance domain can be estimated using the
198 em_cpu_energy() API. The estimation is performed assuming that the schedutil
199 CPUfreq governor is in use in case of CPU device. Currently this calculation is
200 not provided for other type of devices.
201
202 More details about the above APIs can be found in ``<linux/energy_model.h>``
203 or in Section 2.5
204
205
206 2.4 Runtime modifications
207 ^^^^^^^^^^^^^^^^^^^^^^^^^
208
209 Drivers willing to update the EM at runtime should use the following dedicated
210 function to allocate a new instance of the modified EM. The API is listed
211 below::
212
213 struct em_perf_table __rcu *em_table_alloc(struct em_perf_domain *pd);
214
215 This allows to allocate a structure which contains the new EM table with
216 also RCU and kref needed by the EM framework. The 'struct em_perf_table'
217 contains array 'struct em_perf_state state[]' which is a list of performance
218 states in ascending order. That list must be populated by the device driver
219 which wants to update the EM. The list of frequencies can be taken from
220 existing EM (created during boot). The content in the 'struct em_perf_state'
221 must be populated by the driver as well.
222
223 This is the API which does the EM update, using RCU pointers swap::
224
225 int em_dev_update_perf_domain(struct device *dev,
226 struct em_perf_table __rcu *new_table);
227
228 Drivers must provide a pointer to the allocated and initialized new EM
229 'struct em_perf_table'. That new EM will be safely used inside the EM framework
230 and will be visible to other sub-systems in the kernel (thermal, powercap).
231 The main design goal for this API is to be fast and avoid extra calculations
232 or memory allocations at runtime. When pre-computed EMs are available in the
233 device driver, than it should be possible to simply re-use them with low
234 performance overhead.
235
236 In order to free the EM, provided earlier by the driver (e.g. when the module
237 is unloaded), there is a need to call the API::
238
239 void em_table_free(struct em_perf_table __rcu *table);
240
241 It will allow the EM framework to safely remove the memory, when there is
242 no other sub-system using it, e.g. EAS.
243
244 To use the power values in other sub-systems (like thermal, powercap) there is
245 a need to call API which protects the reader and provide consistency of the EM
246 table data::
247
248 struct em_perf_state *em_perf_state_from_pd(struct em_perf_domain *pd);
249
250 It returns the 'struct em_perf_state' pointer which is an array of performance
251 states in ascending order.
252 This function must be called in the RCU read lock section (after the
253 rcu_read_lock()). When the EM table is not needed anymore there is a need to
254 call rcu_real_unlock(). In this way the EM safely uses the RCU read section
255 and protects the users. It also allows the EM framework to manage the memory
256 and free it. More details how to use it can be found in Section 3.2 in the
257 example driver.
258
259 There is dedicated API for device drivers to calculate em_perf_state::cost
260 values::
261
262 int em_dev_compute_costs(struct device *dev, struct em_perf_state *table,
263 int nr_states);
264
265 These 'cost' values from EM are used in EAS. The new EM table should be passed
266 together with the number of entries and device pointer. When the computation
267 of the cost values is done properly the return value from the function is 0.
268 The function takes care for right setting of inefficiency for each performance
269 state as well. It updates em_perf_state::flags accordingly.
270 Then such prepared new EM can be passed to the em_dev_update_perf_domain()
271 function, which will allow to use it.
272
273 More details about the above APIs can be found in ``<linux/energy_model.h>``
274 or in Section 3.2 with an example code showing simple implementation of the
275 updating mechanism in a device driver.
276
277
278 2.5 Description details of this API
279 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
280 .. kernel-doc:: include/linux/energy_model.h
281 :internal:
282
283 .. kernel-doc:: kernel/power/energy_model.c
284 :export:
285
286
287 3. Examples
288 -----------
289
290 3.1 Example driver with EM registration
291 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
292
293 The CPUFreq framework supports dedicated callback for registering
294 the EM for a given CPU(s) 'policy' object: cpufreq_driver::register_em().
295 That callback has to be implemented properly for a given driver,
296 because the framework would call it at the right time during setup.
297 This section provides a simple example of a CPUFreq driver registering a
298 performance domain in the Energy Model framework using the (fake) 'foo'
299 protocol. The driver implements an est_power() function to be provided to the
300 EM framework::
301
302 -> drivers/cpufreq/foo_cpufreq.c
303
304 01 static int est_power(struct device *dev, unsigned long *mW,
305 02 unsigned long *KHz)
306 03 {
307 04 long freq, power;
308 05
309 06 /* Use the 'foo' protocol to ceil the frequency */
310 07 freq = foo_get_freq_ceil(dev, *KHz);
311 08 if (freq < 0);
312 09 return freq;
313 10
314 11 /* Estimate the power cost for the dev at the relevant freq. */
315 12 power = foo_estimate_power(dev, freq);
316 13 if (power < 0);
317 14 return power;
318 15
319 16 /* Return the values to the EM framework */
320 17 *mW = power;
321 18 *KHz = freq;
322 19
323 20 return 0;
324 21 }
325 22
326 23 static void foo_cpufreq_register_em(struct cpufreq_policy *policy)
327 24 {
328 25 struct em_data_callback em_cb = EM_DATA_CB(est_power);
329 26 struct device *cpu_dev;
330 27 int nr_opp;
331 28
332 29 cpu_dev = get_cpu_device(cpumask_first(policy->cpus));
333 30
334 31 /* Find the number of OPPs for this policy */
335 32 nr_opp = foo_get_nr_opp(policy);
336 33
337 34 /* And register the new performance domain */
338 35 em_dev_register_perf_domain(cpu_dev, nr_opp, &em_cb, policy->cpus,
339 36 true);
340 37 }
341 38
342 39 static struct cpufreq_driver foo_cpufreq_driver = {
343 40 .register_em = foo_cpufreq_register_em,
344 41 };
345
346
347 3.2 Example driver with EM modification
348 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
349
350 This section provides a simple example of a thermal driver modifying the EM.
351 The driver implements a foo_thermal_em_update() function. The driver is woken
352 up periodically to check the temperature and modify the EM data::
353
354 -> drivers/soc/example/example_em_mod.c
355
356 01 static void foo_get_new_em(struct foo_context *ctx)
357 02 {
358 03 struct em_perf_table __rcu *em_table;
359 04 struct em_perf_state *table, *new_table;
360 05 struct device *dev = ctx->dev;
361 06 struct em_perf_domain *pd;
362 07 unsigned long freq;
363 08 int i, ret;
364 09
365 10 pd = em_pd_get(dev);
366 11 if (!pd)
367 12 return;
368 13
369 14 em_table = em_table_alloc(pd);
370 15 if (!em_table)
371 16 return;
372 17
373 18 new_table = em_table->state;
374 19
375 20 rcu_read_lock();
376 21 table = em_perf_state_from_pd(pd);
377 22 for (i = 0; i < pd->nr_perf_states; i++) {
378 23 freq = table[i].frequency;
379 24 foo_get_power_perf_values(dev, freq, &new_table[i]);
380 25 }
381 26 rcu_read_unlock();
382 27
383 28 /* Calculate 'cost' values for EAS */
384 29 ret = em_dev_compute_costs(dev, table, pd->nr_perf_states);
385 30 if (ret) {
386 31 dev_warn(dev, "EM: compute costs failed %d\n", ret);
387 32 em_free_table(em_table);
388 33 return;
389 34 }
390 35
391 36 ret = em_dev_update_perf_domain(dev, em_table);
392 37 if (ret) {
393 38 dev_warn(dev, "EM: update failed %d\n", ret);
394 39 em_free_table(em_table);
395 40 return;
396 41 }
397 42
398 43 /*
399 44 * Since it's one-time-update drop the usage counter.
400 45 * The EM framework will later free the table when needed.
401 46 */
402 47 em_table_free(em_table);
403 48 }
404 49
405 50 /*
406 51 * Function called periodically to check the temperature and
407 52 * update the EM if needed
408 53 */
409 54 static void foo_thermal_em_update(struct foo_context *ctx)
410 55 {
411 56 struct device *dev = ctx->dev;
412 57 int cpu;
413 58
414 59 ctx->temperature = foo_get_temp(dev, ctx);
415 60 if (ctx->temperature < FOO_EM_UPDATE_TEMP_THRESHOLD)
416 61 return;
417 62
418 63 foo_get_new_em(ctx);
419 64 }
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.