1 .. SPDX-License-Identifier: GPL-2.0
2
3 ==============================================
4 Timekeeping Virtualization for X86-Based Archi
5 ==============================================
6
7 :Author: Zachary Amsden <zamsden@redhat.com>
8 :Copyright: (c) 2010, Red Hat. All rights res
9
10 .. Contents
11
12 1) Overview
13 2) Timing Devices
14 3) TSC Hardware
15 4) Virtualization Problems
16
17 1. Overview
18 ===========
19
20 One of the most complicated parts of the X86 p
21 the virtualization of this platform is the ple
22 and the complexity of emulating those devices.
23 time introduces a new set of challenges becaus
24 division of time beyond the control of the gue
25
26 First, we will describe the various timekeepin
27 present some of the problems which arise and s
28 specific recommendations for certain classes o
29
30 The purpose of this document is to collect dat
31 timekeeping which may be difficult to find els
32 information relevant to KVM and hardware-based
33
34 2. Timing Devices
35 =================
36
37 First we discuss the basic hardware devices av
38 KVM clock are special enough to warrant a full
39 the following section.
40
41 2.1. i8254 - PIT
42 ----------------
43
44 One of the first timer devices available is th
45 or PIT. The PIT has a fixed frequency 1.19318
46 channels which can be programmed to deliver pe
47 These three channels can be configured in diff
48 counters. Channel 1 and 2 were not available
49 IBM PC, and historically were connected to con
50 speaker. Now the PIT is typically integrated
51 and a separate physical PIT is not used.
52
53 The PIT uses I/O ports 0x40 - 0x43. Access to
54 using single or multiple byte access to the I/
55 available, but not all modes are available to
56 has a connected gate input, required for modes
57 controlled by port 61h, bit 0, as illustrated
58
59 -------------- ----------------
60 | | | |
61 | 1.1932 MHz|---------->| CLOCK OUT |
62 | Clock | | | |
63 -------------- | +->| GATE TIMER 0 |
64 | ----------------
65 |
66 | ----------------
67 | | |
68 |------>| CLOCK OUT |
69 | | |
70 | +->| GATE TIMER 1 |
71 | ----------------
72 |
73 | ----------------
74 | | |
75 |------>| CLOCK OUT |
76 | |
77 Port 61h, bit 0 -------->| GATE TIMER 2 |
78 ----------------
79
80 Port 61h, bit 1 ----------------------------
81
82 The timer modes are now described.
83
84 Mode 0: Single Timeout.
85 This is a one-shot software timeout that coun
86 when the gate is high (always true for timers
87 reaches zero, the output goes high.
88
89 Mode 1: Triggered One-shot.
90 The output is initially set high. When the g
91 line is set high, a countdown is initiated (w
92 lowered), during which the output is set low.
93 the output goes high.
94
95 Mode 2: Rate Generator.
96 The output is initially set high. When the c
97 reaches 1, the output goes low for one count
98 is reloaded and the countdown automatically r
99 low, the count is halted. If the output is l
100 output automatically goes high (this only aff
101
102 Mode 3: Square Wave.
103 This generates a high / low square wave. The
104 determines the length of the pulse, which alt
105 when zero is reached. The count only proceed
106 automatically reloaded on reaching zero. The
107 each clock to generate a full high / low cycl
108 If the count is even, the clock remains high
109 counts; if the clock is odd, the clock is hig
110 for (N-1)/2 counts. Only even values are lat
111 values are not observed when reading. This i
112 which generates sine-like tones by low-pass f
113
114 Mode 4: Software Strobe.
115 After programming this mode and loading the c
116 the output remains high until the counter rea
117 goes low for 1 clock cycle and returns high.
118 Counting only occurs when gate is high.
119
120 Mode 5: Hardware Strobe.
121 After programming and loading the counter, th
122 output remains high. When the gate is raised
123 (which does not stop if the gate is lowered).
124 the output goes low for 1 clock cycle and the
125 not reloaded.
126
127 In addition to normal binary counting, the PIT
128 command port, 0x43 is used to set the counter
129 timers.
130
131 PIT commands, issued to port 0x43, using the f
132
133 Bit 7-4: Command (See table below)
134 Bit 3-1: Mode (000 = Mode 0, 101 = Mode 5, 1
135 Bit 0 : Binary (0) / BCD (1)
136
137 Command table::
138
139 0000 - Latch Timer 0 count for port 0x40
140 sample and hold the count to be read i
141 additional commands ignored until coun
142 mode bits ignored.
143
144 0001 - Set Timer 0 LSB mode for port 0x40
145 set timer to read LSB only and force M
146 mode bits set timer mode
147
148 0010 - Set Timer 0 MSB mode for port 0x40
149 set timer to read MSB only and force L
150 mode bits set timer mode
151
152 0011 - Set Timer 0 16-bit mode for port 0x40
153 set timer to read / write LSB first, t
154 mode bits set timer mode
155
156 0100 - Latch Timer 1 count for port 0x41 - a
157 0101 - Set Timer 1 LSB mode for port 0x41 -
158 0110 - Set Timer 1 MSB mode for port 0x41 -
159 0111 - Set Timer 1 16-bit mode for port 0x41
160
161 1000 - Latch Timer 2 count for port 0x42 - a
162 1001 - Set Timer 2 LSB mode for port 0x42 -
163 1010 - Set Timer 2 MSB mode for port 0x42 -
164 1011 - Set Timer 2 16-bit mode for port 0x42
165
166 1101 - General counter latch
167 Latch combination of counters into cor
168 Bit 3 = Counter 2
169 Bit 2 = Counter 1
170 Bit 1 = Counter 0
171 Bit 0 = Unused
172
173 1110 - Latch timer status
174 Latch combination of counter mode into
175 Bit 3 = Counter 2
176 Bit 2 = Counter 1
177 Bit 1 = Counter 0
178
179 The output of ports 0x40-0x42 followin
180
181 Bit 7 = Output pin
182 Bit 6 = Count loaded (0 if timer has e
183 Bit 5-4 = Read / Write mode
184 01 = MSB only
185 10 = LSB only
186 11 = LSB / MSB (16-bit)
187 Bit 3-1 = Mode
188 Bit 0 = Binary (0) / BCD mode (1)
189
190 2.2. RTC
191 --------
192
193 The second device which was available in the o
194 time clock. The original device is now obsole
195 system chipset, sometimes by an HPET and some
196
197 The RTC is accessed through CMOS variables, wh
198 control which bytes are read. Since there is
199 of the CMOS and read of the RTC require lock p
200 dangerous to allow userspace utilities such as
201 access, as they could corrupt kernel reads and
202
203 The RTC generates an interrupt which is usuall
204 can function as a periodic timer, an additiona
205 interrupts after an update of the CMOS registe
206 The type of interrupt is signalled in the RTC
207
208 The RTC will update the current time fields by
209 system is off. The current time fields should
210 in progress, as indicated in the status regist
211
212 The clock uses a 32.768kHz crystal, so bits 6-
213 programmed to a 32kHz divider if the RTC is to
214
215 This is the RAM map originally used for the RT
216
217 Location Size Description
218 ------------------------------------------
219 00h byte Current second (BCD)
220 01h byte Seconds alarm (BCD)
221 02h byte Current minute (BCD)
222 03h byte Minutes alarm (BCD)
223 04h byte Current hour (BCD)
224 05h byte Hours alarm (BCD)
225 06h byte Current day of week (BCD
226 07h byte Current day of month (BC
227 08h byte Current month (BCD)
228 09h byte Current year (BCD)
229 0Ah byte Register A
230 bit 7 = Update in pro
231 bit 6-4 = Divider for c
232 000 = 4.194
233 001 = 1.049
234 010 = 32 kHz
235 10X = test m
236 110 = reset
237 111 = reset
238 bit 3-0 = Rate selectio
239 000 = period
240 001 = 3.9062
241 010 = 7.8125
242 011 = .12207
243 100 = .24414
244 ...
245 1101 = 125 mS
246 1110 = 250 mS
247 1111 = 500 mS
248 0Bh byte Register B
249 bit 7 = Run (0) / Hal
250 bit 6 = Periodic inte
251 bit 5 = Alarm interru
252 bit 4 = Update-ended
253 bit 3 = Square wave i
254 bit 2 = BCD calendar
255 bit 1 = 12-hour mode
256 bit 0 = 0 (DST off) /
257 OCh byte Register C (read only)
258 bit 7 = interrupt req
259 bit 6 = periodic inte
260 bit 5 = alarm interru
261 bit 4 = update interr
262 bit 3-0 = reserved
263 ODh byte Register D (read only)
264 bit 7 = RTC has power
265 bit 6-0 = reserved
266 32h byte Current century BCD (*)
267 (*) location vendor specific and now determi
268
269 2.3. APIC
270 ---------
271
272 On Pentium and later processors, an on-board t
273 as part of the Advanced Programmable Interrupt
274 accessed through memory-mapped registers and p
275 CPU, used for IPIs and local timer interrupts.
276
277 Although in theory the APIC is a safe and stab
278 in practice, many bugs and glitches have occur
279 the APIC CPU-local memory-mapped hardware. Be
280 the use of the APIC and that workarounds may b
281 these workarounds pose unique constraints for
282 extra overhead incurred from extra reads of me
283 functionality that may be more computationally
284
285 Since the APIC is documented quite well in the
286 avoid repetition of the detail here. It shoul
287 timer is programmed through the LVT (local vec
288 of one-shot or periodic operation, and is base
289 by the programmable divider register.
290
291 2.4. HPET
292 ---------
293
294 HPET is quite complex, and was originally inte
295 support of the X86 PC. It remains to be seen
296 the de facto standard of PC hardware is to emu
297 systems designated as legacy free may support
298 device.
299
300 The HPET spec is rather loose and vague, requi
301 but allowing implementation freedom to support
302 fixed rate on the timer frequency, but does im
303 frequency, error and slew.
304
305 In general, the HPET is recommended as a high
306 time source which is independent of local vari
307 in any given system). The HPET is also memory
308 indicated through ACPI tables by the BIOS.
309
310 Detailed specification of the HPET is beyond t
311 document, as it is also very well documented e
312
313 2.5. Offboard Timers
314 --------------------
315
316 Several cards, both proprietary (watchdog boar
317 timing chips built into the cards which may ha
318 to kernel or user drivers. To the author's kn
319 a clocksource for a Linux or other kernel has
320 general frowned upon as not playing by the agr
321 timer device would require additional support
322 not considered important at this time as no kn
323
324 3. TSC Hardware
325 ===============
326
327 The TSC or time stamp counter is relatively si
328 instruction cycles issued by the processor, wh
329 time. In practice, due to a number of problem
330 timekeeping device to use.
331
332 The TSC is represented internally as a 64-bit
333 RDMSR, RDTSC, or RDTSCP (when available) instr
334 limitations made it possible to write the TSC,
335 was only possible to write the low 32-bits of
336 32-bits of the counter were cleared. Now, how
337 0Fh, for models 3, 4 and 6, and family 06h, mo
338 has been lifted and all 64-bits are writable.
339 write the TSC MSR is not an architectural guar
340
341 The TSC is accessible from CPL-0 and condition
342 means of the CR4.TSD bit, which when enabled,
343
344 Some vendors have implemented an additional in
345 atomically not just the TSC, but an indicator
346 processor number. This can be used to index i
347 determine offset information in SMP systems wh
348 The presence of this instruction must be deter
349 bits.
350
351 Both VMX and SVM provide extension fields in t
352 allows the guest visible TSC to be offset by a
353 promise to allow the TSC to additionally be sc
354 yet widely available.
355
356 3.1. TSC synchronization
357 ------------------------
358
359 The TSC is a CPU-local clock in most implement
360 platforms, the TSCs of different CPUs may star
361 on when the CPUs are powered on. Generally, C
362 the same clock, however, this is not always th
363
364 The BIOS may attempt to resynchronize the TSCs
365 the operating system or other system software
366 Several hardware limitations make the problem
367 write the full 64-bits of the TSC, it may be i
368 newly arriving CPUs to that of the rest of the
369 unsynchronized TSCs. This may be done by BIOS
370 practice, getting a perfectly synchronized TSC
371 values are read from the same clock, which gen
372 socket systems or those with special hardware
373
374 3.2. TSC and CPU hotplug
375 ------------------------
376
377 As touched on already, CPUs which arrive later
378 may not have a TSC value that is synchronized
379 Either system software, BIOS, or SMM code may
380 to a value matching the rest of the system, bu
381 a guarantee. This can have the effect of brin
382 TSC is synchronized back to a state where TSC
383 small, may be exposed to the OS and any virtua
384
385 3.3. TSC and multi-socket / NUMA
386 --------------------------------
387
388 Multi-socket systems, especially large multi-s
389 individual clocksources rather than a single,
390 Since these clocks are driven by different cry
391 perfectly matched frequency, and temperature a
392 cause the CPU clocks, and thus the TSCs to dri
393 exact clock and bus design, the drift may or m
394 error, and may accumulate over time.
395
396 In addition, very large systems may deliberate
397 cores. This technique, known as spread-spectr
398 clock frequency and harmonics of it, which may
399 standards for telecommunications and computer
400
401 It is recommended not to trust the TSCs to rem
402 multiple socket systems for these reasons.
403
404 3.4. TSC and C-states
405 ---------------------
406
407 C-states, or idling states of the processor, e
408 states may be problematic for TSC as well. Th
409 a state, resulting in a TSC which is behind th
410 is resumed. Such CPUs must be detected and fl
411 based on CPU and chipset identifications.
412
413 The TSC in such a case may be corrected by cat
414 clocksource.
415
416 3.5. TSC frequency change / P-states
417 ------------------------------------
418
419 To make things slightly more interesting, some
420 may or may not run the TSC at the same rate, a
421 may be staggered or slewed, at some points in
422 known other than falling within a range of val
423 not be a stable time source, and must be calib
424 external clock to be a usable source of time.
425
426 Whether the TSC runs at a constant rate or sca
427 dependent and must be determined by inspecting
428 specific MSR fields.
429
430 In addition, some vendors have known bugs wher
431 compensated for properly during normal operati
432 inactive, the P-state may be raised temporaril
433 other processors. In such cases, the TSC on h
434 than that of non-halted processors. AMD Turio
435 this problem.
436
437 3.6. TSC and STPCLK / T-states
438 ------------------------------
439
440 External signals given to the processor may al
441 the TSC. This is typically done for thermal e
442 an overheating condition, and typically, there
443 condition has happened.
444
445 3.7. TSC virtualization - VMX
446 -----------------------------
447
448 VMX provides conditional trapping of RDTSC, RD
449 instructions, which is enough for full virtual
450 addition, VMX allows passing through the host
451 field specified in the VMCS. Special instruct
452 write the VMCS field.
453
454 3.8. TSC virtualization - SVM
455 -----------------------------
456
457 SVM provides conditional trapping of RDTSC, RD
458 instructions, which is enough for full virtual
459 addition, SVM allows passing through the host
460 field specified in the SVM control block.
461
462 3.9. TSC feature bits in Linux
463 ------------------------------
464
465 In summary, there is no way to guarantee the T
466 synchronization unless it is explicitly guaran
467 if so, the TSCs in multi-sockets or NUMA syste
468 despite being locally consistent.
469
470 The following feature bits are used by Linux t
471 but they can only be taken to be meaningful fo
472
473 ========================= ==============
474 X86_FEATURE_TSC The TSC is ava
475 X86_FEATURE_RDTSCP The RDTSCP ins
476 X86_FEATURE_CONSTANT_TSC The TSC rate i
477 X86_FEATURE_NONSTOP_TSC The TSC does n
478 X86_FEATURE_TSC_RELIABLE TSC sync check
479 ========================= ==============
480
481 4. Virtualization Problems
482 ==========================
483
484 Timekeeping is especially problematic for virt
485 challenges arise. The most obvious problem is
486 the host and, potentially, a number of virtual
487 operating system does not run with 100% usage
488 it may very well make that assumption. It may
489 exacting bounds when interrupt sources are dis
490 virtual interrupt sources are disabled, and th
491 at any time. This causes problems as the pass
492 of machine interrupts and the associated clock
493 synchronized with real time.
494
495 This same problem can occur on native hardware
496 steal cycles from the naturally on X86 systems
497 BIOS, but not in such an extreme fashion. How
498 cause similar problems to virtualization makes
499 solving many of these problems on bare metal.
500
501 4.1. Interrupt clocking
502 -----------------------
503
504 One of the most immediate problems that occurs
505 is that the system timekeeping routines are of
506 time by counting periodic interrupts. These i
507 or the RTC, but the problem is the same: the h
508 be able to deliver the proper number of interr
509 time may fall behind. This is especially prob
510 is selected, such as 1000 HZ, which is unfortu
511 guests.
512
513 There are three approaches to solving this pro
514 to simply ignore it. Guests which have a sepa
515 'wall clock' or 'real time' may not need any a
516 maintain proper time. If this is not sufficie
517 additional interrupts into the guest in order
518 interrupt rate. This approach leads to compli
519 where host load or guest lag is too much to co
520 solution to the problem has risen: the guest m
521 ticks and compensate for them internally. Alt
522 implementation of this policy in Linux has bee
523 number of buggy variants of lost tick compensa
524 commonly used Linux systems.
525
526 Windows uses periodic RTC clocking as a means
527 thus requires interrupt slewing to keep proper
528 rate (ed: is it 18.2 Hz?) however that it has
529 practice.
530
531 4.2. TSC sampling and serialization
532 -----------------------------------
533
534 As the highest precision time source available
535 has aroused much interest from developers. As
536 many problems unique to its nature as a local,
537 potentially unsynchronized source. One issue
538 but is highlighted because of its very precise
539 definition, the counter, once read is already
540 possible for the counter to be read ahead of t
541 This is a consequence of the superscalar execu
542 which may execute instructions out of order.
543 non-serialized. Forcing serialized execution
544 measurement with the TSC, and requires a seria
545 or an MSR read.
546
547 Since CPUID may actually be virtualized by a t
548 serialization can pose a performance issue for
549 accurate time stamp counter reading may theref
550 it may be necessary for an implementation to g
551 the TSC as seen from other CPUs, even in an ot
552 system.
553
554 4.3. Timespec aliasing
555 ----------------------
556
557 Additionally, this lack of serialization from
558 when using results of the TSC when measured ag
559 the TSC is much higher precision, many possibl
560 while another clock is still expressing the sa
561
562 That is, you may read (T,T+10) while external
563 Due to non-serialized reads, you may actually
564 fluctuates - from (T-1.. T+10). Thus, any tim
565 calibrated against an external value may have
566 Re-calibrating this computation may actually c
567 calibration, to go backwards, compared with ti
568 calibration.
569
570 This problem is particularly pronounced with a
571 the kernel time, which is expressed in the the
572 timespec - but which advances in much larger g
573 at the rate of jiffies, and possibly in catchu
574
575 This aliasing requires care in the computation
576 and any other values derived from TSC computat
577 itself).
578
579 4.4. Migration
580 --------------
581
582 Migration of a virtual machine raises problems
583 First, the migration itself may take time, dur
584 delivered, and after which, the guest time may
585 be able to help to some degree here, as the cl
586 typically small enough to fall in the NTP-corr
587
588 An additional concern is that timers based off
589 clock is exposed) may now be running at differ
590 in some way in the hypervisor by virtualizing
591 migrating to a faster machine may preclude the
592 faster clock cannot be made visible to a guest
593 advancing faster than usual. A slower clock i
594 always be caught up to the original rate. KVM
595 simply storing multipliers and offsets against
596 back into nanosecond resolution values.
597
598 4.5. Scheduling
599 ---------------
600
601 Since scheduling may be based on precise timin
602 scheduling algorithms of an operating system m
603 virtualization. In theory, the effect is rand
604 distributed, but in contrived as well as real
605 causes of virtualization exits, possible conte
606 be the case. The effect of this has not been
607
608 In an attempt to work around this, several imp
609 paravirtualized scheduler clock, which reveals
610 which a virtual machine has been running.
611
612 4.6. Watchdogs
613 --------------
614
615 Watchdog timers, such as the lock detector in
616 running under hardware virtualization due to t
617 misinterpretation of the passage of real time.
618 spurious and can be ignored, but in some circu
619 disable such detection.
620
621 4.7. Delays and precision timing
622 --------------------------------
623
624 Precise timing and delays may not be possible
625 can happen if the system is controlling physic
626 compensate for slower I/O to and from devices.
627 in general for a virtualized system; hardware
628 adequately virtualized without a full real-tim
629 require an RT aware virtualization platform.
630
631 The second issue may cause performance problem
632 significant issue. In many cases these delays
633 configuration or paravirtualization.
634
635 4.8. Covert channels and leaks
636 ------------------------------
637
638 In addition to the above problems, time inform
639 guest about the host in anything but a perfect
640 time. This may allow the guest to infer the p
641 red-pill type detection), and it may allow inf
642 by using CPU utilization itself as a signallin
643 problems would require completely isolated vir
644 real time any longer. This may be useful in c
645 but in general isn't recommended for real-worl
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.