1 ============================
2 Transactional Memory support
3 ============================
4
5 POWER kernel support for this feature is currently limited to supporting
6 its use by user programs. It is not currently used by the kernel itself.
7
8 This file aims to sum up how it is supported by Linux and what behaviour you
9 can expect from your user programs.
10
11
12 Basic overview
13 ==============
14
15 Hardware Transactional Memory is supported on POWER8 processors, and is a
16 feature that enables a different form of atomic memory access. Several new
17 instructions are presented to delimit transactions; transactions are
18 guaranteed to either complete atomically or roll back and undo any partial
19 changes.
20
21 A simple transaction looks like this::
22
23 begin_move_money:
24 tbegin
25 beq abort_handler
26
27 ld r4, SAVINGS_ACCT(r3)
28 ld r5, CURRENT_ACCT(r3)
29 subi r5, r5, 1
30 addi r4, r4, 1
31 std r4, SAVINGS_ACCT(r3)
32 std r5, CURRENT_ACCT(r3)
33
34 tend
35
36 b continue
37
38 abort_handler:
39 ... test for odd failures ...
40
41 /* Retry the transaction if it failed because it conflicted with
42 * someone else: */
43 b begin_move_money
44
45
46 The 'tbegin' instruction denotes the start point, and 'tend' the end point.
47 Between these points the processor is in 'Transactional' state; any memory
48 references will complete in one go if there are no conflicts with other
49 transactional or non-transactional accesses within the system. In this
50 example, the transaction completes as though it were normal straight-line code
51 IF no other processor has touched SAVINGS_ACCT(r3) or CURRENT_ACCT(r3); an
52 atomic move of money from the current account to the savings account has been
53 performed. Even though the normal ld/std instructions are used (note no
54 lwarx/stwcx), either *both* SAVINGS_ACCT(r3) and CURRENT_ACCT(r3) will be
55 updated, or neither will be updated.
56
57 If, in the meantime, there is a conflict with the locations accessed by the
58 transaction, the transaction will be aborted by the CPU. Register and memory
59 state will roll back to that at the 'tbegin', and control will continue from
60 'tbegin+4'. The branch to abort_handler will be taken this second time; the
61 abort handler can check the cause of the failure, and retry.
62
63 Checkpointed registers include all GPRs, FPRs, VRs/VSRs, LR, CCR/CR, CTR, FPCSR
64 and a few other status/flag regs; see the ISA for details.
65
66 Causes of transaction aborts
67 ============================
68
69 - Conflicts with cache lines used by other processors
70 - Signals
71 - Context switches
72 - See the ISA for full documentation of everything that will abort transactions.
73
74
75 Syscalls
76 ========
77
78 Syscalls made from within an active transaction will not be performed and the
79 transaction will be doomed by the kernel with the failure code TM_CAUSE_SYSCALL
80 | TM_CAUSE_PERSISTENT.
81
82 Syscalls made from within a suspended transaction are performed as normal and
83 the transaction is not explicitly doomed by the kernel. However, what the
84 kernel does to perform the syscall may result in the transaction being doomed
85 by the hardware. The syscall is performed in suspended mode so any side
86 effects will be persistent, independent of transaction success or failure. No
87 guarantees are provided by the kernel about which syscalls will affect
88 transaction success.
89
90 Care must be taken when relying on syscalls to abort during active transactions
91 if the calls are made via a library. Libraries may cache values (which may
92 give the appearance of success) or perform operations that cause transaction
93 failure before entering the kernel (which may produce different failure codes).
94 Examples are glibc's getpid() and lazy symbol resolution.
95
96
97 Signals
98 =======
99
100 Delivery of signals (both sync and async) during transactions provides a second
101 thread state (ucontext/mcontext) to represent the second transactional register
102 state. Signal delivery 'treclaim's to capture both register states, so signals
103 abort transactions. The usual ucontext_t passed to the signal handler
104 represents the checkpointed/original register state; the signal appears to have
105 arisen at 'tbegin+4'.
106
107 If the sighandler ucontext has uc_link set, a second ucontext has been
108 delivered. For future compatibility the MSR.TS field should be checked to
109 determine the transactional state -- if so, the second ucontext in uc->uc_link
110 represents the active transactional registers at the point of the signal.
111
112 For 64-bit processes, uc->uc_mcontext.regs->msr is a full 64-bit MSR and its TS
113 field shows the transactional mode.
114
115 For 32-bit processes, the mcontext's MSR register is only 32 bits; the top 32
116 bits are stored in the MSR of the second ucontext, i.e. in
117 uc->uc_link->uc_mcontext.regs->msr. The top word contains the transactional
118 state TS.
119
120 However, basic signal handlers don't need to be aware of transactions
121 and simply returning from the handler will deal with things correctly:
122
123 Transaction-aware signal handlers can read the transactional register state
124 from the second ucontext. This will be necessary for crash handlers to
125 determine, for example, the address of the instruction causing the SIGSEGV.
126
127 Example signal handler::
128
129 void crash_handler(int sig, siginfo_t *si, void *uc)
130 {
131 ucontext_t *ucp = uc;
132 ucontext_t *transactional_ucp = ucp->uc_link;
133
134 if (ucp_link) {
135 u64 msr = ucp->uc_mcontext.regs->msr;
136 /* May have transactional ucontext! */
137 #ifndef __powerpc64__
138 msr |= ((u64)transactional_ucp->uc_mcontext.regs->msr) << 32;
139 #endif
140 if (MSR_TM_ACTIVE(msr)) {
141 /* Yes, we crashed during a transaction. Oops. */
142 fprintf(stderr, "Transaction to be restarted at 0x%llx, but "
143 "crashy instruction was at 0x%llx\n",
144 ucp->uc_mcontext.regs->nip,
145 transactional_ucp->uc_mcontext.regs->nip);
146 }
147 }
148
149 fix_the_problem(ucp->dar);
150 }
151
152 When in an active transaction that takes a signal, we need to be careful with
153 the stack. It's possible that the stack has moved back up after the tbegin.
154 The obvious case here is when the tbegin is called inside a function that
155 returns before a tend. In this case, the stack is part of the checkpointed
156 transactional memory state. If we write over this non transactionally or in
157 suspend, we are in trouble because if we get a tm abort, the program counter and
158 stack pointer will be back at the tbegin but our in memory stack won't be valid
159 anymore.
160
161 To avoid this, when taking a signal in an active transaction, we need to use
162 the stack pointer from the checkpointed state, rather than the speculated
163 state. This ensures that the signal context (written tm suspended) will be
164 written below the stack required for the rollback. The transaction is aborted
165 because of the treclaim, so any memory written between the tbegin and the
166 signal will be rolled back anyway.
167
168 For signals taken in non-TM or suspended mode, we use the
169 normal/non-checkpointed stack pointer.
170
171 Any transaction initiated inside a sighandler and suspended on return
172 from the sighandler to the kernel will get reclaimed and discarded.
173
174 Failure cause codes used by kernel
175 ==================================
176
177 These are defined in <asm/reg.h>, and distinguish different reasons why the
178 kernel aborted a transaction:
179
180 ====================== ================================
181 TM_CAUSE_RESCHED Thread was rescheduled.
182 TM_CAUSE_TLBI Software TLB invalid.
183 TM_CAUSE_FAC_UNAV FP/VEC/VSX unavailable trap.
184 TM_CAUSE_SYSCALL Syscall from active transaction.
185 TM_CAUSE_SIGNAL Signal delivered.
186 TM_CAUSE_MISC Currently unused.
187 TM_CAUSE_ALIGNMENT Alignment fault.
188 TM_CAUSE_EMULATE Emulation that touched memory.
189 ====================== ================================
190
191 These can be checked by the user program's abort handler as TEXASR[0:7]. If
192 bit 7 is set, it indicates that the error is considered persistent. For example
193 a TM_CAUSE_ALIGNMENT will be persistent while a TM_CAUSE_RESCHED will not.
194
195 GDB
196 ===
197
198 GDB and ptrace are not currently TM-aware. If one stops during a transaction,
199 it looks like the transaction has just started (the checkpointed state is
200 presented). The transaction cannot then be continued and will take the failure
201 handler route. Furthermore, the transactional 2nd register state will be
202 inaccessible. GDB can currently be used on programs using TM, but not sensibly
203 in parts within transactions.
204
205 POWER9
206 ======
207
208 TM on POWER9 has issues with storing the complete register state. This
209 is described in this commit::
210
211 commit 4bb3c7a0208fc13ca70598efd109901a7cd45ae7
212 Author: Paul Mackerras <paulus@ozlabs.org>
213 Date: Wed Mar 21 21:32:01 2018 +1100
214 KVM: PPC: Book3S HV: Work around transactional memory bugs in POWER9
215
216 To account for this different POWER9 chips have TM enabled in
217 different ways.
218
219 On POWER9N DD2.01 and below, TM is disabled. ie
220 HWCAP2[PPC_FEATURE2_HTM] is not set.
221
222 On POWER9N DD2.1 TM is configured by firmware to always abort a
223 transaction when tm suspend occurs. So tsuspend will cause a
224 transaction to be aborted and rolled back. Kernel exceptions will also
225 cause the transaction to be aborted and rolled back and the exception
226 will not occur. If userspace constructs a sigcontext that enables TM
227 suspend, the sigcontext will be rejected by the kernel. This mode is
228 advertised to users with HWCAP2[PPC_FEATURE2_HTM_NO_SUSPEND] set.
229 HWCAP2[PPC_FEATURE2_HTM] is not set in this mode.
230
231 On POWER9N DD2.2 and above, KVM and POWERVM emulate TM for guests (as
232 described in commit 4bb3c7a0208f), hence TM is enabled for guests
233 ie. HWCAP2[PPC_FEATURE2_HTM] is set for guest userspace. Guests that
234 makes heavy use of TM suspend (tsuspend or kernel suspend) will result
235 in traps into the hypervisor and hence will suffer a performance
236 degradation. Host userspace has TM disabled
237 ie. HWCAP2[PPC_FEATURE2_HTM] is not set. (although we make enable it
238 at some point in the future if we bring the emulation into host
239 userspace context switching).
240
241 POWER9C DD1.2 and above are only available with POWERVM and hence
242 Linux only runs as a guest. On these systems TM is emulated like on
243 POWER9N DD2.2.
244
245 Guest migration from POWER8 to POWER9 will work with POWER9N DD2.2 and
246 POWER9C DD1.2. Since earlier POWER9 processors don't support TM
247 emulation, migration from POWER8 to POWER9 is not supported there.
248
249 Kernel implementation
250 =====================
251
252 h/rfid mtmsrd quirk
253 -------------------
254
255 As defined in the ISA, rfid has a quirk which is useful in early
256 exception handling. When in a userspace transaction and we enter the
257 kernel via some exception, MSR will end up as TM=0 and TS=01 (ie. TM
258 off but TM suspended). Regularly the kernel will want change bits in
259 the MSR and will perform an rfid to do this. In this case rfid can
260 have SRR0 TM = 0 and TS = 00 (ie. TM off and non transaction) and the
261 resulting MSR will retain TM = 0 and TS=01 from before (ie. stay in
262 suspend). This is a quirk in the architecture as this would normally
263 be a transition from TS=01 to TS=00 (ie. suspend -> non transactional)
264 which is an illegal transition.
265
266 This quirk is described the architecture in the definition of rfid
267 with these lines:
268
269 if (MSR 29:31 ¬ = 0b010 | SRR1 29:31 ¬ = 0b000) then
270 MSR 29:31 <- SRR1 29:31
271
272 hrfid and mtmsrd have the same quirk.
273
274 The Linux kernel uses this quirk in its early exception handling.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.