1 Error Detection And Correction (EDAC) Devices 2 ============================================= 3 4 Main Concepts used at the EDAC subsystem 5 ---------------------------------------- 6 7 There are several things to be aware of that aren't at all obvious, like 8 *sockets, *socket sets*, *banks*, *rows*, *chip-select rows*, *channels*, 9 etc... 10 11 These are some of the many terms that are thrown about that don't always 12 mean what people think they mean (Inconceivable!). In the interest of 13 creating a common ground for discussion, terms and their definitions 14 will be established. 15 16 * Memory devices 17 18 The individual DRAM chips on a memory stick. These devices commonly 19 output 4 and 8 bits each (x4, x8). Grouping several of these in parallel 20 provides the number of bits that the memory controller expects: 21 typically 72 bits, in order to provide 64 bits + 8 bits of ECC data. 22 23 * Memory Stick 24 25 A printed circuit board that aggregates multiple memory devices in 26 parallel. In general, this is the Field Replaceable Unit (FRU) which 27 gets replaced, in the case of excessive errors. Most often it is also 28 called DIMM (Dual Inline Memory Module). 29 30 * Memory Socket 31 32 A physical connector on the motherboard that accepts a single memory 33 stick. Also called as "slot" on several datasheets. 34 35 * Channel 36 37 A memory controller channel, responsible to communicate with a group of 38 DIMMs. Each channel has its own independent control (command) and data 39 bus, and can be used independently or grouped with other channels. 40 41 * Branch 42 43 It is typically the highest hierarchy on a Fully-Buffered DIMM memory 44 controller. Typically, it contains two channels. Two channels at the 45 same branch can be used in single mode or in lockstep mode. When 46 lockstep is enabled, the cacheline is doubled, but it generally brings 47 some performance penalty. Also, it is generally not possible to point to 48 just one memory stick when an error occurs, as the error correction code 49 is calculated using two DIMMs instead of one. Due to that, it is capable 50 of correcting more errors than on single mode. 51 52 * Single-channel 53 54 The data accessed by the memory controller is contained into one dimm 55 only. E. g. if the data is 64 bits-wide, the data flows to the CPU using 56 one 64 bits parallel access. Typically used with SDR, DDR, DDR2 and DDR3 57 memories. FB-DIMM and RAMBUS use a different concept for channel, so 58 this concept doesn't apply there. 59 60 * Double-channel 61 62 The data size accessed by the memory controller is interlaced into two 63 dimms, accessed at the same time. E. g. if the DIMM is 64 bits-wide (72 64 bits with ECC), the data flows to the CPU using a 128 bits parallel 65 access. 66 67 * Chip-select row 68 69 This is the name of the DRAM signal used to select the DRAM ranks to be 70 accessed. Common chip-select rows for single channel are 64 bits, for 71 dual channel 128 bits. It may not be visible by the memory controller, 72 as some DIMM types have a memory buffer that can hide direct access to 73 it from the Memory Controller. 74 75 * Single-Ranked stick 76 77 A Single-ranked stick has 1 chip-select row of memory. Motherboards 78 commonly drive two chip-select pins to a memory stick. A single-ranked 79 stick, will occupy only one of those rows. The other will be unused. 80 81 .. _doubleranked: 82 83 * Double-Ranked stick 84 85 A double-ranked stick has two chip-select rows which access different 86 sets of memory devices. The two rows cannot be accessed concurrently. 87 88 * Double-sided stick 89 90 **DEPRECATED TERM**, see :ref:`Double-Ranked stick <doubleranked>`. 91 92 A double-sided stick has two chip-select rows which access different sets 93 of memory devices. The two rows cannot be accessed concurrently. 94 "Double-sided" is irrespective of the memory devices being mounted on 95 both sides of the memory stick. 96 97 * Socket set 98 99 All of the memory sticks that are required for a single memory access or 100 all of the memory sticks spanned by a chip-select row. A single socket 101 set has two chip-select rows and if double-sided sticks are used these 102 will occupy those chip-select rows. 103 104 * Bank 105 106 This term is avoided because it is unclear when needing to distinguish 107 between chip-select rows and socket sets. 108 109 * High Bandwidth Memory (HBM) 110 111 HBM is a new memory type with low power consumption and ultra-wide 112 communication lanes. It uses vertically stacked memory chips (DRAM dies) 113 interconnected by microscopic wires called "through-silicon vias," or 114 TSVs. 115 116 Several stacks of HBM chips connect to the CPU or GPU through an ultra-fast 117 interconnect called the "interposer". Therefore, HBM's characteristics 118 are nearly indistinguishable from on-chip integrated RAM. 119 120 Memory Controllers 121 ------------------ 122 123 Most of the EDAC core is focused on doing Memory Controller error detection. 124 The :c:func:`edac_mc_alloc`. It uses internally the struct ``mem_ctl_info`` 125 to describe the memory controllers, with is an opaque struct for the EDAC 126 drivers. Only the EDAC core is allowed to touch it. 127 128 .. kernel-doc:: include/linux/edac.h 129 130 .. kernel-doc:: drivers/edac/edac_mc.h 131 132 PCI Controllers 133 --------------- 134 135 The EDAC subsystem provides a mechanism to handle PCI controllers by calling 136 the :c:func:`edac_pci_alloc_ctl_info`. It will use the struct 137 :c:type:`edac_pci_ctl_info` to describe the PCI controllers. 138 139 .. kernel-doc:: drivers/edac/edac_pci.h 140 141 EDAC Blocks 142 ----------- 143 144 The EDAC subsystem also provides a generic mechanism to report errors on 145 other parts of the hardware via :c:func:`edac_device_alloc_ctl_info` function. 146 147 The structures :c:type:`edac_dev_sysfs_block_attribute`, 148 :c:type:`edac_device_block`, :c:type:`edac_device_instance` and 149 :c:type:`edac_device_ctl_info` provide a generic or abstract 'edac_device' 150 representation at sysfs. 151 152 This set of structures and the code that implements the APIs for the same, provide for registering EDAC type devices which are NOT standard memory or 153 PCI, like: 154 155 - CPU caches (L1 and L2) 156 - DMA engines 157 - Core CPU switches 158 - Fabric switch units 159 - PCIe interface controllers 160 - other EDAC/ECC type devices that can be monitored for 161 errors, etc. 162 163 It allows for a 2 level set of hierarchy. 164 165 For example, a cache could be composed of L1, L2 and L3 levels of cache. 166 Each CPU core would have its own L1 cache, while sharing L2 and maybe L3 167 caches. On such case, those can be represented via the following sysfs 168 nodes:: 169 170 /sys/devices/system/edac/.. 171 172 pci/ <existing pci directory (if available)> 173 mc/ <existing memory device directory> 174 cpu/cpu0/.. <L1 and L2 block directory> 175 /L1-cache/ce_count 176 /ue_count 177 /L2-cache/ce_count 178 /ue_count 179 cpu/cpu1/.. <L1 and L2 block directory> 180 /L1-cache/ce_count 181 /ue_count 182 /L2-cache/ce_count 183 /ue_count 184 ... 185 186 the L1 and L2 directories would be "edac_device_block's" 187 188 .. kernel-doc:: drivers/edac/edac_device.h 189 190 191 Heterogeneous system support 192 ---------------------------- 193 194 An AMD heterogeneous system is built by connecting the data fabrics of 195 both CPUs and GPUs via custom xGMI links. Thus, the data fabric on the 196 GPU nodes can be accessed the same way as the data fabric on CPU nodes. 197 198 The MI200 accelerators are data center GPUs. They have 2 data fabrics, 199 and each GPU data fabric contains four Unified Memory Controllers (UMC). 200 Each UMC contains eight channels. Each UMC channel controls one 128-bit 201 HBM2e (2GB) channel (equivalent to 8 X 2GB ranks). This creates a total 202 of 4096-bits of DRAM data bus. 203 204 While the UMC is interfacing a 16GB (8high X 2GB DRAM) HBM stack, each UMC 205 channel is interfacing 2GB of DRAM (represented as rank). 206 207 Memory controllers on AMD GPU nodes can be represented in EDAC thusly: 208 209 GPU DF / GPU Node -> EDAC MC 210 GPU UMC -> EDAC CSROW 211 GPU UMC channel -> EDAC CHANNEL 212 213 For example: a heterogeneous system with 1 AMD CPU is connected to 214 4 MI200 (Aldebaran) GPUs using xGMI. 215 216 Some more heterogeneous hardware details: 217 218 - The CPU UMC (Unified Memory Controller) is mostly the same as the GPU UMC. 219 They have chip selects (csrows) and channels. However, the layouts are different 220 for performance, physical layout, or other reasons. 221 - CPU UMCs use 1 channel, In this case UMC = EDAC channel. This follows the 222 marketing speak. CPU has X memory channels, etc. 223 - CPU UMCs use up to 4 chip selects, So UMC chip select = EDAC CSROW. 224 - GPU UMCs use 1 chip select, So UMC = EDAC CSROW. 225 - GPU UMCs use 8 channels, So UMC channel = EDAC channel. 226 227 The EDAC subsystem provides a mechanism to handle AMD heterogeneous 228 systems by calling system specific ops for both CPUs and GPUs. 229 230 AMD GPU nodes are enumerated in sequential order based on the PCI 231 hierarchy, and the first GPU node is assumed to have a Node ID value 232 following those of the CPU nodes after latter are fully populated:: 233 234 $ ls /sys/devices/system/edac/mc/ 235 mc0 - CPU MC node 0 236 mc1 | 237 mc2 |- GPU card[0] => node 0(mc1), node 1(mc2) 238 mc3 | 239 mc4 |- GPU card[1] => node 0(mc3), node 1(mc4) 240 mc5 | 241 mc6 |- GPU card[2] => node 0(mc5), node 1(mc6) 242 mc7 | 243 mc8 |- GPU card[3] => node 0(mc7), node 1(mc8) 244 245 For example, a heterogeneous system with one AMD CPU is connected to 246 four MI200 (Aldebaran) GPUs using xGMI. This topology can be represented 247 via the following sysfs entries:: 248 249 /sys/devices/system/edac/mc/.. 250 251 CPU # CPU node 252 ├── mc 0 253 254 GPU Nodes are enumerated sequentially after CPU nodes have been populated 255 GPU card 1 # Each MI200 GPU has 2 nodes/mcs 256 ├── mc 1 # GPU node 0 == mc1, Each MC node has 4 UMCs/CSROWs 257 │ ├── csrow 0 # UMC 0 258 │ │ ├── channel 0 # Each UMC has 8 channels 259 │ │ ├── channel 1 # size of each channel is 2 GB, so each UMC has 16 GB 260 │ │ ├── channel 2 261 │ │ ├── channel 3 262 │ │ ├── channel 4 263 │ │ ├── channel 5 264 │ │ ├── channel 6 265 │ │ ├── channel 7 266 │ ├── csrow 1 # UMC 1 267 │ │ ├── channel 0 268 │ │ ├── .. 269 │ │ ├── channel 7 270 │ ├── .. .. 271 │ ├── csrow 3 # UMC 3 272 │ │ ├── channel 0 273 │ │ ├── .. 274 │ │ ├── channel 7 275 │ ├── rank 0 276 │ ├── .. .. 277 │ ├── rank 31 # total 32 ranks/dimms from 4 UMCs 278 ├ 279 ├── mc 2 # GPU node 1 == mc2 280 │ ├── .. # each GPU has total 64 GB 281 282 GPU card 2 283 ├── mc 3 284 │ ├── .. 285 ├── mc 4 286 │ ├── .. 287 288 GPU card 3 289 ├── mc 5 290 │ ├── .. 291 ├── mc 6 292 │ ├── .. 293 294 GPU card 4 295 ├── mc 7 296 │ ├── .. 297 ├── mc 8 298 │ ├── ..
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.