1 .. SPDX-License-Identifier: GPL-2.0
2 .. include:: <isonum.txt>
3
4 ==============================================
5 Reliability, Availability and Serviceability (
6 ==============================================
7
8 This documents different aspects of the RAS fu
9 kernel.
10
11 RAS concepts
12 ************
13
14 Reliability, Availability and Serviceability (
15 servers meant to measure their robustness.
16
17 Reliability
18 is the probability that a system will produc
19
20 * Generally measured as Mean Time Between Fa
21 * Enhanced by features that help to avoid, d
22
23 Availability
24 is the probability that a system is operatio
25
26 * Generally measured as a percentage of down
27 * Often uses mechanisms to detect and correc
28 runtime;
29
30 Serviceability (or maintainability)
31 is the simplicity and speed with which a sys
32 maintained
33
34 * Generally measured on Mean Time Between Re
35
36 Improving RAS
37 -------------
38
39 In order to reduce systems downtime, a system
40 hardware errors, and, when possible correcting
41 also provide mechanisms to detect hardware deg
42 the system administrator to take the action of
43 it causes data loss or system downtime.
44
45 Among the monitoring measures, the most usual
46
47 * CPU – detect errors at instruction executi
48 * Memory – add error correction logic (ECC)
49 * I/O – add CRC checksums for transferred da
50 * Storage – RAID, journal file systems, chec
51 Self-Monitoring, Analysis and Reporting Tech
52
53 By monitoring the number of occurrences of err
54 to identify if the probability of hardware err
55 case, do a preventive maintenance to replace a
56 those errors are correctable.
57
58 Types of errors
59 ---------------
60
61 Most mechanisms used on modern systems use tec
62 Codes that allow error correction when the num
63 is below a threshold. If the number of errors
64 can indicate with a high degree of confidence
65 they can't correct.
66
67 Also, sometimes an error occur on a component
68 example, a part of the memory that it is not c
69
70 That defines some categories of errors:
71
72 * **Correctable Error (CE)** - the error detec
73 corrected the error. Such errors are usually
74 Kernel mechanisms allow the system administr
75
76 * **Uncorrected Error (UE)** - the amount of e
77 correction threshold, and the system was una
78
79 * **Fatal Error** - when an UE error happens o
80 system (for example, a piece of the Kernel g
81 only reliable way to avoid data corruption i
82
83 * **Non-fatal Error** - when an UE error happe
84 like a CPU in power down state or an unused
85 still run, eventually replacing the affected
86 if available.
87
88 Also, when an error happens on a userspace p
89 kill such process and let userspace restart
90
91 The mechanism for handling non-fatal errors is
92 require the help of some userspace application
93 policy desired by the system administrator.
94
95 Identifying a bad hardware component
96 ------------------------------------
97
98 Just detecting a hardware flaw is usually not
99 to pinpoint to the minimal replaceable unit (M
100 to make the hardware reliable again.
101
102 So, it requires not only error logging facilit
103 will translate the error message to the silksc
104 the MRU.
105
106 Typically, it is very complex for memory, as m
107 from different memory modules, in order to pro
108 DMI BIOS usually have a list of memory module
109 using the ``dmidecode`` tool. For example, on
110
111 Memory Device
112 Total Width: 64 bits
113 Data Width: 64 bits
114 Size: 16384 MB
115 Form Factor: SODIMM
116 Set: None
117 Locator: ChannelA-DIMM0
118 Bank Locator: BANK 0
119 Type: DDR4
120 Type Detail: Synchronous
121 Speed: 2133 MHz
122 Rank: 2
123 Configured Clock Speed: 2133 M
124
125 On the above example, a DDR4 SO-DIMM memory mo
126 system's memory labeled as "BANK 0", as given
127 Please notice that, on such system, the *total
128 *data width*. It means that such memory module
129 detection/correction mechanisms.
130
131 Unfortunately, not all systems use the same fi
132 bank. On this example, from an older server, `
133
134 Memory Device
135 Array Handle: 0x1000
136 Error Information Handle: Not
137 Total Width: 72 bits
138 Data Width: 64 bits
139 Size: 8192 MB
140 Form Factor: DIMM
141 Set: 1
142 Locator: DIMM_A1
143 Bank Locator: Not Specified
144 Type: DDR3
145 Type Detail: Synchronous Regis
146 Speed: 1600 MHz
147 Rank: 2
148 Configured Clock Speed: 1600 M
149
150 There, the DDR3 RDIMM memory module is located
151 as "DIMM_A1", as given by the *locator* field.
152 memory module has 64 bits of *data width* and
153 it has 8 extra bits to be used by error detect
154 Such kind of memory is called Error-correcting
155
156 To make things even worse, it is not uncommon
157 labels on their system's board to use exactly
158 the labels provided by the BIOS won't match th
159
160 ECC memory
161 ----------
162
163 As mentioned in the previous section, ECC memo
164 used for error correction. In the above exampl
165 64 bits of *data width*, and 72 bits of *total
166 bits which are used for the error detection an
167 are referred to as the *syndrome*\ [#f1]_\ [#f
168
169 So, when the cpu requests the memory controlle
170 *data width*, the memory controller calculates
171 using Hamming code, or some other error correc
172 producing a code with *total width* size. Such
173 on the memory modules.
174
175 At read, the *total width* bits code is conver
176 ECC code used on write, producing a word with
177 The word with *data width* is sent to the CPU,
178
179 The memory controller also looks at the *syndr
180 there was an error, and if the ECC code was ab
181 If the error was corrected, a Corrected Error
182 Uncorrected Error (UE) happened.
183
184 The information about the CE/UE errors is stor
185 at the memory controller and can be accessed b
186 either by BIOS, by some special CPUs or by Lin
187 bit CPUs, such errors can also be retrieved vi
188 Architecture (MCA)\ [#f3]_.
189
190 .. [#f1] Please notice that several memory con
191 mode called "Lock-Step", where it groups two
192 doing 128-bit reads/writes. That gives 16 bi
193 significantly improves the error correction
194 that, when an error happens, there's no way
195 to blame. So, it has to blame both memory mo
196
197 .. [#f2] Some memory controllers also allow us
198 On such mode, the same data is written to tw
199 the system checks both memory modules, in or
200 identical data. On such configuration, when
201 way to know what memory module is to blame.
202 memory modules (or 4 memory modules, if the
203 mode).
204
205 .. [#f3] For more details about the Machine Ch
206 please read Documentation/arch/x86/x86_64/ma
207
208 EDAC - Error Detection And Correction
209 *************************************
210
211 .. note::
212
213 "bluesmoke" was the name for this device dr
214 was "out-of-tree" and maintained at http://
215 That site is mostly archaic now and can be
216 purposes.
217
218 When the subsystem was pushed upstream for
219 Kernel 2.6.16, it was renamed to ``EDAC``.
220
221 Purpose
222 -------
223
224 The ``edac`` kernel module's goal is to detect
225 that occur within the computer system running
226
227 Memory
228 ------
229
230 Memory Correctable Errors (CE) and Uncorrectab
231 primary errors being harvested. These types of
232 the ``edac_mc`` device.
233
234 Detecting CE events, then harvesting those eve
235 **can** but must not necessarily be a predicto
236 CE events only, the system can and will contin
237 has been damaged yet.
238
239 However, preventive maintenance and proactive
240 modules exhibiting CEs can reduce the likeliho
241 and system panics.
242
243 Other hardware elements
244 -----------------------
245
246 A new feature for EDAC, the ``edac_device`` cl
247 the 2.6.23 version of the kernel.
248
249 This new device type allows for non-memory typ
250 to have their states harvested and presented t
251 interface.
252
253 Some architectures have ECC detectors for L1,
254 along with DMA engines, fabric switches, main
255 interconnections, and various other hardware d
256 reports it, then a edac_device device probably
257 harvest and present that to userspace.
258
259
260 PCI bus scanning
261 ----------------
262
263 In addition, PCI devices are scanned for PCI B
264 in order to determine if errors are occurring
265
266 The presence of PCI Parity errors must be exam
267 There are several add-in adapters that do **no
268 with regards to Parity generation and reportin
269 the vendor should tie the parity status bits t
270 to generate parity. Some vendors do not do th
271 can "float" giving false positives.
272
273 There is a PCI device attribute located in sys
274 the EDAC PCI scanning code. If that attribute
275 scanning is skipped for that device. The attri
276
277 broken_parity_status
278
279 and is located in ``/sys/devices/pci<XXX>/0000
280 PCI devices.
281
282
283 Versioning
284 ----------
285
286 EDAC is composed of a "core" module (``edac_co
287 Controller (MC) driver modules. On a given sys
288 and one MC driver will be loaded. Both the COR
289 ``edac_device`` driver) have individual versio
290 release level of their respective modules.
291
292 Thus, to "report" on what version a system is
293 both the CORE's and the MC driver's versions.
294
295
296 Loading
297 -------
298
299 If ``edac`` was statically linked with the ker
300 is necessary. If ``edac`` was built as modules
301 the ``edac`` pieces that you need. You should
302 hardware-specific modules and have the depende
303 core modules.
304
305 Example::
306
307 $ modprobe amd76x_edac
308
309 loads both the ``amd76x_edac.ko`` memory contr
310 ``edac_mc.ko`` core module.
311
312
313 Sysfs interface
314 ---------------
315
316 EDAC presents a ``sysfs`` interface for contro
317 lives in the /sys/devices/system/edac director
318
319 Within this directory there currently reside 2
320
321 ======= ==============================
322 mc memory controller(s) system
323 pci PCI control and status system
324 ======= ==============================
325
326
327
328 Memory Controller (mc) Model
329 ----------------------------
330
331 Each ``mc`` device controls a set of memory mo
332 are laid out in a Chip-Select Row (``csrowX``)
333 There can be multiple csrows and multiple chan
334
335 .. [#f4] Nowadays, the term DIMM (Dual In-line
336 used to refer to a memory module, although t
337 packaging alternatives, like SO-DIMM, SIMM,
338 specification (Version 2.7) defines a memory
339 Platform Error Record (CPER) section to be a
340 (Type 17). Along this document, and inside t
341 "dimm" is used for all memory modules, even
342 different kind of packaging.
343
344 Memory controllers allow for several csrows, w
345 typical value. Yet, the actual number of csrow
346 a given motherboard, memory controller and mem
347
348 Dual channels allow for dual data length (e. g
349 data transfers to/from the CPU from/to memory.
350 for more than 2 channels, like Fully Buffered
351 controllers. The following example will assume
352
353 +------------+-----------------------+
354 | CS Rows | Channels |
355 +------------+-----------+-----------+
356 | | ``ch0`` | ``ch1`` |
357 +============+===========+===========+
358 | |**DIMM_A0**|**DIMM_B0**|
359 +------------+-----------+-----------+
360 | ``csrow0`` | rank0 | rank0 |
361 +------------+-----------+-----------+
362 | ``csrow1`` | rank1 | rank1 |
363 +------------+-----------+-----------+
364 | |**DIMM_A1**|**DIMM_B1**|
365 +------------+-----------+-----------+
366 | ``csrow2`` | rank0 | rank0 |
367 +------------+-----------+-----------+
368 | ``csrow3`` | rank1 | rank1 |
369 +------------+-----------+-----------+
370
371 In the above example, there are 4 physical slo
372 for memory DIMMs:
373
374 +---------+---------+
375 | DIMM_A0 | DIMM_B0 |
376 +---------+---------+
377 | DIMM_A1 | DIMM_B1 |
378 +---------+---------+
379
380 Labels for these slots are usually silk-screen
381 Slots labeled ``A`` are channel 0 in this exam
382 channel 1. Notice that there are two csrows po
383 These csrows are allocated their csrow assignm
384 which the memory DIMM is placed. Thus, when 1
385 Channel, the csrows cross both DIMMs.
386
387 Memory DIMMs come single or dual "ranked". A r
388 In the example above 2 dual ranked DIMMs are s
389 both csrow0 and csrow1 are populated. On the o
390 ranked DIMMs are placed in slots DIMM_A0 and D
391 have just one csrow (csrow0) and csrow1 will b
392 repeats itself for csrow2 and csrow3. Also not
393 controllers don't have any logic to identify t
394 ``rankX`` directories below.
395
396 The representation of the above is reflected i
397 tree in EDAC's sysfs interface. Starting in di
398 ``/sys/devices/system/edac/mc``, each memory c
399 represented by its own ``mcX`` directory, wher
400 index of the MC::
401
402 ..../edac/mc/
403 |
404 |->mc0
405 |->mc1
406 |->mc2
407 ....
408
409 Under each ``mcX`` directory each ``csrowX`` i
410 ``csrowX``, where ``X`` is the csrow index::
411
412 .../mc/mc0/
413 |
414 |->csrow0
415 |->csrow2
416 |->csrow3
417 ....
418
419 Notice that there is no csrow1, which indicate
420 of a single ranked DIMMs. This should also app
421 order to have dual-channel mode be operational
422 csrow3 are populated, this indicates a dual ra
423 channels 0 and 1.
424
425 Within each of the ``mcX`` and ``csrowX`` dire
426 control and attribute files.
427
428 ``mcX`` directories
429 -------------------
430
431 In ``mcX`` directories are EDAC control and at
432 this ``X`` instance of the memory controllers.
433
434 For a description of the sysfs API, please see
435
436 Documentation/ABI/testing/sysfs-device
437
438
439 ``dimmX`` or ``rankX`` directories
440 ----------------------------------
441
442 The recommended way to use the EDAC subsystem
443 provided by the ``dimmX`` or ``rankX`` directo
444
445 A typical EDAC system has the following struct
446 ``/sys/devices/system/edac/``\ [#f6]_::
447
448 /sys/devices/system/edac/
449 ├── mc
450 │ ├── mc0
451 │ │ ├── ce_count
452 │ │ ├── ce_noinfo_co
453 │ │ ├── dimm0
454 │ │ │ ├── dimm
455 │ │ │ ├── dimm
456 │ │ │ ├── dimm
457 │ │ │ ├── dimm
458 │ │ │ ├── dimm
459 │ │ │ ├── dimm
460 │ │ │ ├── dimm
461 │ │ │ ├── size
462 │ │ │ └── ueve
463 │ │ ├── max_location
464 │ │ ├── mc_name
465 │ │ ├── reset_counte
466 │ │ ├── seconds_sinc
467 │ │ ├── size_mb
468 │ │ ├── ue_count
469 │ │ ├── ue_noinfo_co
470 │ │ └── uevent
471 │ ├── mc1
472 │ │ ├── ce_count
473 │ │ ├── ce_noinfo_co
474 │ │ ├── dimm0
475 │ │ │ ├── dimm
476 │ │ │ ├── dimm
477 │ │ │ ├── dimm
478 │ │ │ ├── dimm
479 │ │ │ ├── dimm
480 │ │ │ ├── dimm
481 │ │ │ ├── dimm
482 │ │ │ ├── size
483 │ │ │ └── ueve
484 │ │ ├── max_location
485 │ │ ├── mc_name
486 │ │ ├── reset_counte
487 │ │ ├── seconds_sinc
488 │ │ ├── size_mb
489 │ │ ├── ue_count
490 │ │ ├── ue_noinfo_co
491 │ │ └── uevent
492 │ └── uevent
493 └── uevent
494
495 In the ``dimmX`` directories are EDAC control
496 this ``X`` memory module:
497
498 - ``size`` - Total memory managed by this csro
499
500 This attribute file displays, in count
501 that this csrow contains.
502
503 - ``dimm_ue_count`` - Uncorrectable Errors cou
504
505 This attribute file displays the total
506 errors that have occurred on this DIMM
507 this counter will not have a chance to
508 will panic the system.
509
510 - ``dimm_ce_count`` - Correctable Errors count
511
512 This attribute file displays the total
513 errors that have occurred on this DIMM
514 important to examine. CEs provide earl
515 DIMM is beginning to fail. This count
516 monitored for non-zero values and repo
517 to the system administrator.
518
519 - ``dimm_dev_type`` - Device type attribute f
520
521 This attribute file will display what
522 being utilized on this DIMM.
523 Examples:
524
525 - x1
526 - x2
527 - x4
528 - x8
529
530 - ``dimm_edac_mode`` - EDAC Mode of operation
531
532 This attribute file will display what
533 and correction is being utilized.
534
535 - ``dimm_label`` - memory module label control
536
537 This control file allows this DIMM to
538 to it. With this label in the module,
539 the output can provide the DIMM label
540 This becomes vital for panic events to
541 cause of the UE event.
542
543 DIMM Labels must be assigned after boo
544 that correctly identifies the physical
545 silk screen label. This information is
546 motherboard specific and determination
547 must occur in userland at this time.
548
549 - ``dimm_location`` - location of the memory m
550
551 The location can have up to 3 levels,
552 memory controller identifies the locat
553 Depending on the type of memory and me
554 can be:
555
556 - *csrow* and *channel* - used
557 doesn't identify a single DI
558 - *branch*, *channel*, *slot*
559 controllers;
560 - *channel*, *slot* - used on
561
562 - ``dimm_mem_type`` - Memory Type attribute fi
563
564 This attribute file will display what
565 on this csrow. Normally, either buffer
566 Examples:
567
568 - Registered-DDR
569 - Unbuffered-DDR
570
571 .. [#f5] On some systems, the memory controlle
572 to identify the memory module. On such syste
573 On modern Intel memory controllers, the memo
574 memory modules directly. On such systems, th
575
576 .. [#f6] There are also some ``power`` directo
577 symlinks inside the sysfs mapping that are a
578 the sysfs subsystem. Currently, they serve n
579
580 ``csrowX`` directories
581 ----------------------
582
583 When CONFIG_EDAC_LEGACY_SYSFS is enabled, sysf
584 directories. As this API doesn't work properly
585 modern Intel Memory Controllers, this is being
586 ``dimmX`` directories.
587
588 In the ``csrowX`` directories are EDAC control
589 this ``X`` instance of csrow:
590
591
592 - ``ue_count`` - Total Uncorrectable Errors co
593
594 This attribute file displays the total
595 errors that have occurred on this csro
596 this counter will not have a chance to
597 will panic the system.
598
599
600 - ``ce_count`` - Total Correctable Errors coun
601
602 This attribute file displays the total
603 errors that have occurred on this csro
604 important to examine. CEs provide earl
605 DIMM is beginning to fail. This count
606 monitored for non-zero values and repo
607 to the system administrator.
608
609
610 - ``size_mb`` - Total memory managed by this c
611
612 This attribute file displays, in count
613 that this csrow contains.
614
615
616 - ``mem_type`` - Memory Type attribute file
617
618 This attribute file will display what
619 on this csrow. Normally, either buffer
620 Examples:
621
622 - Registered-DDR
623 - Unbuffered-DDR
624
625
626 - ``edac_mode`` - EDAC Mode of operation attri
627
628 This attribute file will display what
629 and correction is being utilized.
630
631
632 - ``dev_type`` - Device type attribute file
633
634 This attribute file will display what
635 being utilized on this DIMM.
636 Examples:
637
638 - x1
639 - x2
640 - x4
641 - x8
642
643
644 - ``ch0_ce_count`` - Channel 0 CE Count attrib
645
646 This attribute file will display the c
647 DIMM located in channel 0.
648
649
650 - ``ch0_ue_count`` - Channel 0 UE Count attrib
651
652 This attribute file will display the c
653 DIMM located in channel 0.
654
655
656 - ``ch0_dimm_label`` - Channel 0 DIMM Label co
657
658
659 This control file allows this DIMM to
660 to it. With this label in the module,
661 the output can provide the DIMM label
662 This becomes vital for panic events to
663 cause of the UE event.
664
665 DIMM Labels must be assigned after boo
666 that correctly identifies the physical
667 silk screen label. This information is
668 motherboard specific and determination
669 must occur in userland at this time.
670
671
672 - ``ch1_ce_count`` - Channel 1 CE Count attrib
673
674
675 This attribute file will display the c
676 DIMM located in channel 1.
677
678
679 - ``ch1_ue_count`` - Channel 1 UE Count attrib
680
681
682 This attribute file will display the c
683 DIMM located in channel 0.
684
685
686 - ``ch1_dimm_label`` - Channel 1 DIMM Label co
687
688 This control file allows this DIMM to
689 to it. With this label in the module,
690 the output can provide the DIMM label
691 This becomes vital for panic events to
692 cause of the UE event.
693
694 DIMM Labels must be assigned after boo
695 that correctly identifies the physical
696 silk screen label. This information is
697 motherboard specific and determination
698 must occur in userland at this time.
699
700
701 System Logging
702 --------------
703
704 If logging for UEs and CEs is enabled, then sy
705 information indicating that errors have been d
706
707 EDAC MC0: CE page 0x283, offset 0xce0, grain
708 EDAC MC0: CE page 0x1e5, offset 0xfb0, grain
709
710
711 The structure of the message is:
712
713 +-------------------------------------
714 | Content
715 +=====================================
716 | The memory controller
717 +-------------------------------------
718 | Error type
719 +-------------------------------------
720 | Memory page
721 +-------------------------------------
722 | Offset in the page
723 +-------------------------------------
724 | The byte granularity
725 | or resolution of the error
726 +-------------------------------------
727 | The error syndrome
728 +-------------------------------------
729 | Memory row
730 +-------------------------------------
731 | Memory channel
732 +-------------------------------------
733 | DIMM label, if set prior
734 +-------------------------------------
735 | And then an optional, driver-specifi
736 | message that may have additional
737 | information.
738 +-------------------------------------
739
740 Both UEs and CEs with no info will lack all bu
741 type, a notice of "no info" and then an option
742 message.
743
744
745 PCI Bus Parity Detection
746 ------------------------
747
748 On Header Type 00 devices, the primary status
749 parity error regardless of whether parity is e
750 not. (The spec indicates parity is generated i
751 Type 01 bridges, the secondary status register
752 if parity occurred on the bus on the other sid
753
754
755 Sysfs configuration
756 -------------------
757
758 Under ``/sys/devices/system/edac/pci`` are con
759 follows:
760
761
762 - ``check_pci_parity`` - Enable/Disable PCI Pa
763
764 This control file enables or disables
765 operation. Writing a 1 to this file en
766 a 0 to this file disables the scanning
767
768 Enable::
769
770 echo "1" >/sys/devices/system/
771
772 Disable::
773
774 echo "0" >/sys/devices/system/
775
776
777 - ``pci_parity_count`` - Parity Count
778
779 This attribute file will display the n
780 have been detected.
781
782
783 Module parameters
784 -----------------
785
786 - ``edac_mc_panic_on_ue`` - Panic on UE contro
787
788 An uncorrectable error will cause a ma
789 desirable. It is a bad idea to contin
790 occurs - it is indeterminate what was
791 system context might be so mangled tha
792 corruption. If the kernel has MCE conf
793 notice the UE.
794
795 LOAD TIME::
796
797 module/kernel parameter: edac_
798
799 RUN TIME::
800
801 echo "1" > /sys/module/edac_co
802
803
804 - ``edac_mc_log_ue`` - Log UE control file
805
806
807 Generate kernel messages describing un
808 are reported through the system messag
809 will be accumulated even when UE loggi
810
811 LOAD TIME::
812
813 module/kernel parameter: edac_
814
815 RUN TIME::
816
817 echo "1" > /sys/module/edac_co
818
819
820 - ``edac_mc_log_ce`` - Log CE control file
821
822
823 Generate kernel messages describing co
824 errors are reported through the system
825 CE statistics will be accumulated even
826
827 LOAD TIME::
828
829 module/kernel parameter: edac_
830
831 RUN TIME::
832
833 echo "1" > /sys/module/edac_co
834
835
836 - ``edac_mc_poll_msec`` - Polling period contr
837
838
839 The time period, in milliseconds, for
840 Too small a value wastes resources. T
841 necessary handling of errors and might
842 locating the error. 1000 milliseconds
843 default. Systems which require all the
844 increase this.
845
846 LOAD TIME::
847
848 module/kernel parameter: edac_
849
850 RUN TIME::
851
852 echo "1000" > /sys/module/edac
853
854
855 - ``panic_on_pci_parity`` - Panic on PCI PARIT
856
857
858 This control file enables or disables
859 error has been detected.
860
861
862 module/kernel parameter::
863
864 edac_panic_on_pci_pe=[
865
866 Enable::
867
868 echo "1" > /sys/module/edac_co
869
870 Disable::
871
872 echo "0" > /sys/module/edac_co
873
874
875
876 EDAC device type
877 ----------------
878
879 In the header file, edac_pci.h, there is a ser
880 and APIs for the EDAC_DEVICE.
881
882 User space access to an edac_device is through
883
884 At the location ``/sys/devices/system/edac`` (
885 will appear.
886
887 There is a three level tree beneath the above
888 the ``test_device_edac`` device (found at the
889 website) installs itself as::
890
891 /sys/devices/system/edac/test-instance
892
893 in this directory are various controls, a syml
894 directories.
895
896 The standard default controls are:
897
898 ============== ======================
899 log_ce boolean to log CE even
900 log_ue boolean to log UE even
901 panic_on_ue boolean to ``panic`` t
902 (default off, can be s
903 poll_msec time period between PO
904 ============== ======================
905
906 The test_device_edac device adds at least one
907
908 ============== ======================
909 test_bits which in the current t
910 show how it is install
911 add one or more such c
912 for specific uses.
913 One out-of-tree driver
914 for ERROR INJECTION op
915 injection registers
916 ============== ======================
917
918 The symlink points to the 'struct dev' that is
919
920 Instances
921 ---------
922
923 One or more instance directories are present.
924 case:
925
926 +----------------+
927 | test-instance0 |
928 +----------------+
929
930
931 In this directory there are two default counte
932 counter in deeper subdirectories.
933
934 ============== ======================
935 ce_count total of CE events of
936 ue_count total of UE events of
937 ============== ======================
938
939 Blocks
940 ------
941
942 At the lowest directory level is the ``block``
943 or more blocks specified in each instance:
944
945 +-------------+
946 | test-block0 |
947 +-------------+
948
949 In this directory the default attributes are:
950
951 ============== ======================
952 ce_count which is counter of CE
953 of hardware being moni
954 ue_count which is counter of UE
955 of hardware being moni
956 ============== ======================
957
958
959 The ``test_device_edac`` device adds 4 attribu
960
961 ================== ===================
962 test-block-bits-0 for every POLL
963 is incremented
964 test-block-bits-1 every 10 cycle
965 and test-block
966 test-block-bits-2 every 100 cycl
967 and test-block
968 test-block-bits-3 every 1000 cyc
969 and test-block
970 ================== ===================
971
972
973 ================== ===================
974 reset-counters writing ANY th
975 reset all the
976 ================== ===================
977
978
979 Use of the ``test_device_edac`` driver should
980 unique drivers for their hardware systems.
981
982 The ``test_device_edac`` sample driver is loca
983 http://bluesmoke.sourceforge.net project site
984
985
986 Usage of EDAC APIs on Nehalem and newer Intel
987 ----------------------------------------------
988
989 On older Intel architectures, the memory contr
990 Bridge chipset. Nehalem, Sandy Bridge, Ivy Bri
991 newer Intel architectures integrated an enhanc
992 controller (MC) inside the CPUs.
993
994 This chapter will cover the differences of the
995 found on newer Intel CPUs, such as ``i7core_ed
996 ``sbx_edac`` drivers.
997
998 .. note::
999
1000 The Xeon E7 processor families use a separ
1001 controller, called Intel Scalable Memory B
1002 apply for such families.
1003
1004 1) There is one Memory Controller per Quick P
1005 (QPI). At the driver, the term "socket" me
1006 associated with a physical CPU socket.
1007
1008 Each MC have 3 physical read channels, 3 p
1009 3 logic channels. The driver currently see
1010 Each channel can have up to 3 DIMMs.
1011
1012 The minimum known unity is DIMMs. There ar
1013 As EDAC API maps the minimum unity is csro
1014 maps channel/DIMM into different csrows.
1015
1016 For example, supposing the following layou
1017
1018 Ch0 phy rd0, wr0 (0x063f4031): 2 rank
1019 dimm 0 1024 Mb offset: 0, bank: 8,
1020 dimm 1 1024 Mb offset: 4, bank: 8,
1021 Ch1 phy rd1, wr1 (0x063f4031): 2 rank
1022 dimm 0 1024 Mb offset: 0, bank: 8,
1023 Ch2 phy rd3, wr3 (0x063f4031): 2 rank
1024 dimm 0 1024 Mb offset: 0, bank: 8,
1025
1026 The driver will map it as::
1027
1028 csrow0: channel 0, dimm0
1029 csrow1: channel 0, dimm1
1030 csrow2: channel 1, dimm0
1031 csrow3: channel 2, dimm0
1032
1033 exports one DIMM per csrow.
1034
1035 Each QPI is exported as a different memory
1036
1037 2) The MC has the ability to inject errors to
1038 implement this functionality via some erro
1039
1040 For injecting a memory error, there are so
1041 ``/sys/devices/system/edac/mc/mc?/``:
1042
1043 - ``inject_addrmatch/*``:
1044 Controls the error injection mask regis
1045 several characteristics of the address
1046
1047 dimm = the affected dimm. Numbers ar
1048 rank = the memory rank;
1049 channel = the channel that will gene
1050 bank = the affected bank;
1051 page = the page address;
1052 column (or col) = the address column
1053
1054 each of the above values can be set to
1055
1056 At driver init, all values are set to a
1057
1058 For example, to generate an error at ra
1059 any bank, any page, any column::
1060
1061 echo 2 >/sys/devices/system/e
1062 echo 1 >/sys/devices/system/e
1063
1064 To return to the default behaviour of
1065
1066 echo any >/sys/devices/system
1067 echo any >/sys/devices/system
1068
1069 - ``inject_eccmask``:
1070 specifies what bits will have troub
1071
1072 - ``inject_section``:
1073 specifies what ECC cache section will
1074
1075 3 for both
1076 2 for the highest
1077 1 for the lowest
1078
1079 - ``inject_type``:
1080 specifies the type of error, being a c
1081
1082 bit 0 - repeat
1083 bit 1 - ecc
1084 bit 2 - parity
1085
1086 - ``inject_enable``:
1087 starts the error generation when somet
1088
1089 All inject vars can be read. root permissi
1090
1091 Datasheet states that the error will only
1092 address that matches inject_addrmatch. It
1093 also produce an error.
1094
1095 For example, the following code will gener
1096 at socket 0, on any DIMM/address on channe
1097
1098 echo 2 >/sys/devices/system/edac/mc/m
1099 echo 2 >/sys/devices/system/edac/mc/m
1100 echo 64 >/sys/devices/system/edac/mc/
1101 echo 3 >/sys/devices/system/edac/mc/m
1102 echo 1 >/sys/devices/system/edac/mc/m
1103 dd if=/dev/mem of=/dev/null seek=16k
1104
1105 For socket 1, it is needed to replace "mc0
1106 commands.
1107
1108 The generated error message will look like
1109
1110 EDAC MC0: UE row 0, channel-a= 0 chan
1111
1112 3) Corrected Error memory register counters
1113
1114 Those newer MCs have some registers to cou
1115 uses those registers to report Corrected E
1116 DIMMs.
1117
1118 However, those counters don't work with Un
1119 offers some counters that also work with U
1120 granularity than the default ones), the dr
1121 UDIMM memories.
1122
1123 They can be read by looking at the content
1124
1125 $ for i in /sys/devices/system/edac/mc/m
1126 /sys/devices/system/edac/mc/mc0/all_c
1127 0
1128 /sys/devices/system/edac/mc/mc0/all_c
1129 0
1130 /sys/devices/system/edac/mc/mc0/all_c
1131 0
1132
1133 What happens here is that errors on differ
1134 dimm number will increment the same counte
1135 So, in this memory mapping::
1136
1137 csrow0: channel 0, dimm0
1138 csrow1: channel 0, dimm1
1139 csrow2: channel 1, dimm0
1140 csrow3: channel 2, dimm0
1141
1142 The hardware will increment udimm0 for an
1143 csrow0, csrow2 or csrow3;
1144
1145 The hardware will increment udimm1 for an
1146 csrow0, csrow2 or csrow3;
1147
1148 The hardware will increment udimm2 for an
1149 csrow0, csrow2 or csrow3;
1150
1151 4) Standard error counters
1152
1153 The standard error counters are generated
1154 by the driver. Since, with UDIMM, this is
1155 possible that some errors could be lost. W
1156 contents of the registers
1157
1158 Reference documents used on ``amd64_edac``
1159 ------------------------------------------
1160
1161 ``amd64_edac`` module is based on the followi
1162 (available from http://support.amd.com/en-us/
1163
1164 1. :Title: BIOS and Kernel Developer's Guide
1165 Opteron Processors
1166 :AMD publication #: 26094
1167 :Revision: 3.26
1168 :Link: http://support.amd.com/TechDocs/260
1169
1170 2. :Title: BIOS and Kernel Developer's Guide
1171 Processors
1172 :AMD publication #: 32559
1173 :Revision: 3.00
1174 :Issue Date: May 2006
1175 :Link: http://support.amd.com/TechDocs/325
1176
1177 3. :Title: BIOS and Kernel Developer's Guide
1178 Processors
1179 :AMD publication #: 31116
1180 :Revision: 3.00
1181 :Issue Date: September 07, 2007
1182 :Link: http://support.amd.com/TechDocs/311
1183
1184 4. :Title: BIOS and Kernel Developer's Guide
1185 Models 30h-3Fh Processors
1186 :AMD publication #: 49125
1187 :Revision: 3.06
1188 :Issue Date: 2/12/2015 (latest release)
1189 :Link: http://support.amd.com/TechDocs/491
1190
1191 5. :Title: BIOS and Kernel Developer's Guide
1192 Models 60h-6Fh Processors
1193 :AMD publication #: 50742
1194 :Revision: 3.01
1195 :Issue Date: 7/23/2015 (latest release)
1196 :Link: http://support.amd.com/TechDocs/507
1197
1198 6. :Title: BIOS and Kernel Developer's Guide
1199 Models 00h-0Fh Processors
1200 :AMD publication #: 48751
1201 :Revision: 3.03
1202 :Issue Date: 2/23/2015 (latest release)
1203 :Link: http://support.amd.com/TechDocs/487
1204
1205 Credits
1206 =======
1207
1208 * Written by Doug Thompson <dougthompson@xmiss
1209
1210 - 7 Dec 2005
1211 - 17 Jul 2007 Updated
1212
1213 * |copy| Mauro Carvalho Chehab
1214
1215 - 05 Aug 2009 Nehalem interface
1216 - 26 Oct 2016 Converted to ReST and cleanup
1217
1218 * EDAC authors/maintainers:
1219
1220 - Doug Thompson, Dave Jiang, Dave Peterson
1221 - Mauro Carvalho Chehab
1222 - Borislav Petkov
1223 - original author: Thayne Harbaugh
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.