1 Ramoops oops/panic logger 1 Ramoops oops/panic logger
2 ========================= 2 =========================
3 3
4 Sergiu Iordache <sergiu@chromium.org> 4 Sergiu Iordache <sergiu@chromium.org>
5 5
6 Updated: 10 Feb 2021 6 Updated: 10 Feb 2021
7 7
8 Introduction 8 Introduction
9 ------------ 9 ------------
10 10
11 Ramoops is an oops/panic logger that writes it 11 Ramoops is an oops/panic logger that writes its logs to RAM before the system
12 crashes. It works by logging oopses and panics 12 crashes. It works by logging oopses and panics in a circular buffer. Ramoops
13 needs a system with persistent RAM so that the 13 needs a system with persistent RAM so that the content of that area can
14 survive after a restart. 14 survive after a restart.
15 15
16 Ramoops concepts 16 Ramoops concepts
17 ---------------- 17 ----------------
18 18
19 Ramoops uses a predefined memory area to store 19 Ramoops uses a predefined memory area to store the dump. The start and size
20 and type of the memory area are set using thre 20 and type of the memory area are set using three variables:
21 21
22 * ``mem_address`` for the start 22 * ``mem_address`` for the start
23 * ``mem_size`` for the size. The memory size 23 * ``mem_size`` for the size. The memory size will be rounded down to a
24 power of two. 24 power of two.
25 * ``mem_type`` to specify if the memory type 25 * ``mem_type`` to specify if the memory type (default is pgprot_writecombine).
26 * ``mem_name`` to specify a memory region de 26 * ``mem_name`` to specify a memory region defined by ``reserve_mem`` command
27 line parameter. 27 line parameter.
28 28
29 Typically the default value of ``mem_type=0`` 29 Typically the default value of ``mem_type=0`` should be used as that sets the pstore
30 mapping to pgprot_writecombine. Setting ``mem_ 30 mapping to pgprot_writecombine. Setting ``mem_type=1`` attempts to use
31 ``pgprot_noncached``, which only works on some 31 ``pgprot_noncached``, which only works on some platforms. This is because pstore
32 depends on atomic operations. At least on ARM, 32 depends on atomic operations. At least on ARM, pgprot_noncached causes the
33 memory to be mapped strongly ordered, and atom 33 memory to be mapped strongly ordered, and atomic operations on strongly ordered
34 memory are implementation defined, and won't w 34 memory are implementation defined, and won't work on many ARMs such as omaps.
35 Setting ``mem_type=2`` attempts to treat the m 35 Setting ``mem_type=2`` attempts to treat the memory region as normal memory,
36 which enables full cache on it. This can impro 36 which enables full cache on it. This can improve the performance.
37 37
38 The memory area is divided into ``record_size` 38 The memory area is divided into ``record_size`` chunks (also rounded down to
39 power of two) and each kmesg dump writes a ``r 39 power of two) and each kmesg dump writes a ``record_size`` chunk of
40 information. 40 information.
41 41
42 Limiting which kinds of kmsg dumps are stored 42 Limiting which kinds of kmsg dumps are stored can be controlled via
43 the ``max_reason`` value, as defined in includ 43 the ``max_reason`` value, as defined in include/linux/kmsg_dump.h's
44 ``enum kmsg_dump_reason``. For example, to sto 44 ``enum kmsg_dump_reason``. For example, to store both Oopses and Panics,
45 ``max_reason`` should be set to 2 (KMSG_DUMP_O 45 ``max_reason`` should be set to 2 (KMSG_DUMP_OOPS), to store only Panics
46 ``max_reason`` should be set to 1 (KMSG_DUMP_P 46 ``max_reason`` should be set to 1 (KMSG_DUMP_PANIC). Setting this to 0
47 (KMSG_DUMP_UNDEF), means the reason filtering 47 (KMSG_DUMP_UNDEF), means the reason filtering will be controlled by the
48 ``printk.always_kmsg_dump`` boot param: if uns 48 ``printk.always_kmsg_dump`` boot param: if unset, it'll be KMSG_DUMP_OOPS,
49 otherwise KMSG_DUMP_MAX. 49 otherwise KMSG_DUMP_MAX.
50 50
51 The module uses a counter to record multiple d 51 The module uses a counter to record multiple dumps but the counter gets reset
52 on restart (i.e. new dumps after the restart w 52 on restart (i.e. new dumps after the restart will overwrite old ones).
53 53
54 Ramoops also supports software ECC protection 54 Ramoops also supports software ECC protection of persistent memory regions.
55 This might be useful when a hardware reset was 55 This might be useful when a hardware reset was used to bring the machine back
56 to life (i.e. a watchdog triggered). In such c 56 to life (i.e. a watchdog triggered). In such cases, RAM may be somewhat
57 corrupt, but usually it is restorable. 57 corrupt, but usually it is restorable.
58 58
59 Setting the parameters 59 Setting the parameters
60 ---------------------- 60 ----------------------
61 61
62 Setting the ramoops parameters can be done in 62 Setting the ramoops parameters can be done in several different manners:
63 63
64 A. Use the module parameters (which have the 64 A. Use the module parameters (which have the names of the variables described
65 as before). For quick debugging, you can also 65 as before). For quick debugging, you can also reserve parts of memory during
66 boot and then use the reserved memory for ram 66 boot and then use the reserved memory for ramoops. For example, assuming a
67 machine with > 128 MB of memory, the followin 67 machine with > 128 MB of memory, the following kernel command line will tell
68 the kernel to use only the first 128 MB of me 68 the kernel to use only the first 128 MB of memory, and place ECC-protected
69 ramoops region at 128 MB boundary:: 69 ramoops region at 128 MB boundary::
70 70
71 mem=128M ramoops.mem_address=0x8000000 71 mem=128M ramoops.mem_address=0x8000000 ramoops.ecc=1
72 72
73 B. Use Device Tree bindings, as described in 73 B. Use Device Tree bindings, as described in
74 ``Documentation/devicetree/bindings/reserved- 74 ``Documentation/devicetree/bindings/reserved-memory/ramoops.yaml``.
75 For example:: 75 For example::
76 76
77 reserved-memory { 77 reserved-memory {
78 #address-cells = <2>; 78 #address-cells = <2>;
79 #size-cells = <2>; 79 #size-cells = <2>;
80 ranges; 80 ranges;
81 81
82 ramoops@8f000000 { 82 ramoops@8f000000 {
83 compatible = "ramoops" 83 compatible = "ramoops";
84 reg = <0 0x8f000000 0 84 reg = <0 0x8f000000 0 0x100000>;
85 record-size = <0x4000> 85 record-size = <0x4000>;
86 console-size = <0x4000 86 console-size = <0x4000>;
87 }; 87 };
88 }; 88 };
89 89
90 C. Use a platform device and set the platform 90 C. Use a platform device and set the platform data. The parameters can then
91 be set through that platform data. An example 91 be set through that platform data. An example of doing that is:
92 92
93 .. code-block:: c 93 .. code-block:: c
94 94
95 #include <linux/pstore_ram.h> 95 #include <linux/pstore_ram.h>
96 [...] 96 [...]
97 97
98 static struct ramoops_platform_data ramoops_ 98 static struct ramoops_platform_data ramoops_data = {
99 .mem_size = <...>, 99 .mem_size = <...>,
100 .mem_address = <...>, 100 .mem_address = <...>,
101 .mem_type = <...>, 101 .mem_type = <...>,
102 .record_size = <...>, 102 .record_size = <...>,
103 .max_reason = <...>, 103 .max_reason = <...>,
104 .ecc = <...>, 104 .ecc = <...>,
105 }; 105 };
106 106
107 static struct platform_device ramoops_dev = 107 static struct platform_device ramoops_dev = {
108 .name = "ramoops", 108 .name = "ramoops",
109 .dev = { 109 .dev = {
110 .platform_data = &ramoops_data 110 .platform_data = &ramoops_data,
111 }, 111 },
112 }; 112 };
113 113
114 [... inside a function ...] 114 [... inside a function ...]
115 int ret; 115 int ret;
116 116
117 ret = platform_device_register(&ramoops_dev) 117 ret = platform_device_register(&ramoops_dev);
118 if (ret) { 118 if (ret) {
119 printk(KERN_ERR "unable to register pl 119 printk(KERN_ERR "unable to register platform device\n");
120 return ret; 120 return ret;
121 } 121 }
122 122
123 D. Using a region of memory reserved via ``re 123 D. Using a region of memory reserved via ``reserve_mem`` command line
124 parameter. The address and size will be de 124 parameter. The address and size will be defined by the ``reserve_mem``
125 parameter. Note, that ``reserve_mem`` may 125 parameter. Note, that ``reserve_mem`` may not always allocate memory
126 in the same location, and cannot be relied 126 in the same location, and cannot be relied upon. Testing will need
127 to be done, and it may not work on every m 127 to be done, and it may not work on every machine, nor every kernel.
128 Consider this a "best effort" approach. Th 128 Consider this a "best effort" approach. The ``reserve_mem`` option
129 takes a size, alignment and name as argume 129 takes a size, alignment and name as arguments. The name is used
130 to map the memory to a label that can be r 130 to map the memory to a label that can be retrieved by ramoops.
131 131
132 reserve_mem=2M:4096:oops ramoops.mem_ 132 reserve_mem=2M:4096:oops ramoops.mem_name=oops
133 133
134 You can specify either RAM memory or periphera 134 You can specify either RAM memory or peripheral devices' memory. However, when
135 specifying RAM, be sure to reserve the memory 135 specifying RAM, be sure to reserve the memory by issuing memblock_reserve()
136 very early in the architecture code, e.g.:: 136 very early in the architecture code, e.g.::
137 137
138 #include <linux/memblock.h> 138 #include <linux/memblock.h>
139 139
140 memblock_reserve(ramoops_data.mem_addr 140 memblock_reserve(ramoops_data.mem_address, ramoops_data.mem_size);
141 141
142 Dump format 142 Dump format
143 ----------- 143 -----------
144 144
145 The data dump begins with a header, currently 145 The data dump begins with a header, currently defined as ``====`` followed by a
146 timestamp and a new line. The dump then contin 146 timestamp and a new line. The dump then continues with the actual data.
147 147
148 Reading the data 148 Reading the data
149 ---------------- 149 ----------------
150 150
151 The dump data can be read from the pstore file 151 The dump data can be read from the pstore filesystem. The format for these
152 files is ``dmesg-ramoops-N``, where N is the r 152 files is ``dmesg-ramoops-N``, where N is the record number in memory. To delete
153 a stored record from RAM, simply unlink the re 153 a stored record from RAM, simply unlink the respective pstore file.
154 154
155 Persistent function tracing 155 Persistent function tracing
156 --------------------------- 156 ---------------------------
157 157
158 Persistent function tracing might be useful fo 158 Persistent function tracing might be useful for debugging software or hardware
159 related hangs. The functions call chain log is 159 related hangs. The functions call chain log is stored in a ``ftrace-ramoops``
160 file. Here is an example of usage:: 160 file. Here is an example of usage::
161 161
162 # mount -t debugfs debugfs /sys/kernel/debug/ 162 # mount -t debugfs debugfs /sys/kernel/debug/
163 # echo 1 > /sys/kernel/debug/pstore/record_ft 163 # echo 1 > /sys/kernel/debug/pstore/record_ftrace
164 # reboot -f 164 # reboot -f
165 [...] 165 [...]
166 # mount -t pstore pstore /mnt/ 166 # mount -t pstore pstore /mnt/
167 # tail /mnt/ftrace-ramoops 167 # tail /mnt/ftrace-ramoops
168 0 ffffffff8101ea64 ffffffff8101bcda native_ 168 0 ffffffff8101ea64 ffffffff8101bcda native_apic_mem_read <- disconnect_bsp_APIC+0x6a/0xc0
169 0 ffffffff8101ea44 ffffffff8101bcf6 native_ 169 0 ffffffff8101ea44 ffffffff8101bcf6 native_apic_mem_write <- disconnect_bsp_APIC+0x86/0xc0
170 0 ffffffff81020084 ffffffff8101a4b5 hpet_di 170 0 ffffffff81020084 ffffffff8101a4b5 hpet_disable <- native_machine_shutdown+0x75/0x90
171 0 ffffffff81005f94 ffffffff8101a4bb iommu_s 171 0 ffffffff81005f94 ffffffff8101a4bb iommu_shutdown_noop <- native_machine_shutdown+0x7b/0x90
172 0 ffffffff8101a6a1 ffffffff8101a437 native_ 172 0 ffffffff8101a6a1 ffffffff8101a437 native_machine_emergency_restart <- native_machine_restart+0x37/0x40
173 0 ffffffff811f9876 ffffffff8101a73a acpi_re 173 0 ffffffff811f9876 ffffffff8101a73a acpi_reboot <- native_machine_emergency_restart+0xaa/0x1e0
174 0 ffffffff8101a514 ffffffff8101a772 mach_re 174 0 ffffffff8101a514 ffffffff8101a772 mach_reboot_fixups <- native_machine_emergency_restart+0xe2/0x1e0
175 0 ffffffff811d9c54 ffffffff8101a7a0 __const 175 0 ffffffff811d9c54 ffffffff8101a7a0 __const_udelay <- native_machine_emergency_restart+0x110/0x1e0
176 0 ffffffff811d9c34 ffffffff811d9c80 __delay 176 0 ffffffff811d9c34 ffffffff811d9c80 __delay <- __const_udelay+0x30/0x40
177 0 ffffffff811d9d14 ffffffff811d9c3f delay_t 177 0 ffffffff811d9d14 ffffffff811d9c3f delay_tsc <- __delay+0xf/0x20
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.