1 =======================
2 NUMA Memory Performance
3 =======================
4
5 NUMA Locality
6 =============
7
8 Some platforms may have multiple types of memo
9 node. These disparate memory ranges may share
10 as CPU cache coherence, but may have different
11 different media types and buses affect bandwid
12
13 A system supports such heterogeneous memory by
14 under different domains, or "nodes", based on
15 characteristics. Some memory may share the sa
16 are provided as memory only nodes. While memor
17 CPUs, they may still be local to one or more c
18 other nodes. The following diagram shows one s
19 nodes with local memory and a memory only node
20
21 +------------------+ +------------------+
22 | Compute Node 0 +-----+ Compute Node 1 |
23 | Local Node0 Mem | | Local Node1 Mem |
24 +--------+---------+ +--------+---------+
25 | |
26 +--------+---------+ +--------+---------+
27 | Slower Node2 Mem | | Slower Node3 Mem |
28 +------------------+ +--------+---------+
29
30 A "memory initiator" is a node containing one
31 CPUs or separate memory I/O devices that can i
32 A "memory target" is a node containing one or
33 ranges accessible from one or more memory init
34
35 When multiple memory initiators exist, they ma
36 performance when accessing a given memory targ
37 pair may be organized into different ranked ac
38 this relationship. The highest performing init
39 is considered to be one of that target's local
40 the highest access class, 0. Any given target
41 local initiators, and any given initiator may
42 memory targets.
43
44 To aid applications matching memory targets wi
45 kernel provides symlinks to each other. The fo
46 relationship for the access class "0" memory i
47
48 # symlinks -v /sys/devices/system/node
49 relative: /sys/devices/system/node/nod
50
51 # symlinks -v /sys/devices/system/node
52 relative: /sys/devices/system/node/nod
53
54 A memory initiator may have multiple memory ta
55 class. The target memory's initiators in a giv
56 nodes' access characteristics share the same p
57 linked initiator nodes. Each target within an
58 though, do not necessarily perform the same as
59
60 The access class "1" is used to allow differen
61 that are CPUs and hence suitable for generic t
62 IO initiators such as GPUs and NICs. Unlike a
63 nodes containing CPUs are considered.
64
65 NUMA Performance
66 ================
67
68 Applications may wish to consider which node t
69 be allocated from based on the node's performa
70 the system provides these attributes, the kern
71 node sysfs hierarchy by appending the attribut
72 memory node's access class 0 initiators as fol
73
74 /sys/devices/system/node/nodeY/access0
75
76 These attributes apply only when accessed from
77 are linked under the this access's initiators.
78
79 The performance characteristics the kernel pro
80 are exported are as follows::
81
82 # tree -P "read*|write*" /sys/devices/
83 /sys/devices/system/node/nodeY/access0
84 |-- read_bandwidth
85 |-- read_latency
86 |-- write_bandwidth
87 `-- write_latency
88
89 The bandwidth attributes are provided in MiB/s
90
91 The latency attributes are provided in nanosec
92
93 The values reported here correspond to the rat
94 for the platform.
95
96 Access class 1 takes the same form but only in
97 memory activity.
98
99 NUMA Cache
100 ==========
101
102 System memory may be constructed in a hierarch
103 performance characteristics in order to provid
104 slower performing memory cached by a smaller h
105 system physical addresses memory initiators a
106 by the last memory level in the hierarchy. The
107 higher performing memory to transparently cach
108 slower levels.
109
110 The term "far memory" is used to denote the la
111 hierarchy. Each increasing cache level provide
112 initiator access, and the term "near memory" r
113 cache provided by the system.
114
115 This numbering is different than CPU caches wh
116 L1, L2, L3) uses the CPU-side view where each
117 performing. In contrast, the memory cache leve
118 level memory, so the higher numbered cache lev
119 nearer to the CPU, and further from far memory
120
121 The memory-side caches are not directly addres
122 software accesses a system address, the system
123 near memory cache if it is present. If it is n
124 accesses the next level of memory until there
125 cache level, or it reaches far memory.
126
127 An application does not need to know about cac
128 to use the system. Software may optionally que
129 attributes in order to maximize the performanc
130 If the system provides a way for the kernel to
131 for example with ACPI HMAT (Heterogeneous Memo
132 the kernel will append these attributes to the
133
134 When the kernel first registers a memory cache
135 will create the following directory::
136
137 /sys/devices/system/node/nodeX/memory_
138
139 If that directory is not present, the system e
140 a memory-side cache, or that information is no
141
142 The attributes for each level of cache is prov
143 level index::
144
145 /sys/devices/system/node/nodeX/memory_
146 /sys/devices/system/node/nodeX/memory_
147 /sys/devices/system/node/nodeX/memory_
148
149 Each cache level's directory provides its attr
150 following shows a single cache level and the a
151 software to query::
152
153 # tree /sys/devices/system/node/node0/
154 /sys/devices/system/node/node0/memory_
155 |-- index1
156 | |-- indexing
157 | |-- line_size
158 | |-- size
159 | `-- write_policy
160
161 The "indexing" will be 0 if it is a direct-map
162 for any other indexed based, multi-way associa
163
164 The "line_size" is the number of bytes accesse
165 level on a miss.
166
167 The "size" is the number of bytes provided by
168
169 The "write_policy" will be 0 for write-back, a
170 write-through caching.
171
172 See Also
173 ========
174
175 [1] https://www.uefi.org/sites/default/files/r
176 - Section 5.2.27
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.