What is NUMA? (翻译 by chatgpt)

发布时间 2023-12-04 21:43:17作者: 摩斯电码

原文:https://www.kernel.org/doc/html/latest/mm/numa.html

This question can be answered from a couple of perspectives: the hardware view and the Linux software view.
这个问题可以从硬件视角和Linux软件视角来回答。

From the hardware perspective, a NUMA system is a computer platform that comprises multiple components or assemblies each of which may contain 0 or more CPUs, local memory, and/or IO buses. For brevity and to disambiguate the hardware view of these physical components/assemblies from the software abstraction thereof, we'll call the components/assemblies 'cells' in this document.
从硬件视角来看,NUMA系统是由多个组件或装配组成的计算机平台,每个组件/装配可能包含0个或多个CPU、本地内存和/或IO总线。为了简洁并消除这些物理组件/装配的硬件视图与软件抽象之间的歧义,在本文中我们将这些组件/装配称为“单元”。

Each of the 'cells' may be viewed as an SMP [symmetric multi-processor] subset of the system--although some components necessary for a stand-alone SMP system may not be populated on any given cell. The cells of the NUMA system are connected together with some sort of system interconnect--e.g., a crossbar or point-to-point link are common types of NUMA system interconnects. Both of these types of interconnects can be aggregated to create NUMA platforms with cells at multiple distances from other cells.
每个“单元”可以被视为系统的SMP[对称多处理器]子集,尽管一些独立SMP系统所需的组件可能在任何给定的单元上都没有被填充。NUMA系统的单元通过某种系统互连连接在一起,例如,交叉开关或点对点链接是常见的NUMA系统互连类型。这些类型的互连都可以聚合起来创建具有多个距离的单元的NUMA平台。

For Linux, the NUMA platforms of interest are primarily what is known as Cache Coherent NUMA or ccNUMA systems. With ccNUMA systems, all memory is visible to and accessible from any CPU attached to any cell and cache coherency is handled in hardware by the processor caches and/or the system interconnect.
对于Linux来说,感兴趣的NUMA平台主要是所谓的Cache Coherent NUMA或ccNUMA系统。在ccNUMA系统中,所有内存对于连接到任何单元的任何CPU都是可见和可访问的,并且缓存一致性由处理器缓存和/或系统互连硬件处理。

Memory access time and effective memory bandwidth varies depending on how far away the cell containing the CPU or IO bus making the memory access is from the cell containing the target memory. For example, access to memory by CPUs attached to the same cell will experience faster access times and higher bandwidths than accesses to memory on other, remote cells. NUMA platforms can have cells at multiple remote distances from any given cell.
内存访问时间和有效内存带宽取决于包含进行内存访问的CPU或IO总线的单元与包含目标内存的单元之间的距离。例如,连接到同一单元的CPU对内存的访问将比连接到其他远程单元的内存的访问具有更快的访问时间和更高的带宽。NUMA平台可以具有多个远程距离的单元。

Platform vendors don't build NUMA systems just to make software developers' lives interesting. Rather, this architecture is a means to provide scalable memory bandwidth. However, to achieve scalable memory bandwidth, system and application software must arrange for a large majority of the memory references [cache misses] to be to "local" memory--memory on the same cell, if any--or to the closest cell with memory.
平台供应商建立NUMA系统并不是为了让软件开发人员的生活变得有趣。相反,这种架构是为了提供可扩展的内存带宽。然而,为了实现可扩展的内存带宽,系统和应用软件必须安排大多数内存引用[缓存未命中]指向“本地”内存--即同一单元上的内存,如果有的话--或者指向最接近的具有内存的单元。

This leads to the Linux software view of a NUMA system:
这导致了对NUMA系统的Linux软件视图:

Linux divides the system's hardware resources into multiple software abstractions called "nodes". Linux maps the nodes onto the physical cells of the hardware platform, abstracting away some of the details for some architectures. As with physical cells, software nodes may contain 0 or more CPUs, memory and/or IO buses. And, again, memory accesses to memory on "closer" nodes--nodes that map to closer cells--will generally experience faster access times and higher effective bandwidth than accesses to more remote cells.
Linux将系统的硬件资源划分为多个称为“节点”的软件抽象。Linux将这些节点映射到硬件平台的物理单元上,并在某些体系结构上抽象掉一些细节。与物理单元一样,软件节点可能包含0个或多个CPU、内存和/或IO总线。而且,对“更近”节点上的内存的访问--映射到更近单元的节点--通常会比对更远单元的访问具有更快的访问时间和更高的有效带宽。

For some architectures, such as x86, Linux will "hide" any node representing a physical cell that has no memory attached, and reassign any CPUs attached to that cell to a node representing a cell that does have memory. Thus, on these architectures, one cannot assume that all CPUs that Linux associates with a given node will see the same local memory access times and bandwidth.
对于一些体系结构,例如x86,Linux将“隐藏”任何代表没有连接内存的物理单元的节点,并重新分配连接到该单元的任何CPU到代表具有内存的单元的节点。因此,在这些体系结构上,不能假设Linux关联到给定节点的所有CPU将看到相同的本地内存访问时间和带宽。

In addition, for some architectures, again x86 is an example, Linux supports the emulation of additional nodes. For NUMA emulation, linux will carve up the existing nodes--or the system memory for non-NUMA platforms--into multiple nodes. Each emulated node will manage a fraction of the underlying cells' physical memory. NUMA emulation is useful for testing NUMA kernel and application features on non-NUMA platforms, and as a sort of memory resource management mechanism when used together with cpusets. [see CPUSETS]
此外,对于一些体系结构,再次以x86为例,Linux支持额外节点的仿真。对于NUMA仿真,Linux将现有节点--或非NUMA平台的系统内存--划分为多个节点。每个仿真节点将管理底层单元的物理内存的一部分。NUMA仿真对于在非NUMA平台上测试NUMA内核和应用程序功能以及与cpusets一起使用时作为一种内存资源管理机制是有用的。参见CPUSETS

For each node with memory, Linux constructs an independent memory management subsystem, complete with its own free page lists, in-use page lists, usage statistics and locks to mediate access. In addition, Linux constructs for each memory zone [one or more of DMA, DMA32, NORMAL, HIGH_MEMORY, MOVABLE], an ordered "zonelist". A zonelist specifies the zones/nodes to visit when a selected zone/node cannot satisfy the allocation request. This situation, when a zone has no available memory to satisfy a request, is called "overflow" or "fallback".
对于每个具有内存的节点,Linux构建了一个独立的内存管理子系统,包括自己的空闲页列表、使用中的页列表、使用统计和用于调解访问的锁。此外,Linux为每个内存区域[一个或多个DMA、DMA32、NORMAL、HIGH_MEMORY、MOVABLE]构建了一个有序的“区域列表”。区域列表指定了在选定的区域/节点无法满足分配请求时要访问的区域/节点。当一个区域没有可用内存来满足请求时,这种情况被称为“溢出”或“回退”。

Because some nodes contain multiple zones containing different types of memory, Linux must decide whether to order the zonelists such that allocations fall back to the same zone type on a different node, or to a different zone type on the same node. This is an important consideration because some zones, such as DMA or DMA32, represent relatively scarce resources. Linux chooses a default Node ordered zonelist. This means it tries to fallback to other zones from the same node before using remote nodes which are ordered by NUMA distance.
因为一些节点包含不同类型的内存的多个区域,Linux必须决定是按照顺序排列区域列表,以便分配回退到同一节点上的相同区域类型,还是分配回退到同一节点上的不同区域类型。这是一个重要的考虑因素,因为一些区域,比如DMA或DMA32,代表相对稀缺的资源。Linux选择默认的节点有序区域列表。这意味着它会尝试在使用NUMA距离排序的远程节点之前,从同一节点的其他区域回退。

By default, Linux will attempt to satisfy memory allocation requests from the node to which the CPU that executes the request is assigned. Specifically, Linux will attempt to allocate from the first node in the appropriate zonelist for the node where the request originates. This is called "local allocation." If the "local" node cannot satisfy the request, the kernel will examine other nodes' zones in the selected zonelist looking for the first zone in the list that can satisfy the request.
默认情况下,Linux将尝试满足内存分配请求,从执行请求的CPU所分配的节点开始。具体来说,Linux将尝试从节点的适当区域列表中的第一个节点中分配。这被称为“本地分配”。如果“本地”节点无法满足请求,内核将检查选定区域列表中其他节点的区域,寻找能够满足请求的第一个区域。

Local allocation will tend to keep subsequent access to the allocated memory "local" to the underlying physical resources and off the system interconnect-- as long as the task on whose behalf the kernel allocated some memory does not later migrate away from that memory. The Linux scheduler is aware of the NUMA topology of the platform--embodied in the "scheduling domains" data structures [see Scheduler Domains]--and the scheduler attempts to minimize task migration to distant scheduling domains. However, the scheduler does not take a task's NUMA footprint into account directly. Thus, under sufficient imbalance, tasks can migrate between nodes, remote from their initial node and kernel data structures.
本地分配将倾向于使分配的内存的后续访问“本地”到底层物理资源,并且不会通过系统互连--只要内核为某些任务分配的内存不会后来迁移到远离该内存的地方。Linux调度程序知道平台的NUMA拓扑结构--体现在“调度域”数据结构中--并且调度程序试图最小化任务迁移到远离的调度域。然而,调度程序并不直接考虑任务的NUMA足迹。因此,在足够的不平衡下,任务可以在节点之间迁移,远离它们的初始节点和内核数据结构。

System administrators and application designers can restrict a task's migration to improve NUMA locality using various CPU affinity command line interfaces, such as taskset(1) and numactl(1), and program interfaces such as sched_setaffinity(2). Further, one can modify the kernel's default local allocation behavior using Linux NUMA memory policy. [see NUMA Memory Policy].
系统管理员和应用程序设计人员可以使用各种CPU亲和力命令行界面,如taskset(1)和numactl(1),以及程序接口,如sched_setaffinity(2),来限制任务的迁移,以改善NUMA局部性。此外,可以使用Linux NUMA内存策略来修改内核的默认本地分配行为。参见NUMA内存策略

System administrators can restrict the CPUs and nodes' memories that a non- privileged user can specify in the scheduling or NUMA commands and functions using control groups and CPUsets. [see CPUSETS]
系统管理员可以使用控制组和CPUsets来限制非特权用户在调度或NUMA命令和函数中指定的CPU和节点内存。参见CPUSETS

On architectures that do not hide memoryless nodes, Linux will include only zones [nodes] with memory in the zonelists. This means that for a memoryless node the "local memory node"--the node of the first zone in CPU's node's zonelist--will not be the node itself. Rather, it will be the node that the kernel selected as the nearest node with memory when it built the zonelists. So, default, local allocations will succeed with the kernel supplying the closest available memory. This is a consequence of the same mechanism that allows such allocations to fallback to other nearby nodes when a node that does contain memory overflows.
在不隐藏无内存节点的体系结构上,Linux将只包括具有内存的区域[节点]在区域列表中。这意味着对于无内存节点,CPU的节点区域列表中的“本地内存节点”--第一个区域的节点--将不是节点本身。相反,它将是内核在构建区域列表时选择的最近的具有内存的节点。因此,默认情况下,本地分配将成功,内核将提供最接近的可用内存。这是允许这种分配在节点溢出时回退到其他附近节点的相同机制的结果。

Some kernel allocations do not want or cannot tolerate this allocation fallback behavior. Rather they want to be sure they get memory from the specified node or get notified that the node has no free memory. This is usually the case when a subsystem allocates per CPU memory resources, for example.
一些内核分配不希望或不能容忍这种分配回退行为。相反,它们希望确保它们从指定的节点获取内存,或者得到通知该节点没有空闲内存。当一个子系统分配每CPU内存资源时,通常会出现这种情况。

A typical model for making such an allocation is to obtain the node id of the node to which the "current CPU" is attached using one of the kernel's numa_node_id() or CPU_to_node() functions and then request memory from only the node id returned. When such an allocation fails, the requesting subsystem may revert to its own fallback path. The slab kernel memory allocator is an example of this. Or, the subsystem may choose to disable or not to enable itself on allocation failure. The kernel profiling subsystem is an example of this.
进行这种分配的典型模型是使用内核的numa_node_id()或CPU_to_node()函数获取“当前CPU”连接的节点的节点ID,然后仅从返回的节点ID请求内存。当这样的分配失败时,请求的子系统可能会恢复到自己的回退路径。slab内核内存分配器就是一个例子。或者,该子系统可以选择在分配失败时禁用或不启用自身。内核profiling子系统就是一个例子。

If the architecture supports--does not hide--memoryless nodes, then CPUs attached to memoryless nodes would always incur the fallback path overhead or some subsystems would fail to initialize if they attempted to allocated memory exclusively from a node without memory. To support such architectures transparently, kernel subsystems can use the numa_mem_id() or cpu_to_mem() function to locate the "local memory node" for the calling or specified CPU. Again, this is the same node from which default, local page allocations will be attempted.
如果体系结构支持--不隐藏--无内存节点,那么连接到无内存节点的CPU将始终产生回退路径开销,或者一些子系统将无法初始化,如果它们试图从没有内存的节点中分配内存。为了透明地支持这样的体系结构,内核子系统可以使用numa_mem_id()或cpu_to_mem()函数来定位调用或指定CPU的“本地内存节点”。同样,这是默认情况下,本地页面分配将尝试的节点。