使用MLC(Intel Memory Latency Checker )测试Numa对内存延迟的影响

Numa配置对数据库环境有swap的影响已经众所周知，但Numa对内存延迟、磁盘吞吐量都有影响，最近看到MLC（Intel Memory Latency Checker ）这个工具，测试了下跨numa的内存延迟有多大

# 准备工作
# cat /sys/kernel/mm/transparent_hugepage/defrag
always defer defer+madvise [madvise] never
 
# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

# cat /proc/sys/vm/nr_hugepages
0

echo 4000 > /proc/sys/vm/nr_hugepages


# 内存延迟测试
# ./mlc --latency_matrix
Intel(R) Memory Latency Checker - v3.11b
Command line parameters: --latency_matrix

Using buffer size of 1800.000MiB
Measuring idle latencies for random access (in ns)...
        Numa node
Numa node        0       1
       0     104.7   287.5
       1     292.6   104.6

# 内存带宽测试       
[root@k8s-10-128-64-5 Linux]# ./mlc --bandwidth_matrix
Intel(R) Memory Latency Checker - v3.11b
Command line parameters: --bandwidth_matrix

Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes
Measuring Memory Bandwidths between nodes within system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
        Numa node
Numa node        0       1
       0    143101.8    78546.8
       1    78872.6 143302.3

可以看到跨Numa是有2~3倍的延迟以及带宽影响的，所以假如追求极致性能，建议--cpunodebind=0 --membind=0 或--cpunodebind=1 --membind=1来绑定到同个Numa节点。