监控系列讲座（十一）常见系统监控指标之内存

3. 内存监控指标

内存是系统重要的指标之一，而且是必须提前规划好的资源。因为他和CPU不同，如果CPU资源不足，会导致系统变慢。如果内存资源不足，会触发oom，直接杀掉级别较低的程序。当然，我们还有一些swap可以作为临时的内存周转，主要还是为了防止oom，但是硬盘的速度和内存比简直是一天一地，肯定会拖慢系统速度。最后，现在很多容器技术是不可以使用缓存的，一旦内存满了，直接oom了，比如我们的k8s就是这样的例子。

3.1. 系统上查看内存指标

同样是两种方式

命令和工具：top，vmstat，free
文件：/proc/meminfo

工具同样是从文件中取得的数据，所以文件中的指标是最全的。

MemTotal:        3898628 kB
MemFree:         3508100 kB
MemAvailable:    3700256 kB
Buffers:           21516 kB
Cached:           210792 kB
SwapCached:            0 kB
Active:           113460 kB
Inactive:         145828 kB
Active(anon):      19828 kB
Inactive(anon):     8716 kB
Active(file):      93632 kB
Inactive(file):   137112 kB
Unevictable:          16 kB
Mlocked:              16 kB
SwapTotal:        102396 kB
SwapFree:         102396 kB
Dirty:                24 kB
Writeback:             0 kB
AnonPages:         27000 kB
Mapped:            35060 kB
Shmem:              8956 kB
Slab:              71708 kB
SReclaimable:      36152 kB
SUnreclaim:        35556 kB
KernelStack:        2304 kB
PageTables:         1376 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2051708 kB
Committed_AS:     138840 kB
VmallocTotal:   263061440 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:              960 kB
CmaTotal:         262144 kB
CmaFree:          223260 kB

尽管我们通常不会监控这么细，但是还是希望大家可以深入理解一下，因为有些特殊场景可能会用到，比如内核调优或者特殊软件需求（oracle的bigpage之类）。

而监控中，我们通常是从vmstat和free这两个方面来监控

root@node2:~# free -m
              total        used        free      shared  buff/cache   available
Mem:           3807         119        3425           8         262        3613
Swap:            99           0          99
root@node2:~# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 3507864  21520 247080    0    0     0     0   26   41  0  0 100  0  0

free中的几个选项

Mem:
- total: 总共的内存
- used: 使用的内存
- free: 空闲的内存
- shared：共享的内存（shmem）
- buff/cache：缓存内存区（buffer 写缓存/cache读缓存）
- avilable：可用的内存
Swap:
- total：一共的交换分区
- used：使用的交换分区
- free：空闲的交换分区

应该只有buff/cache不好理解。

A buffer is something that has yet to be "written" to disk.　　---buffer 写缓存，数据存储时，先保存到磁盘缓冲区，然后再写入到永久空间
A cache is something that has been "reed" from the disk adn stored for later use. --cache 读缓存，数据从磁盘读出后，暂留在缓冲区，预备程序接下来的使用，

2.2. zabbix上的内存监控指标

和我们使用free看到的指标基本一样

image-20200722205507381.png

file

2.3. grafana上的内存监控指标

比zabbix上的多一些，对于内存的指标监控的更加详细

image-20200721211732297.png

file

Apps：把用户空间占用的内存拿出来单独作为指标
PagetTables：物理和虚拟内存的映射表占用了一部分内存
SwapCache：从swap分区中拿出来但是没有使用的内存
Slab：内核用来缓存数据的

2.4. node_exporter上的内存监控指标

这个上面的指标是最全的，几乎涵盖了我们上面说的所有指标

# HELP node_memory_Active_anon_bytes Memory information field Active_anon_bytes.
# TYPE node_memory_Active_anon_bytes gauge
node_memory_Active_anon_bytes 2.23305728e+08
# HELP node_memory_Active_bytes Memory information field Active_bytes.
# TYPE node_memory_Active_bytes gauge
node_memory_Active_bytes 3.76311808e+08
# HELP node_memory_Active_file_bytes Memory information field Active_file_bytes.
# TYPE node_memory_Active_file_bytes gauge
node_memory_Active_file_bytes 1.5300608e+08
# HELP node_memory_AnonPages_bytes Memory information field AnonPages_bytes.
# TYPE node_memory_AnonPages_bytes gauge
node_memory_AnonPages_bytes 1.43417344e+08
# HELP node_memory_Bounce_bytes Memory information field Bounce_bytes.
# TYPE node_memory_Bounce_bytes gauge
node_memory_Bounce_bytes 0
# HELP node_memory_Buffers_bytes Memory information field Buffers_bytes.
# TYPE node_memory_Buffers_bytes gauge
node_memory_Buffers_bytes 5.2629504e+07
# HELP node_memory_Cached_bytes Memory information field Cached_bytes.
# TYPE node_memory_Cached_bytes gauge
node_memory_Cached_bytes 3.90656e+08
# HELP node_memory_CmaFree_bytes Memory information field CmaFree_bytes.
# TYPE node_memory_CmaFree_bytes gauge
node_memory_CmaFree_bytes 2.2861824e+08
# HELP node_memory_CmaTotal_bytes Memory information field CmaTotal_bytes.
# TYPE node_memory_CmaTotal_bytes gauge
node_memory_CmaTotal_bytes 2.68435456e+08
# HELP node_memory_CommitLimit_bytes Memory information field CommitLimit_bytes.
# TYPE node_memory_CommitLimit_bytes gauge
node_memory_CommitLimit_bytes 1.996095488e+09
# HELP node_memory_Committed_AS_bytes Memory information field Committed_AS_bytes.
# TYPE node_memory_Committed_AS_bytes gauge
node_memory_Committed_AS_bytes 9.13432576e+08
# HELP node_memory_Dirty_bytes Memory information field Dirty_bytes.
# TYPE node_memory_Dirty_bytes gauge
node_memory_Dirty_bytes 36864
# HELP node_memory_Inactive_anon_bytes Memory information field Inactive_anon_bytes.
# TYPE node_memory_Inactive_anon_bytes gauge
node_memory_Inactive_anon_bytes 2.871296e+07
# HELP node_memory_Inactive_bytes Memory information field Inactive_bytes.
# TYPE node_memory_Inactive_bytes gauge
node_memory_Inactive_bytes 2.10337792e+08
# HELP node_memory_Inactive_file_bytes Memory information field Inactive_file_bytes.
# TYPE node_memory_Inactive_file_bytes gauge
node_memory_Inactive_file_bytes 1.81624832e+08
# HELP node_memory_KernelStack_bytes Memory information field KernelStack_bytes.
# TYPE node_memory_KernelStack_bytes gauge
node_memory_KernelStack_bytes 3.899392e+06
# HELP node_memory_Mapped_bytes Memory information field Mapped_bytes.
# TYPE node_memory_Mapped_bytes gauge
node_memory_Mapped_bytes 1.35671808e+08
# HELP node_memory_MemAvailable_bytes Memory information field MemAvailable_bytes.
# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 3.52661504e+09
# HELP node_memory_MemFree_bytes Memory information field MemFree_bytes.
# TYPE node_memory_MemFree_bytes gauge
node_memory_MemFree_bytes 3.224031232e+09
# HELP node_memory_MemTotal_bytes Memory information field MemTotal_bytes.
# TYPE node_memory_MemTotal_bytes gauge
node_memory_MemTotal_bytes 3.992195072e+09
# HELP node_memory_Mlocked_bytes Memory information field Mlocked_bytes.
# TYPE node_memory_Mlocked_bytes gauge
node_memory_Mlocked_bytes 16384
# HELP node_memory_NFS_Unstable_bytes Memory information field NFS_Unstable_bytes.
# TYPE node_memory_NFS_Unstable_bytes gauge
node_memory_NFS_Unstable_bytes 0
# HELP node_memory_PageTables_bytes Memory information field PageTables_bytes.
# TYPE node_memory_PageTables_bytes gauge
node_memory_PageTables_bytes 1.308672e+07
# HELP node_memory_Percpu_bytes Memory information field Percpu_bytes.
# TYPE node_memory_Percpu_bytes gauge
node_memory_Percpu_bytes 1.327104e+06
# HELP node_memory_SReclaimable_bytes Memory information field SReclaimable_bytes.
# TYPE node_memory_SReclaimable_bytes gauge
node_memory_SReclaimable_bytes 4.685824e+07
# HELP node_memory_SUnreclaim_bytes Memory information field SUnreclaim_bytes.
# TYPE node_memory_SUnreclaim_bytes gauge
node_memory_SUnreclaim_bytes 5.6180736e+07
# HELP node_memory_Shmem_bytes Memory information field Shmem_bytes.
# TYPE node_memory_Shmem_bytes gauge
node_memory_Shmem_bytes 1.15044352e+08
# HELP node_memory_Slab_bytes Memory information field Slab_bytes.
# TYPE node_memory_Slab_bytes gauge
node_memory_Slab_bytes 1.03038976e+08
# HELP node_memory_SwapCached_bytes Memory information field SwapCached_bytes.
# TYPE node_memory_SwapCached_bytes gauge
node_memory_SwapCached_bytes 0
# HELP node_memory_SwapFree_bytes Memory information field SwapFree_bytes.
# TYPE node_memory_SwapFree_bytes gauge
node_memory_SwapFree_bytes 0
# HELP node_memory_SwapTotal_bytes Memory information field SwapTotal_bytes.
# TYPE node_memory_SwapTotal_bytes gauge
node_memory_SwapTotal_bytes 0
# HELP node_memory_Unevictable_bytes Memory information field Unevictable_bytes.
# TYPE node_memory_Unevictable_bytes gauge
node_memory_Unevictable_bytes 16384
# HELP node_memory_VmallocChunk_bytes Memory information field VmallocChunk_bytes.
# TYPE node_memory_VmallocChunk_bytes gauge
node_memory_VmallocChunk_bytes 0
# HELP node_memory_VmallocTotal_bytes Memory information field VmallocTotal_bytes.
# TYPE node_memory_VmallocTotal_bytes gauge
node_memory_VmallocTotal_bytes 2.6937491456e+11
# HELP node_memory_VmallocUsed_bytes Memory information field VmallocUsed_bytes.
# TYPE node_memory_VmallocUsed_bytes gauge
node_memory_VmallocUsed_bytes 0
# HELP node_memory_WritebackTmp_bytes Memory information field WritebackTmp_bytes.
# TYPE node_memory_WritebackTmp_bytes gauge
node_memory_WritebackTmp_bytes 0
# HELP node_memory_Writeback_bytes Memory information field Writeback_bytes.
# TYPE node_memory_Writeback_bytes gauge
node_memory_Writeback_bytes 0
# HELP node_sockstat_FRAG6_memory Number of FRAG6 sockets in state memory.
# TYPE node_sockstat_FRAG6_memory gauge
node_sockstat_FRAG6_memory 0
# HELP node_sockstat_FRAG_memory Number of FRAG sockets in state memory.
# TYPE node_sockstat_FRAG_memory gauge
node_sockstat_FRAG_memory 0
# HELP node_udp_queues Number of allocated memory in the kernel for UDP datagrams in bytes.
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.0103168e+07
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 8.26662912e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1

为了方便大家学习，请大家加我的微信，我会把大家加到微信群（微信群的二维码会经常变）和qq群821119334，问题答案云原生技术课堂，有问题可以一起讨论

个人微信
640.jpeg
腾讯课堂
640-20200506145837072.jpeg
微信公众号
640-20200506145842007.jpeg
专题讲座

2020 CKA考试视频真题讲解 https://www.bilibili.com/video/BV167411K7hp

2020 CKA考试指南 https://www.bilibili.com/video/BV1sa4y1479B/

2020年 5月CKA考试真题 https://mp.weixin.qq.com/s/W9V4cpYeBhodol6AYtbxIA