问题现象

用户反馈多台机器PMC raid卡下硬盘写入性能延迟高的问题，机器OS为ESXi系统。

问题分析

日志分析

2023-11-13T17:34:53.663Z cpu13:2098447)WARNING: ScsiDeviceIO: 1513: Device naa.600508b1001c55603c71cb71e23b8301 performance has deteriorated. I/O latency increased from average value of 25206 microseconds to 757505 microseconds.
2023-11-13T17:34:54.980Z cpu0:2098449)ScsiDeviceIO: 1513: Device naa.600508b1001c55603c71cb71e23b8301 performance has improved. I/O latency reduced from 757505 microseconds to 147922 microseconds.
2023-11-13T17:35:02.615Z cpu62:2098460)ScsiDeviceIO: 1513: Device naa.600508b1001c55603c71cb71e23b8301 performance has improved. I/O latency reduced from 147922 microseconds to 49903 microseconds.
2023-11-13T17:52:47.544Z cpu30:2098457)WARNING: ScsiDeviceIO: 1513: Device naa.600508b1001c55603c71cb71e23b8301 performance has deteriorated. I/O latency increased from average value of 25277 microseconds to 763337 microseconds.
2023-11-13T17:52:48.769Z cpu32:2098446)ScsiDeviceIO: 1513: Device naa.600508b1001c55603c71cb71e23b8301 performance has improved. I/O latency reduced from 763337 microseconds to 149346 microseconds.
2023-11-13T17:53:06.086Z cpu17:2098445)ScsiDeviceIO: 1513: Device naa.600508b1001c55603c71cb71e23b8301 performance has improved. I/O latency reduced from 149346 microseconds to 50366 microseconds.
2023-11-14T16:01:46.047Z cpu31:2098445)WARNING: ScsiDeviceIO: 1513: Device naa.600508b1001c55603c71cb71e23b8301 performance has deteriorated. I/O latency increased from average value of 25397 microseconds to 780078 microseconds.
2023-11-14T16:01:46.618Z cpu0:2098451)ScsiDeviceIO: 1513: Device naa.600508b1001c55603c71cb71e23b8301 performance has improved. I/O latency reduced from 780078 microseconds to 154412 microseconds.

从日志中确实可以看到IO延迟较大。

继续查看vcenter下硬盘的延迟情况

查看结果为，硬盘的延迟均表现为写延迟较高（且非常高），读延迟正常，读延迟无任何问题。

image.png

vcenter下的告警也均为写延迟

image.png

image.png

分析结论：
从各个层面来看，均为写延迟，且延迟数值较高，因此怀疑为cache层面问题。

继续排查cache

logical drive的cache策略如下

Logical Device number 1
   Logical Device name                        : VD_1
   Disk Name                                  : Not Applicable
   Block Size of member drives                : 512 Bytes
   Array                                      : 1
   RAID level                                 : 6
   Status of Logical Device                   : Optimal
   Parity Initialization Status               : Completed
   Size                                       : 122076928 MB
   Stripe-unit size                           : 256 KB
   Full Stripe Size                           : 2048 KB
   Interface Type                             : Serial Attached SCSI
   Device Type                                : Data
   Boot Type                                  : None
   Heads                                      : 255
   Sectors Per Track                          : 32
   Cylinders                                  : 65535
   Caching                                    : Enabled
   Mount Points                               : Not Applicable
   LD Acceleration Method                     : Controller Cache

logical driver已经设置了cache策略，解释说明如下：
LD Acceleration Method ： Setting of the LD acceleration method. Controller cache or SSD I/O BypassormaxCache.

controller cache策略如下

   Cache Properties
   --------------------------------------------------------
   Cache Status                               : Ok
   Cache Serial Number                        : Not Applicable
   Cache memory                               : 3856 MB
   Read Cache Percentage                      : 100 percent
   Write Cache Percentage                     : 0 percent
   No-Battery Write Cache                     : Disabled
   Wait for Cache Room                        : Disabled
   Write Cache Bypass Threshold Size          : 1040 KB
   --------------------------------------------------------

从上面结果可以看出controller的cache策略将cache100%分配给了读，而没有给分配为写，这是不正常。
关于此参数的解释说明：
Cache Ratio : The controller cache ratio setting determines the controller ability to adjust theamount
of memory for read-ahead cache versus write cache.
Cache Ratio (Read) : Sets the ratio of controller cache memory used for read-ahead cache
versus write cache. Cache ratio values range from 0-100, in increments of 5

解决方法

调整read cache占比10%，write cache占比90%.

image.png

调整命令参考如下：

image.png