填坑日常之 EDAC DIMM CE Error

2015-12-30 下午 ngaios 监控发现硬盘满报警
2015-12-31 早上开始排查原因

经过排查发现 log 目录下的三个系统日志非常大，竟有 8.7G 大小

读日志内容后发现有大量的 EDAC DIMM CE Error 出现

经过一番 Google 后得知这是由于内存错误，导致系统自动启动错误恢复机制，但恢复失败写入日志，继续修复，循环下去导致日志文件大小暴增

粗略看了看 linux 的内核文档之 edac doc

根据这一段

Dual channels allows for 128 bit data transfers to the CPU from memory.Some newer chipsets allow for more than 2 channels, like Fully Buffered DIMMs(FB-DIMMs). The following example will assume 2 channels:            Channel 0   Channel 1    ===================================    csrow0  | DIMM_A0   | DIMM_B0 |    csrow1  | DIMM_A0   | DIMM_B0 |    ===================================    ===================================    csrow2  | DIMM_A1   | DIMM_B1 |    csrow3  | DIMM_A1   | DIMM_B1 |    ===================================

于是在机器上执行

root@ubuntu:/var/log# grep "[0-9]" /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count/sys/devices/system/edac/mc/mc0/csrow0/ch0_ce_count:0/sys/devices/system/edac/mc/mc0/csrow1/ch0_ce_count:0/sys/devices/system/edac/mc/mc0/csrow2/ch0_ce_count:4213901959

参考前面的文档，可得出问题的是 DIMM_A1

执行 root@ubuntu:/var/log# dmidecode -t memory，在结果中可以找到 DIMM_A1 的信息

Memory Device    Array Handle: 0x0032    Error Information Handle: Not Provided    Total Width: 72 bits    Data Width: 64 bits    Size: 4096 MB    Form Factor: DIMM    Set: None    Locator: DIMM_A1    Bank Locator: BANK0    Type: DDR3    Type Detail: Other    Speed: 1333 MHz    Manufacturer: Manufacturer0    Serial Number: SerNum1    Asset Tag: AssetTagNum1    Part Number: PartNum1

---

后续：
- 为了避免以后再发生这种日志撑满硬盘的情况，修改 logrotate 的配置文件，缩短日志备份周期，减少日志备份保留数量，启用备份压缩

---

参考：
- How can I find which memory have CE error?
- edac doc