NVMe 驱动中断绑定解决CPU lock

来源：互联网发布：Wordpress优化编辑：程序博客网时间：2024/04/28 20:03

NVMe驱动中断绑定

1. 要想理清绑定先把irqbalance关闭

2. Kernel 和NVMe 驱动都可以和中断绑定相关

a. CentOS 6.6 之前的版本，NVMe 设备默认是绑定在最近的NUMA node的的第一个core,. 如果有多个强悍的PCIe SSD 在一个core上就有问题啦(CPU lock)，哈哈！

[root@memblaze-lyk2 wa-script]# cat /etc/issue
CentOS release 6.6 (Final)

[root@memblaze-lyk2 wa-script]# fio --rw=write --iodepth=64 --numjobs=12 --loop=1 --bs=4k --ioengine=libaio --output=precond-1.log --name=preCondJob --filename=/dev/nvme0n1 --direct=1

[root@memblaze-lyk2 wa-script]# cat /proc/interrupts | grep nvme
109: 524206 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q0, nvme0q1
110: 301 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q2
111: 297792 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q3
112: 111 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q4
113: 189682 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q5
114: 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q6
115: 104160 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q7
116: 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q8
117: 40872 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q9
118: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q10
119: 10006 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q11
120: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q12
121: 1475 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q13
122: 231 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q14
123: 113 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q15
124: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q16
[root@memblaze-lyk2 wa-script]# lspci | grep Non
06:00.0 Non-Volatile memory controller: Device 1c5f:0540 (rev 05)

b. CentOS 7.1 就变的智能啦，NVMe 设备默认是绑定在最近的NUMA node的所有core上

[root@memblaze-lyk1 ~]# cat /proc/interrupts | grep nvme
121: 154 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q0, nvme0q1
122: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q2
123: 40 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q3
124: 3 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q4
125: 4 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q5
126: 2 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q6
127: 5 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q7
128: 2 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q8
129: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q9
130: 3 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q10
131: 7 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q11
132: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q12
[root@memblaze-lyk1 ~]# uname -a
Linux memblaze-lyk1 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@memblaze-lyk1 ~]# fio --rw=write --iodepth=64 --numjobs=12 --loop=1 --bs=4k --ioengine=libaio --output=precond-1.log --name=preCondJob --filename=/dev/nvme0n1 --direct=1
^C[root@memblaze-lyk1 ~]cat /proc/interrupts | grep nvme
121: 154 0 0 0 0 0 0 0 175630 0 0 0 IR-PCI-MSI-edge nvme0q0, nvme0q1
122: 0 0 0 0 0 0 0 0 0 0 1557 0 IR-PCI-MSI-edge nvme0q2
123: 208478 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q3
124: 3 0 0 0 144 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q4
125: 4 0 0 0 0 0 146254 0 0 0 0 0 IR-PCI-MSI-edge nvme0q5
126: 2 0 0 0 0 0 0 0 69 0 0 0 IR-PCI-MSI-edge nvme0q6
127: 5 0 0 0 0 0 0 0 0 0 172702 0 IR-PCI-MSI-edge nvme0q7
128: 100 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q8
129: 0 0 177603 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q9
130: 3 0 0 0 78 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme0q10
131: 7 0 0 0 0 0 215733 0 0 0 0 0 IR-PCI-MSI-edge nvme0q11
132: 0 0 0 0 0 0 0 0 88 0 0 0 IR-PCI-MSI-edge nvme0q12
[root@memblaze-lyk1 ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)

[root@memblaze-lyk1 ~]# lspci | grep Non
06:00.0 Non-Volatile memory controller: Device 1c5f:0540 (rev 05)

0 0