FIO 的随机写和顺序写

先整几个实验数据，这里的数据都是用fio在dedup-fs生成的。数据大小是bs为4k，总共256个。

1. requence write

fio --directory=./ --direct=1 --rw=write --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=100 --iodepth=1 --numjobs=1  --group_reporting --name=4ktestwrite --size=1M

Inode: 13   Type: regular    Mode:  0644   Flags: 0x0
Generation: 2972670169    Version: 0x00000000
User:     0   Group:     0   Size: 1048576
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 2056
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x54640d9d -- Wed Nov 12 17:47:09 2014
atime: 0x54640d9d -- Wed Nov 12 17:47:09 2014
mtime: 0x54640d9d -- Wed Nov 12 17:47:09 2014
Size of extra inode fields: 4
BLOCKS:
(0-11):4096-4107, (IND):30873, (12-255):4108-4351
TOTAL: 257

2. random write

fio --directory=./ --direct=1 --rw=randwrite --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=100 --iodepth=1 --numjobs=1  --group_reporting --name=4ktestwrite --size=1M

Inode: 12   Type: regular    Mode:  0644   Flags: 0x0
Generation: 2972670168    Version: 0x00000000
User:     0   Group:     0   Size: 1048576
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 2056
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x54640d90 -- Wed Nov 12 17:46:56 2014
atime: 0x54640d90 -- Wed Nov 12 17:46:56 2014
mtime: 0x54640d90 -- Wed Nov 12 17:46:56 2014
Size of extra inode fields: 4
BLOCKS:
(0):16384, (1):30720, (2):30860, (3):30881, (4):30751, (5):30772, (6):30868, (7):30720, (8):30870, (9):30754, (10):30821, (11):30863, (IND):30721, (12):30793, (13):30720, (14):30720, (15):30720, (16):30720, (17):30805, (18):30730, (19):30720, (20):30826, (21):30849, (22):30831, (23):30720, (24):30743, (25):30733, (26):30720, (27):30808, (28):30791, (29):30720, (30):30776, (31):30720, (32):30720, (33):30738, (34):30720, (35):30798, (36):30766, (37):30844, (38):30786, (39):30851, (40):30828, (41):30850, (42):30820, (43):30720, (44):30794, (45):30720, (46):30795, (47):30755, (48):30720, (49):30753, (50):30783, (51):30720, (52):30862, (53):30720, (54):30823, (55):30819, (56):30848, (57):30748, (58):30720, (59):30775, (60):30797, (61):30720, (62):30720, (63):30720, (64):30865, (65):30758, (66):30787, (67):30759, (68):30778, (69):30720, (70):30720, (71):30780, (72):30752, (73):30720, (74):30720, (75):30745, (76):30817, (77):30720, (78):30765, (79):30720, (80):30864, (81):30720, (82):30822, (83):30768, (84):30852, (85):30720, (86):30761, (87):30857, (88):30737, (89):30816, (90):30720, (91):30720, (92):30872, (93):30840, (94):30720, (95):30785, (96):30720, (97):30833, (98):30720, (99):30720, (100):30720, (101):30720, (102):30855, (103):30720, (104):30720, (105):30720, (106):30720, (107):30773, (108):30790, (109):30720, (110):30853, (111):30720, (112):30740, (113):30720, (114):30720, (115):30742, (116):30811, (117):30720, (118):30720, (119):30812, (120):30720, (121):30729, (122):30720, (123):30720, (124):30856, (125):30877, (126):30815, (127):30832, (128):30746, (129):30777, (130):30723, (131):30720, (132):30720, (133):30725, (134):30760, (135):30720, (136):30722, (137):30720, (138):30720, (139):30720, (140):30847, (141):30770, (142):30720, (143):30720, (144):30720, (145):30720, (146):30720, (147):30720, (148):30813, (149):30843, (150):30782, (151):30744, (152):30841, (153):30756, (154):30774, (155):30720, (156):30869, (157):30720, (158):30720, (159):30720, (160):30842, (161):30720, (162):30837, (163):30720, (164):30854, (165):30839, (166):30769, (167):30727, (168):30720, (169):30720, (170):30846, (171):30724, (172):30720, (173):30720, (174):30720, (175):30762, (176):30731, (177):30720, (178):30804, (179):30720, (180):30726, (181):30866, (182):30720, (183):30827, (184):30858, (185):30878, (186):30720, (187):30720, (188):30830, (189):30735, (190):30720, (191):30750, (192):30734, (193):30720, (194):30824, (195):30803, (196):30836, (197):30834, (198):30859, (199):30882, (200):30736, (201):30767, (202):30867, (203):30818, (204):30845, (205):30801, (206):30871, (207):30796, (208):30728, (209):30800, (210):30720, (211):30829, (212):30720, (213):30825, (214):30720, (215):30807, (216):30720, (217):30810, (218):30835, (219):30720, (220):30720, (221):30763, (222):30784, (223):30838, (224):30720, (225):30720, (226):30720, (227):30747, (228):30720, (229):30779, (230):30720, (231):30788, (232):30739, (233):30861, (234):30720, (235):30802, (236):30741, (237):30764, (238):30757, (239):30809, (240):30792, (241):30732, (242):30799, (243):30720, (244):30781, (245):30720, (246):30771, (247):30814, (248):30720, (249):30806, (250):30720, (251):30749, (252):30720, (253):30720, (254):30720, (255):30789
TOTAL: 257

3. requence dedep 50%

fio --directory=./ --direct=1 --rw=write --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=100 --iodepth=1 --numjobs=1  --group_reporting --name=4ktestwritededup --size=1M --dedupe_percentage=50

Inode: 14   Type: regular    Mode:  0644   Flags: 0x0
Generation: 2972670170    Version: 0x00000000
User:     0   Group:     0   Size: 1048576
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 2056
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x54640de6 -- Wed Nov 12 17:48:22 2014
atime: 0x54640de6 -- Wed Nov 12 17:48:22 2014
mtime: 0x54640de6 -- Wed Nov 12 17:48:22 2014
Size of extra inode fields: 4
BLOCKS:
(0-6):12288-12294, (7-11):12292-12296, (IND):30874, (12-22):12297-12307, (23-28):12294-12299, (29-39):12299-12309, (40-43):12302-12305, (44-46):12304-12306, (47-48):12305-12306, (49-52):12306-12309, (53-54):12307-12308, (55-57):12308-12310, (58-61):12310-12313, (62-63):12311-12312, (64-66):12312-12314, (67-70):12313-12316, (71-79):12316-12324, (80-83):12318-12321, (84-85):12320-12321, (86-88):12321-12323, (89-93):12322-12326, (94-95):12323-12324, (96-98):12324-12326, (99-105):12326-12332, (106-116):12329-12339, (117-120):12330-12333, (121-123):12332-12334, (124-129):12333-12338, (130-131):12337-12338, (132-133):12338-12339, (134-136):12339-12341, (137-138):12340-12341, (139-141):12341-12343, (142-144):12343-12345, (145-147):12345-12347, (148-152):12346-12350, (153-154):12348-12349, (155-157):12349-12351, (158-160):12350-12352, (161-162):12351-12352, (163):12352, (164):12360, (165-167):12360-12362, (168-169):12362-12363, (170-172):12363-12365, (173-177):12365-12369, (178-179):12367-12368, (180-181):12368-12369, (182-183):12369-12370, (184-187):12370-12373, (188-190):12373-12375, (191-193):12375-12377, (194-195):12376-12377, (196-198):12377-12379, (199-206):12378-12385, (207-208):12382-12383, (209-211):12383-12385, (212-219):12385-12392, (220-223):12390-12393, (224-225):12393-12394, (226-230):12394-12398, (231-237):12397-12403, (238-241):12398-12401, (242-249):12401-12408, (250-255):12402-12407

4. random dedup 50%

fio --directory=./ --direct=1 --rw=randwrite --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=100 --iodepth=1 --numjobs=1  --group_reporting --name=4ktestrandwritededup --size=1M --dedupe_percentage=50

Inode: 15   Type: regular    Mode:  0644   Flags: 0x0
Generation: 2972670171    Version: 0x00000000
User:     0   Group:     0   Size: 1048576
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 2056
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x54640dfd -- Wed Nov 12 17:48:45 2014
atime: 0x54640dfc -- Wed Nov 12 17:48:44 2014
mtime: 0x54640dfd -- Wed Nov 12 17:48:45 2014
Size of extra inode fields: 4
BLOCKS:
(0):30968, (1):30960, (2):30720, (3):30975, (4):30886, (5):30957, (6):30977, (7):30913, (8):30720, (9):30923, (10):30720, (11):30980, (IND):30875, (12):30924, (13):30906, (14):30941, (15):30720, (16):30906, (17):30954, (18):30933, (19):30720, (20):30900, (21):30979, (22):30720, (23):30720, (24):30720, (25):30910, (26):30935, (27):30720, (28):30903, (29):30720, (30):30720, (31):30964, (32):30720, (33):30720, (34):30720, (35):30720, (36):30944, (37):30720, (38):30966, (39):30720, (40):30720, (41):30720, (42):30720, (43):30896, (44):30720, (45):30946, (46):30981, (47-48):30970-30971, (49):30720, (50):30990, (51):30954, (52):30952, (53):30720, (54):30879, (55):30962, (56):30720, (57):30720, (58):30976, (59):30969, (60):30931, (61):30894, (62):30922, (63):30908, (64):30720, (65):30720, (66):30720, (67):30986, (68):30720, (69):30974, (70):30892, (71):30720, (72):30927, (73):30942, (74):30936, (75):30915, (76):30937, (77):30905, (78):30720, (79):30720, (80):30720, (81):30720, (82):30720, (83):30910, (84):30720, (85):30720, (86):30720, (87):30898, (88):30945, (89):30720, (90):30720, (91):30984, (92):30940, (93):30916, (94):30720, (95):30720, (96):30720, (97):30720, (98):30956, (99):30934, (100):30919, (101):30965, (102):30893, (103):30720, (104):30898, (105):30720, (106):30985, (107):30923, (108):30720, (109):30720, (110):30720, (111):30974, (112):30963, (113):30884, (114):30895, (115):30979, (116):30991, (117):30970, (118):30960, (119):30993, (120):30955, (121):30939, (122):30720, (123):30720, (124):30988, (125):30962, (126):30943, (127):30975, (128):30891, (129):30968, (130):30925, (131):30957, (132):30897, (133):30911, (134):30965, (135):30920, (136):30928, (137):30720, (138):30720, (139):30899, (140):30720, (141):30720, (142):30983, (143):30957, (144):30876, (145):30949, (146):30720, (147):30930, (148):30720, (149):30720, (150):30720, (151):30880, (152):30939, (153):30961, (154):30927, (155):30899, (156):30720, (157):30963, (158):30720, (159):30897, (160):30720, (161):30946, (162):30915, (163):30980, (164):30720, (165):30720, (166):30950, (167):30984, (168):30958, (169):30720, (170):30942, (171):30951, (172):30720, (173):30887, (174):30720, (175):30720, (176):30928, (177):30988, (178):30940, (179):30907, (180):30720, (181):30720, (182):30912, (183):30981, (184):30720, (185):30978, (186):30959, (187):30720, (188):30989, (189):30720, (190):30880, (191):30720, (192):30992, (193):30991, (194):30980, (195):30720, (196):30929, (197):30720, (198):30939, (199):30926, (200):30900, (201):30909, (202):30720, (203):30720, (204):30920, (205):30720, (206):30720, (207):30959, (208):30973, (209):30927, (210):30943, (211):30886, (212):30949, (213):30720, (214):30720, (215):30938, (216):30968, (217):30987, (218):30720, (219):30947, (220):30993, (221):30947, (222):30720, (223):30902, (224):30919, (225):30901, (226):30899, (227):30720, (228):30720, (229):30720, (230):30950, (231):30903, (232):30720, (233):30720, (234):30720, (235):30972, (236):30953, (237):30720, (238):30924, (239):30948, (240):30720, (241):30720, (242):30883, (243):30930, (244):30973, (245):30720, (246):30926, (247):30720, (248):30918, (249):30956, (250):30720, (251):30720, (252):30901, (253):30932, (254):30918, (255):30982
TOTAL: 257

1. 顺序写的情况一目了然，同一个文件确实是按照block号连续排列的。

2. 随机写可以看出里面产生了很多block号都指向同一个数据块(30720)，就我看来(30720)这个数据块存的是零数据，或者数据空洞，除了这一点，其他的块号都指向不同的数据块。从数据来看，block的排列上也被打乱了，或者产生的数据就是随机分布的。于是根据数据分布我做了两个图表，先看第一个，你会发现从第一个block到最后一个block如果是连续找block就会显得很分散。如果真是这样的话，磁头来回在磁道上寻址的话性能肯定很低，所以我对block号进行了一次排序，也就是第二张图。你发现了啥？其实它是连续的。也就是说同一个文件它的block号其实是连着的，跟下面的实验在普通ext4上结果是一致的，只是说经过dedup以后，它对文件的block的顺序给打乱了，但是也仅限于在连续的block号池子里的进行打乱。

按块的顺序排序得到的图，请注意与下面块号区分开。

按块号的顺序得到的图

3. 顺序写加上dedup可以看到数据块号重叠了，比如前面12个块((0-6):12288-12294, (7-11):12292-12296)，第4-5-6块数据，占用的块号是12292-12293-12294，第7-8-9同样也是指向这几个数据块，所以整个这个文件实际占用块总数为120个(12288-12288)，因为是50%的dedup percentage. 所以算下来差不多。

4. 随机写加上50%的dedup， 和随机写不加dedup参数得到的结果差不多，只是除了0数据或者空洞数据以外，其他的块也有相同的，也就是说dedup percentage的参数是起作用了的。

--------------------------------------------------华丽的分割线--------------------------------------------------

再来看看在普通的ext2文件系统上的情况，使用同样的命令。

从所以的情况来看，在没有dedup功能的文件系统上，顺序写和随机写是一样的，产生的是连续的block。只是说random里面有数据空洞或者零数据，加上dedup参数同理。

1. requence write
debugfs 1.41.12 (17-May-2010)
debugfs:  stat <12>
Inode: 12   Type: regular    Mode:  0644   Flags: 0x0
Generation: 1938776540    Version: 0x00000000
User:     0   Group:     0   Size: 1048576
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 2056
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x5464409a -- Thu Nov 13 00:24:42 2014
atime: 0x5464409a -- Thu Nov 13 00:24:42 2014
mtime: 0x5464409a -- Thu Nov 13 00:24:42 2014
Size of extra inode fields: 0
BLOCKS:
(0-11):1545-1556, (IND):1557, (12-255):1558-1801
TOTAL: 257

2. random write
Inode: 13   Type: regular    Mode:  0644   Flags: 0x0
Generation: 1938776541    Version: 0x00000000
User:     0   Group:     0   Size: 1048576
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 2056
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x54644117 -- Thu Nov 13 00:26:47 2014
atime: 0x54644117 -- Thu Nov 13 00:26:47 2014
mtime: 0x54644117 -- Thu Nov 13 00:26:47 2014
Size of extra inode fields: 0
BLOCKS:
(0-11):24576-24587, (IND):24588, (12-255):24589-24832
TOTAL: 257


3. sequence write dedup
debugfs:  stat <14>
Inode: 14   Type: regular    Mode:  0644   Flags: 0x0
Generation: 1938776542    Version: 0x00000000
User:     0   Group:     0   Size: 1048576
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 2056
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x5464414c -- Thu Nov 13 00:27:40 2014
atime: 0x5464414c -- Thu Nov 13 00:27:40 2014
mtime: 0x5464414c -- Thu Nov 13 00:27:40 2014
Size of extra inode fields: 0
BLOCKS:
(0-11):2048-2059, (IND):2060, (12-255):2061-2304
TOTAL: 257

4. random write dedup
debugfs:  stat <15>
Inode: 15   Type: regular    Mode:  0644   Flags: 0x0
Generation: 1938776543    Version: 0x00000000
User:     0   Group:     0   Size: 1048576
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 2056
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x54644157 -- Thu Nov 13 00:27:51 2014
atime: 0x54644157 -- Thu Nov 13 00:27:51 2014
mtime: 0x54644157 -- Thu Nov 13 00:27:51 2014
Size of extra inode fields: 0
BLOCKS:
(0-11):8192-8203, (IND):8204, (12-255):8205-8448
TOTAL: 257

--------------------------------------------------华丽的分割线--------------------------------------------------

总结：

1. FIO跑dedup-fs性能的时候不需要加上dedup参数，因为random这个参数产生的文件里面本身就包含大概37%到42%的数据空洞或者零数据。

2. 工具没法控制一个文件所占用的block的分布状况，一般情况下，同一个文件的block都会文件系统或者I/O调度器安排在相近的block。也就是说FIO的连续或者随机产生的数据在设备上都是连续分布的。

最后来一张I/O 流程图