pandas.read_csv——分块读取大文件

来源：互联网发布：java 兼职编辑：程序博客网时间：2024/05/22 17:11
    read_csv中有个参数chunksize，通过指定一个chunksize分块大小来读取文件，返回的是一个可迭代的对象TextFileReader，IO Tools 举例如下：In [138]: reader = pd.read_table('tmp.sv', sep='|', chunksize=4)In [139]: readerOut[139]: <pandas.io.parsers.TextFileReader at 0x120d2f290>In [140]: for chunk in reader:   .....:     print(chunk)   .....:    Unnamed: 0         0         1         2         30           0  0.469112 -0.282863 -1.509059 -1.1356321           1  1.212112 -0.173215  0.119209 -1.0442362           2 -0.861849 -2.104569 -0.494929  1.0718043           3  0.721555 -0.706771 -1.039575  0.271860   Unnamed: 0         0         1         2         30           4 -0.424972  0.567020  0.276232 -1.0874011           5 -0.673690  0.113648 -1.478427  0.5249882           6  0.404705  0.577046 -1.715002 -1.0392683           7 -0.370647 -1.157892 -1.344312  0.844885   Unnamed: 0         0        1         2         30           8  1.075770 -0.10905  1.643563 -1.4693881           9  0.357021 -0.67460 -1.776904 -0.968914    指定iterator=True 也可以返回一个可迭代对象TextFileReader ：In [141]: reader = pd.read_table('tmp.sv', sep='|', iterator=True)In [142]: reader.get_chunk(5)Out[142]:    Unnamed: 0         0         1         2         30           0  0.469112 -0.282863 -1.509059 -1.1356321           1  1.212112 -0.173215  0.119209 -1.0442362           2 -0.861849 -2.104569 -0.494929  1.0718043           3  0.721555 -0.706771 -1.039575  0.2718604           4 -0.424972  0.567020  0.276232 -1.087401
阅读全文
0 0