uniq 去重 Linux shell，实现类似mysql的distinct的功能

来源：互联网发布：mysql 事件是否开启编辑：程序博客网时间：2024/05/05 22:41

uniq 命令去重，只能去掉连续的重复的项

test 测试文本

[shuohai.lhl@v035114 bin]$ cat test 711[10.235.170.22:8123]  0             10.232.20.178     1648042        1648042         0             2013-07-29 20:01:35   711[10.235.170.22:8123]  11            10.232.20.178     6445135        6129386         315749        2013-07-29 20:19:40   711[10.235.170.22:8123]  129           10.232.20.178     2220052        2220052         0             2013-07-29 19:33:04   711[10.235.170.22:8123]  111           10.232.20.178     4939938        4754688         185250        2013-07-29 20:19:40   711[10.235.170.22:8123]  123           10.232.20.178     284227         284227          0             2013-07-29 19:33:00   711[10.235.170.22:8123]  112           10.232.20.178     4393638        4262994         130644        2013-07-29 20:19:40   711[10.235.170.22:8123]  121           10.232.20.178     413891         413891          0             2013-07-29 19:33:00   711[10.235.170.22:8123]  130           10.232.20.178     2803552        2803552         0             2013-07-29 19:33:04   711[10.235.170.22:8123]  10            10.232.20.178     6558889        6235520         323369        2013-07-29 20:19:40   711[10.235.170.22:8123]  106           10.232.20.178     5080974        4886037         194937        2013-07-29 20:19:40   711[10.235.170.22:8123]  101           10.232.20.178     3655253        3655253         0             2013-07-29 20:19:20   711[10.235.170.22:8123]  124           10.232.20.178     424403         424403          0             2013-07-29 19:33:05   711[10.235.170.22:8123]  12            10.232.20.178     5507575        5271088         236487        2013-07-29 20:19:40   711[10.235.170.22:8123]  115           10.232.20.178     1936544        1936544         0             2013-07-29 20:04:40   711[10.235.170.22:8123]  100           10.232.20.178     2791119        2791119         0             2013-07-29 20:13:20   711[10.235.170.22:8123]  119           10.232.20.178     829978         829978          0             2013-07-29 19:48:05   711[10.235.170.22:8123]  109           10.232.20.178     5205710        4994482         211228        2013-07-29 20:19:40   711[10.235.170.22:8123]  114           10.232.20.178     2174793        2174793         0             2013-07-29 20:07:05   711[10.235.170.22:8123]  118           10.232.20.178     1119162        1119162         0             2013-07-29 19:54:20   711[10.235.170.22:8123]  113           10.232.20.178     2965206        2965206         0             2013-07-29 20:14:40   711[10.235.170.22:8123]  103           10.232.20.178     5642924        5396261         246663        2013-07-29 20:19:40   711[10.235.170.22:8123]  104           10.232.20.178     6269570        5970006         299564        2013-07-29 20:19:40   711[10.235.170.22:8123]  105           10.232.20.178     5939748        5665489         274259        2013-07-29 20:19:40   711[10.235.170.22:8123]  107           10.232.20.178     4796367        4623086         173281        2013-07-29 20:19:40   711[10.235.170.22:8123]  117           10.232.20.178     1662411        1662411         0             2013-07-29 20:01:40   711[10.235.170.22:8123]  125           10.232.20.178     283898         283898          0             2013-07-29 19:33:05   711[10.235.170.22:8123]  1             10.232.20.178     1669975        1669975         0             2013-07-29 20:01:45   711[10.235.170.22:8123]  126           10.232.20.178     1133277        1133277         0             2013-07-29 19:33:05   711[10.235.170.22:8123]  110           10.232.20.178     5399121        5176800         222321        2013-07-29 20:19:40   711[10.235.170.22:8123]  13            10.232.20.178     4599532        4599532         0             2013-07-29 19:33:04   711[10.235.170.22:8123]  116           10.232.20.178     1738524        1738524         0             2013-07-29 20:02:25   711[10.235.170.22:8123]  102           10.232.20.178     4360832        4230019         130813        2013-07-29 20:19:40   711[10.235.170.22:8123]  128           10.232.20.178     2534709        2534709         0             2013-07-29 19:33:04   711[10.235.170.22:8123]  120           10.232.20.178     715130         715130          0             2013-07-29 19:43:15   711[10.235.170.22:8123]  108           10.232.20.178     4900775        4720912         179863        2013-07-29 20:19:40   711[10.235.170.22:8123]  127           10.232.20.178     1386770        1386770         0             2013-07-29 19:33:04   711[10.235.170.22:8123]  122           10.232.20.178     424419         424419          0             2013-07-29 19:33:05   711[10.235.170.22:8123]  178           10.232.24.46      2962807        2962807         0             2013-07-29 19:33:05

打印第六列中不同值，先sort，然后在uniq

[shuohai.lhl@v035114 bin]$ cat test| awk '{print $6}' | sort |uniq -c     24 0      1 130644      1 130813      1 173281      1 179863      1 185250      1 194937      1 211228      1 222321      1 236487      1 246663      1 274259      1 299564      1 315749      1 323369