jq学习使用基础笔记

来源：互联网发布：视频清晰度增强软件编辑：程序博客网时间：2024/05/22 01:45

使用shell脚本解析json数据不像java或者python一样轻便,较为麻烦,还好有一个jq.其使用起来比较像grep ,awk,sed等命令,也是管道命令.此外jq也没有乱七八槽的依赖,只需要一个binary文件即可.其官网介绍见这里{:target=”_blank”}

官方帮助文档见这里

安装jq

在Ubuntu系统下:

apt install jq

使用

有json数据文件example.json,其内容如下:

{ "took" : 39, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 325305, "max_score" : 0.0, "hits" : [ ] }, "aggregations" : { "group" : { "doc_count_error_upper_bound" : -1, "sum_other_doc_count" : 319035, "buckets" : [ { "key" : "http://www.csdn.net/", "doc_count" : 1334, "importTotalSuccess" : { "value" : 9580402.0 }, "importTotalTime" : { "value" : 2.473873E7 }, "importTotalAll" : { "value" : 1.2589526E7 }, "importTotalRepeat" : { "value" : 3006465.0 }, "importTotalSize" : { "value" : 5.40141167E8 } }, { "key" : "https://www.baidu.com/", "doc_count" : 1346, "importTotalSuccess" : { "value" : 8749457.0 }, "importTotalTime" : { "value" : 1.712813E7 }, "importTotalAll" : { "value" : 1.0137599E7 }, "importTotalRepeat" : { "value" : 1385769.0 }, "importTotalSize" : { "value" : 4.62986378E8 } }, { "key" : "http://www.cnblogs.com/", "doc_count" : 3590, "importTotalSuccess" : { "value" : 7106773.0 }, "importTotalTime" : { "value" : 2.9117391E7 }, "importTotalAll" : { "value" : 9490425.0 }, "importTotalRepeat" : { "value" : 2380536.0 }, "importTotalSize" : { "value" : 5.16549393E8 } } ] } } }

使用jq格式化json数据(也可以检验json数据是否规范):

jq '.' ./example.json

输出结果:

{  "took": 39,  "timed_out": false,  "_shards": {    "total": 5,    "successful": 5,    "failed": 0  },  "hits": {    "total": 325305,    "max_score": 0,    "hits": []  },  "aggregations": {    "group": {      "doc_count_error_upper_bound": -1,      "sum_other_doc_count": 319035,      "buckets": [        {          "key": "http://www.csdn.net/",          "doc_count": 1334,          "importTotalSuccess": {            "value": 9580402          },          "importTotalTime": {            "value": 24738730          },          "importTotalAll": {            "value": 12589526          },          "importTotalRepeat": {            "value": 3006465          },          "importTotalSize": {            "value": 540141167          }        },        {          "key": "https://www.baidu.com/",          "doc_count": 1346,          "importTotalSuccess": {            "value": 8749457          },          "importTotalTime": {            "value": 17128130          },          "importTotalAll": {            "value": 10137599          },          "importTotalRepeat": {            "value": 1385769          },          "importTotalSize": {            "value": 462986378          }        },        {          "key": "http://www.cnblogs.com/",          "doc_count": 3590,          "importTotalSuccess": {            "value": 7106773          },          "importTotalTime": {            "value": 29117391          },          "importTotalAll": {            "value": 9490425          },          "importTotalRepeat": {            "value": 2380536          },          "importTotalSize": {            "value": 516549393          }        }      ]    }  }}

2.获取某键的值,以hits.total值为例:

jq '.hits.total' ./example.json

输出:

如果元素不存在,返回null

3.解析数组,以aggregations.group.buckets为例:

jq '.aggregations.group.buckets[0]' ./example.json

输出:

{  "key": "http://www.csdn.net/",  "doc_count": 1334,  "importTotalSuccess": {    "value": 9580402  },  "importTotalTime": {    "value": 24738730  },  "importTotalAll": {    "value": 12589526  },  "importTotalRepeat": {    "value": 3006465  },  "importTotalSize": {    "value": 540141167  }}

4.自定义输出格式,获取json中某些key的值,并组合为新的json串

首先以获取aggregations.group.buckets数组中的第一个元素的键key和doc_count的值并更改键名为A和B为例:

jq '.aggregations.group.buckets[0] | {A:.key,B:.doc_count}

输出:

{  "A": "http://www.csdn.net/",  "B": 1334}

获取aggregations.group.buckets数组中所有元素的键key和doc_count的值:

jq '.aggregations.group.buckets[] | {A:.key,B:.doc_count}' ./example.json

输出:

{  "A": "http://www.csdn.net/",  "B": 1334}{  "A": "https://www.baidu.com/",  "B": 1346}{  "A": "http://www.cnblogs.com/",  "B": 3590}

5.自定义输出多项为数组的形式,只要在上一步的命令两端加个中括号即可:

jq '[.aggregations.group.buckets[] | {A:.key,B:.doc_count}]' ./example.json

输出:

[  {    "A": "http://www.csdn.net/",    "B": 1334  },  {    "A": "https://www.baidu.com/",    "B": 1346  },  {    "A": "http://www.cnblogs.com/",    "B": 3590  }]

6.默认情况下,jq规范地打印输出json,通过查阅jq帮助文档可知使用参数-c能够简洁的将每个json对象打印一行:

jq -c '[.aggregations.group.buckets[] | {A:.key,B:.doc_count}]' ./example.json

输出

[{"A":"http://www.csdn.net/","B":1334},{"A":"https://www.baidu.com/","B":1346},{"A":"http://www.cnblogs.com/","B":3590}]

7.提取某些项以不含键的数组形式输出:

jq -c  '[.aggregations.group.buckets[] | .key,.importTotalSuccess.value,.importTotalTime.value,.importTotalAll.value,.importTotalRepeat.value,.importTotalSize.value]' ./example.json

输出:

["http://www.csdn.net/",9580402,24738730,12589526,3006465,540141167,"https://www.baidu.com/",8749457,17128130,10137599,1385769,462986378,"http://www.cnblogs.com/",7106773,29117391,9490425,2380536,516549393]

以每个aggregations.group.buckets数组中的元素作为一行打印此数组中的值:

jq -c  '.aggregations.group.buckets[] | [.key,.importTotalSuccess.value,.importTotalTime.value,.importTotalAll.value,.importTotalRepeat.value,.importTotalSize.value]' ./example.json

输出:

["http://www.csdn.net/",9580402,24738730,12589526,3006465,540141167]["https://www.baidu.com/",8749457,17128130,10137599,1385769,462986378]["http://www.cnblogs.com/",7106773,29117391,9490425,2380536,516549393]

8.内建函数
jq包含一些内建函数:keys,has,’+’,’-‘,’*’,’/’,’%’,length,in等等见官方帮助文档.其中keys获取json中的key元素.has判断是否存在某key:

jq 'keys' ./example.json

输出:

[  "_shards",  "aggregations",  "hits",  "timed_out",  "took"]

jq 'has("hits")' ./example.json

输出:

true

其他

注意区分json对象,json字符串以及json格式的数据之间的区别:
json对象:

{    a: 1,    b: {        c: "abc"    }}

json字符串

'{"a":1,"b":{"c":"abc"}}'

正确的json格式:

{    "a": 1,    "b": {        "c": "abc"    }}

0 0