Kibana Timelion Supports Percentiles

Introduction

Timelion has a good .es method which enables users to get certain metrics over the query results. The existing metrics include: count, avg, sum, min, max, cardinality. But I am eager to do percentile query with timelion, so how to make it?

P.S. Kibana Version in this post: 4.5.4

Solution

It is easy to modify the existing codes to make this work. But it will require some understanding of the codes. Let’s serve the code first, and explain it right now.

Changes 1

Changed file: installedPlugins/timelion/server/series_functions/es/lib/create_date_agg.js (the path is based on the relative path of kibana root directory)

Changed content

   dateAgg.time_buckets.aggs = {};   _.each(config.metric, function (metric, i) {     var metric = metric.split(':');     if (metric[0] === 'count') {       // This is pretty lame, but its how the "doc_count" metric has to be implemented at the moment       // It simplifies the aggregation tree walking code considerably       dateAgg.time_buckets.aggs[metric] = {         /*         bucket_script: {           buckets_path: '_count',           script: {inline: '_value', lang: 'expression'}         }         */         'avg': {'field': 'hack', 'missing': 0}       };     } else if (metric[0] && metric[1]) {        var metricName = metric[0] + '(' + metric[1] + ')';        dateAgg.time_buckets.aggs[metricName] = {};        dateAgg.time_buckets.aggs[metricName][metric[0]] = {field: metric[1]}; // the following is the newly added code       if (metric[0] == 'percentiles') {        var percentList = metric[2].split(',');        percentList = percentList.map(x => parseInt(x));        dateAgg.time_buckets.aggs[metricName][metric[0]] = {          field: metric[1],          percents: percentList        };      }      } else {        throw new Error ('`metric` requires metric:field or simply count');      }   });   return dateAgg; };

This file is used to construct the payload from timelion to Elasticsearch. So what we got to do here is to make up a payload body specific for query percentile metrics, which looks like:

{  field: 'filename',  percents: [1, 5, 25, 50, 75, 99]}

Changes 2

Changed file: installedPlugins/timelion/server/series_functions/es/lib/agg_response_to_series_list.js

Changed content:

 var _ = require('lodash'); export function timeBucketsToPairs(buckets) {   var timestamps = _.pluck(buckets, 'key');   var series = {};    _.each(buckets, function (bucket) {      _.forOwn(bucket, function (val, key) {        if (_.isPlainObject(val)) { // the following is the changed content        // percentiles values        if (val.values) {          _.forOwn(val.values, function (v, k) {             var k = key + ':' + k;             series[k] = series[k] || [];             series[k].push(v);           });         } else {           series[key] = series[key] || [];           series[key].push(val.value);         }        }      });    });

This file is responsible for extracting the response from Elasticsearch. A typical response looks like the following:

{    "key": "Mobile_Web_Tablet",    "doc_count": 3,    "sum:fieldA": {        "value": 0.0    },    "avg:fieldB": {        "value": 1    },    "percentiles:fieldC": {        "values": {            "1.0": 1000.0,            "5.0": 1000.0,            "25.0": 1000.0,            "50.0": 1000.0,            "75.0": 1000.0,            "95.0": 1000.0,            "99.0": 1000.0        }    }}

As you can see, for avg, sum, min, max, you can just extract value from val.value, but for percentiles, it is not that straight forward. So please refer to the code to find out.

Usage of percentiles

Well, you are good to restart kibana now, and enjoy the timelion that supports percentile.
Now it supports the following metric, just feel free to use percentiles. The format is percentiles:fieldName:percentileA,percentileB,percentileC...

.es(q='_type:session', metric='percentiles:time_to_playback_start:1,5,25,50,75,99')

You are supposed to get a result like this:

Contact me

If you got any question, you are welcome to contact me via:

• email: [e-mail]
• wechat: nisxiya