debug date filter in logstash
来源:互联网 发布:淘宝运营计划方案 编辑:程序博客网 时间:2024/03/28 23:03
Background
Recently some guys came by and were curious about the date filter in logstash (both 2.4 and 5.x, but I am using logstash 2.4 in this post). The basic function for date filter is to convert a free style date string to a timestamp (or datetime), which shall be used to set @timestamp
of the this message.
But it seemed that people often came across issues during this procedure, so I post this passage to make this whole process easier.
PS. this post does not contain much information, and is quite easy for understanding. It serves as a quick help to new guys here.
Solution
In this post, I am going to set up a debug solution of debugging date filter in logstash. With this set up, we can:
- directly set the input to logstash
- set up filters in logstash, add debug into through ruby filter
- output in debug mode
Step 1 setup logstash.json
Download logstash 2.4 or logstash 5.X and unzip them to your local PC.
add a logstash.json
in this folder (or anywhere you like). The content of this file may look like the following.
input { stdin {}}filter { # used for convert a customized date string to a timestamp # which will be set as '@timestamp' field of the event date { # message means the full body of a single incoming event match => [ "message", "dd/MMM/yyyy:HH:mm:ss Z"] } # used to add additional field to this event ruby { # convert timestamp to customized time string init => "require 'date'" # BUT be careful, for logstash 5.X, you cannot direct set event's field, but instead you can use event.set method to make it. Please refer here: https://www.elastic.co/guide/en/logstash/current/event-api.html code => "event['@testST'] = DateTime.strptime('1318996912','%s').strftime('%a, %d %b %Y')" }}output { stdout { codec => rubydebug }}
I will explain some items of the configuration here.
input { stdin {}}
This will take the input data directly from standard input. So later we can freely set the input data through terminal.
output { stdout { codec => rubydebug }}
It sets the output with fairly debug info.
As for the other filters in the logstash, please refer to the official doc.
Step 2 off you go
With logstash.json created, now let’s play.
echo "14/Jul/2017:00:57:29 +0800" | bin/logstash -e -f ./logstash.json
In this demo, I am feeding logstash with a single piece of message 14/Jul/2017:00:57:29 +0800
, which is referred as message
in logstash configuration.
Our task here is to extract the date string and put that date as the @timestamp
of this message. (Otherwise logstash is going to set @timestamp
to when it receives this message, which might not meet our need)
You can check with the output and see whether your converting with date filter
is correctly working. If it is not working as expected, you can easily change your logstash configuration file and test again! It saves a lot of time.
Contact me
If you get any question, you are welcome to contact me via:
- email: nisxiya@yeah.net
- wechat: nisxiya
- debug date filter in logstash
- logstash 中filter中的date插件
- logstash date插件介绍
- logstash date插件
- Logstash Filter 配置
- Logstash学习--Filter
- logstash filter插件
- logstash filter 学习
- Logstash filter插件开发
- Logstash Filter学习
- LogStash的Filter的使用
- Logstash学习记录--logstash input output filter 插件总结
- logstash input output filter 插件总结
- logstash-filter-elasticsearch 具体使用实例
- logstash 学习三 过滤器插件(Filter)
- Logstash中grok filter example例子
- Logstash input output filter插件总结
- Logstash中ruby filter的调试器
- 谷歌电子市场开发流程(8)-自定义控件
- C++中函数重载的实现原理
- html之marquee详解
- 51Nod-1500-苹果曼和树
- html5新增标签
- debug date filter in logstash
- vue 组件之间使用eventbus传值
- 51Nod-1510-最小化序列
- 可编辑div:contenteditable之ctrl+enter换行,enter发送
- 线性表(List)---链式存储结构(双向链表)
- VirtualBox 创建com对象失败 应用程序被中断
- 1021 Fibonacci Again
- 1019 Least Common Multiple
- vue-cli更换端口号