logstash中使用codec multiline合并行,使用filter grok匹配所需字段信息出错
Logstash | 作者 506281581 | 发布于2017年11月27日 | 阅读数:6030
使用logstash采集日志文件中的日志信息,
logstash启动配置文件logstash.conf配置如下:
1 input {
2 file{
3 path => "/mnt/DShare/nginx/access.log.1"
4 type => "nginx"
5 start_position => "beginning"
6 codec => multiline {
7 pattern => "[=]"
8 negate => "false"
9 what => "previous"
10 }
11 }
12 }
13 filter {
14 grok {
15 match => { "message" => "%{ERRORID:ErrorId} %{BAYID:bayId}"
16 }
17 }
18 }
19
20 output {
21 stdout{
22 codec => rubydebug
23 }
24}
file中使用codec的multiline插件进行行的合并,filter中使用grok正则匹配所需字段,问题是:单独匹配%{ERRORID:ErrorId} 或%{BAYID:bayId}时均能正常匹配数据,但当两个字段同时进行匹配时报"_grokparsefailure"错误。
输出结果
正常输出:
匹配pattern:ERRORID
"path" => "/mnt/DShare/nginx/access.log.1",
"ErrorId" => "Error005",
"@timestamp" => 2017-11-27T07:57:27.352Z,
"@version" => "1",
"host" => "ubuntu-VirtualBox",
"message" => "[2017-11-23 13:20:48,996],travelRecord convert failed, bayId:bay005, orientation:WEST, validate failed for CaptureDirection value : aaaa\r\n[2017-11-23 13:20:49,996],travelRecord convert failed, ErrorId=Error005\r",
"type" => "nginx",
"tags" => [
[0] "multiline"
]
}
匹配模式BAYID:
{
"path" => "/mnt/DShare/nginx/access.log.1",
"@timestamp" => 2017-11-27T07:45:31.364Z,
"bayId" => "bay007",
"@version" => "1",
"host" => "ubuntu-VirtualBox",
"message" => "[2017-11-23 13:20:40,996],travelRecord convert failed, bayId:bay007, orientation:SOUTH, validate failed for CaptureDirection value : aaaa\r\n[2017-11-23 13:20:49,996],travelRecord convert failed, ErrorId=Error007\r",
"type" => "nginx",
"tags" => [
[0] "multiline"
]
}
当同时匹配ERRORID和BAYID时解析错误,输出结果:
{
"path" => "/mnt/DShare/nginx/access.log.1",
"@timestamp" => 2017-11-27T08:00:53.351Z,
"@version" => "1",
"host" => "ubuntu-VirtualBox",
"message" => "[2017-11-23 13:20:40,996],travelRecord convert failed, bayId:bay007, orientation:SOUTH, validate failed for CaptureDirection value : aaaa\r\n[2017-11-23 13:20:49,996],travelRecord convert failed, ErrorId=Error007\r",
"type" => "nginx",
"tags" => [
[0] "multiline",
[1] "_grokparsefailure"
]
}
logstash启动配置文件logstash.conf配置如下:
1 input {
2 file{
3 path => "/mnt/DShare/nginx/access.log.1"
4 type => "nginx"
5 start_position => "beginning"
6 codec => multiline {
7 pattern => "[=]"
8 negate => "false"
9 what => "previous"
10 }
11 }
12 }
13 filter {
14 grok {
15 match => { "message" => "%{ERRORID:ErrorId} %{BAYID:bayId}"
16 }
17 }
18 }
19
20 output {
21 stdout{
22 codec => rubydebug
23 }
24}
file中使用codec的multiline插件进行行的合并,filter中使用grok正则匹配所需字段,问题是:单独匹配%{ERRORID:ErrorId} 或%{BAYID:bayId}时均能正常匹配数据,但当两个字段同时进行匹配时报"_grokparsefailure"错误。
输出结果
正常输出:
匹配pattern:ERRORID
"path" => "/mnt/DShare/nginx/access.log.1",
"ErrorId" => "Error005",
"@timestamp" => 2017-11-27T07:57:27.352Z,
"@version" => "1",
"host" => "ubuntu-VirtualBox",
"message" => "[2017-11-23 13:20:48,996],travelRecord convert failed, bayId:bay005, orientation:WEST, validate failed for CaptureDirection value : aaaa\r\n[2017-11-23 13:20:49,996],travelRecord convert failed, ErrorId=Error005\r",
"type" => "nginx",
"tags" => [
[0] "multiline"
]
}
匹配模式BAYID:
{
"path" => "/mnt/DShare/nginx/access.log.1",
"@timestamp" => 2017-11-27T07:45:31.364Z,
"bayId" => "bay007",
"@version" => "1",
"host" => "ubuntu-VirtualBox",
"message" => "[2017-11-23 13:20:40,996],travelRecord convert failed, bayId:bay007, orientation:SOUTH, validate failed for CaptureDirection value : aaaa\r\n[2017-11-23 13:20:49,996],travelRecord convert failed, ErrorId=Error007\r",
"type" => "nginx",
"tags" => [
[0] "multiline"
]
}
当同时匹配ERRORID和BAYID时解析错误,输出结果:
{
"path" => "/mnt/DShare/nginx/access.log.1",
"@timestamp" => 2017-11-27T08:00:53.351Z,
"@version" => "1",
"host" => "ubuntu-VirtualBox",
"message" => "[2017-11-23 13:20:40,996],travelRecord convert failed, bayId:bay007, orientation:SOUTH, validate failed for CaptureDirection value : aaaa\r\n[2017-11-23 13:20:49,996],travelRecord convert failed, ErrorId=Error007\r",
"type" => "nginx",
"tags" => [
[0] "multiline",
[1] "_grokparsefailure"
]
}
1 个回复
luohuanfeng
赞同来自: