2017/10/08 15:17:12.134290 client.go:214: DBG Publish: {
"@timestamp": "2017-10-08T15:17:02.134Z",
"beat": {
"hostname": "izuf6f8i2ttdohwal5u4pzz",
"name": "izuf6f8i2ttdohwal5u4pzz",
"version": "5.6.2"
},
"input_type": "log",
"message": "23:53:18.061 [main] ERROR - Context initialization failed",
"offset": 58,
"source": "/mnt/data/usr/elasticsearch-5.x/beats/simple-logs/hello.txt",
"tags": [
"datalake-log"
],
"type": "log"
}
接下来第二次事件offset为116，但是会将上一次发送时的offset读取重新发送上一次的事件，推送两次Event，以此类推
Publish: {
"@timestamp": "2017-10-08T15:22:24.665Z",
"beat": {
"hostname": "izuf6f8i2ttdohwal5u4pzz",
"name": "izuf6f8i2ttdohwal5u4pzz",
"version": "5.6.2"
},
"input_type": "log",
"message": "23:53:18.061 [main] ERROR - Context initialization failed",
"offset": 58,
"source": "/mnt/data/usr/elasticsearch-5.x/beats/simple-logs/hello.txt",
"tags": [
"datalake-log"
],
"type": "log"
}
Publish: {
"@timestamp": "2017-10-08T15:22:24.665Z",
"beat": {
"hostname": "izuf6f8i2ttdohwal5u4pzz",
"name": "izuf6f8i2ttdohwal5u4pzz",
"version": "5.6.2"
},
"input_type": "log",
"message": "23:53:18.061 [main] ERROR - Context initialization failed",
"offset": 116,
"source": "/mnt/data/usr/elasticsearch-5.x/beats/simple-logs/hello.txt",
"tags": [
"datalake-log"
],
"type": "log"
}
导致logstash重复消费数据，我不知道是我的问题还是beats内部机制本来就这样，有办法避免这个情况吗？
filebeat.yml配置：
filebeat.config.prospectors:
path: /etc/filebeat/filebeat.yml
reload.enabled: false
reload.period: 10s
filebeat.prospectors:
- input_type: log
paths:
- /mnt/data/usr/elasticsearch-5.x/beats/simple-logs/*
include_lines: ["^.*] ERROR.*", "^.*] WARN.*"]
multiline.pattern: ^(?:[01]\d|2[0-3])(?::[0-5]\d){2}\.(\d){3} \[
multiline.negate: true
multiline.match: after
tags: ["datalake-log"]
output.logstash:
hosts: ["localhost:5044"]
logging.level: debug
logging.to_files: true
logging.to_syslog: false
logging.files:
path: /mnt/data/usr/elasticsearch-5.x/beats/logs
name: mybeat.log
keepfiles: 7
logging.selectors: ["*"]
4 个回复
juneryang
赞同来自:
medcl - 今晚打老虎。
赞同来自:
Ohhhhhhhhhhhh!
赞同来自:
zdy
赞同来自:
你怎么修改文件。你试过了$ echo 'new line' >> /usr/src/app/log.log吗？filebeat试图根据元数据来检测文件是否是新的。一些工具/编辑器创建一个全新的文件保存并取消旧链接。这被filebeat检测为新文件，迫使它重新发送所有内容。