居然是你

使用filebeat6.0遇到的问题.

Beats | 作者 luohuanfeng | 发布于2017年11月29日 | 阅读数:13145

filebeat版本6.0,kafka版本0.11.0.1.
filebeat配置如下:
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- type: log
paths:
- /var/log/lhf
document_type: lhftcc
#============================= Filebeat modules ===============================

filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml

# Set to true to enable config reloading
reload.enabled: false

# Period on which files under path should be checked for changes
#reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false

#================================ kafka =====================================
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["test02:9092"]

# message topic selection + partitioning
topic: '%{[type]}'
partition.round_robin:
reachable_only: false

version: 0.10.0.0
required_acks: 1
compression: none
max_message_bytes: 1000000
启动之后,手动向/var/log/lhf插入数据.
filebeat日志如下:
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer

在kafka上没有看到创建topic.
如果使用filebeat5.5的话,就一切正常.
---------------------------------------
补充
1,手动插入数据后,filebeat的那个日志是一直重复刷新的.上面只是节选一段.
2,filebeat下载的是6.0的tar包.解压后修改了yml,直接使用./filebeat启动的.
3,filebeat5.5可以正常的使用,主要是想试试6.0版本Filebeat prospectors中的document_type有没有被禁用.如果禁用了的话,topic: '%{[type]}'要如何获取文档类型.
 
已邀请:

keyso

赞同来自: luohuanfeng

新增字段log_topic,
fields:  log_topic: test
 
topic: '%{[fields.log_topic]}'

zhangmeng

赞同来自:

有问题想请教,能留个联系方式吗?

要回复问题请先登录注册