用了Elasticsearch,一口气上5T

filebeat-output-kafka报错

Beats | 作者 Lincoln | 发布于2018年03月02日 | 阅读数:13506

架构filebeat+kafka+elk
 
 
filbeat配置
 
#------------------------------- Kafka output ----------------------------------
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["kfk1:9092", "kfk2:9092"]

# message topic selection + partitioning
#topic: '%{[fields.log_topic]}'
topic: 'citymain77'
partition.round_robin:
reachable_only: false

required_acks: 1
compression: gzip
max_message_bytes: 1000000


 
filebeat警告信息
2018-03-02T09:53:31+08:00 INFO Harvester started for file: /app/log/citymain.log.2017-11-10.log
2018-03-02T09:53:31+08:00 INFO Harvester started for file: /app/log/citymain.log.2017-12-04.log
2018-03-02T09:53:32+08:00 WARN kafka message: Initializing new client
2018-03-02T09:53:32+08:00 WARN client/metadata fetching metadata for all topics from broker 10.25.80.146:9092

2018-03-02T09:53:32+08:00 WARN Connected to broker at 10.25.80.146:9092 (unregistered)

2018-03-02T09:53:32+08:00 WARN client/brokers registered new broker #232 at szd-l0086693.siapp.h:9092
2018-03-02T09:53:32+08:00 WARN client/brokers registered new broker #233 at szd-l0086692.siapp.h:9092
2018-03-02T09:53:32+08:00 WARN kafka message: Successfully initialized new client
2018-03-02T09:53:32+08:00 WARN client/metadata fetching metadata for [citymain231] from broker 10.25.80.146:9092

2018-03-02T09:53:32+08:00 WARN kafka message: client/metadata found some partitions to be leaderless
2018-03-02T09:53:32+08:00 WARN client/metadata retrying after 250ms... (3 attempts remaining)

2018-03-02T09:53:32+08:00 WARN client/metadata fetching metadata for [citymain231] from broker 10.25.80.146:9092

2018-03-02T09:53:32+08:00 WARN producer/broker/233 starting up

2018-03-02T09:53:32+08:00 WARN producer/broker/233 state change to [open] on citymain231/0

2018-03-02T09:53:32+08:00 WARN Connected to broker at szd-l0086692.siapp.h:9092 (registered as #233)
已邀请:

locatelli

赞同来自:

可以检查:
-- 'describe'你的topic,看是不是有partition没有leader
-- 如果所有partition都正常,下一步排除通信问题,从客户端到broker的通信时间如果太长也有可能会造成类似的错误。比如你的bootstrap server写的是kfk1,log里显示的是szd-xxxx,应该尽量使用broker的FQDN

Lincoln - 80s

赞同来自:

--describ topic 
/bin/kafka-topics.sh --zookeeper zk-host:2181 --describe --topic tpc77
18/03/01 16:59:34 INFO zkclient.ZkClient: zookeeper state changed (SyncConnected)
Topic:citymain77 PartitionCount:1 ReplicationFactor:1 Configs:
Topic: tpc77 Partition: 0 Leader: 233 Replicas: 233 Isr: 233
18/03/01 16:59:34 INFO zkclient.ZkEventThread: Terminate ZkClient event thread.
18/03/01 16:59:34 INFO zookeeper.ZooKeeper: Session: 0x161daa6c9640d29 closed
18/03/01 16:59:34 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x161daa6c9640d29


--logstash配置.conf
input{
kafka{
bootstrap_servers => ["10.25.80.146:9092,10.25.80.129:9092"]
auto_offset_reset => "latest"
decorate_events => true
topics => ["citymain77"]
type => "citymain77"
}
}
output {
if [type] == "citymain77" {
elasticsearch {
hosts => ["10.20.112.229:9200","10.20.112.231:9200"]
manage_template => false
index => "cty77-%{+YYYY.MM.dd}"
document_type => "log"
}
}

if [type] == "citymain231" {
elasticsearch {
hosts => ["10.20.112.229:9200","10.20.112.231:9200"]
manage_template => false
index => "cty231-%{+YYYY.MM.dd}"
document_type => "log"
}
}
}




 
--logsatsh日志
[root@SZB-L0052000 conf.d]# cat /var/log/logstash/logstash-plain.log
[2018-03-01T17:55:33,937][ERROR][logstash.inputs.kafka ] Invalid setting for kafka input plugin:

input {
kafka {
# This setting must be a string
# Expected string, got ["10.25.80.146:9092", "10.25.80.129:9092"]
bootstrap_servers => ["10.25.80.146:9092", "10.25.80.129:9092"]
...
}
}
[2018-03-01T17:55:33,958][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Something is wrong with your configuration."}
[root@SZB-L0052000 conf.d]#

zhangmeng

赞同来自:

想请教一些问题,能加个qq吗

要回复问题请先登录注册