Filebeat直接往ES中传输数据(按小时区分)、每小时建立索引时会有超时报错、图行会有断图
错误内容如下:
[node-7-master] failed to put mappings on indices [[[filebeat_nginx_turn.2018.10.09.18/rtQGejGWSCmB7ZvnEA-9_g]]], type [doc]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:124) ~[elasticsearch-6.3.2.jar:6.3.2]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_111]
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:123) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:626) [elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Filebeat配置如下:
filebeat.prospectors:
- paths:
- /data1/logs/nginx/jason_access.log
tail_files: true
json.keys_under_root: true
json.add_error_key: true
output.elasticsearch:
hosts: ["192.168.31.111:9200","192.168.31.109:9200","192.168.31.110:9200","192.168.5.165:9200","192.168.5.167:9200"]
worker: 32
index: "filebeat_nginx_turn.%{+yyyy.MM.dd.HH}"
setup.template.name: "filebeat_nginx"
setup.template.overwrite: false
setup.template.pattern: "filebeat_nginx*"
kinbana上看到的现象
错误内容如下:
[node-7-master] failed to put mappings on indices [[[filebeat_nginx_turn.2018.10.09.18/rtQGejGWSCmB7ZvnEA-9_g]]], type [doc]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:124) ~[elasticsearch-6.3.2.jar:6.3.2]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_111]
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:123) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:626) [elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Filebeat配置如下:
filebeat.prospectors:
- paths:
- /data1/logs/nginx/jason_access.log
tail_files: true
json.keys_under_root: true
json.add_error_key: true
output.elasticsearch:
hosts: ["192.168.31.111:9200","192.168.31.109:9200","192.168.31.110:9200","192.168.5.165:9200","192.168.5.167:9200"]
worker: 32
index: "filebeat_nginx_turn.%{+yyyy.MM.dd.HH}"
setup.template.name: "filebeat_nginx"
setup.template.overwrite: false
setup.template.pattern: "filebeat_nginx*"
kinbana上看到的现象
5 个回复
kennywu76 - Wood
赞同来自: fying99 、rochy
解决办法是控制集群的总的索引数量,shard数量。 如果同时创建的索引非常多,最好避免通过写入数据自动创建索引。 可以通过cron任务,预先顺序的创建好索引。
zqc0512 - andy zhou
赞同来自: fying99
机器怎样?量应该大吧,按照条数存吧,不按照小时,这个索引多了后,集群有压力的。
fying99
赞同来自:
优化线程池是否可以解决?如果优化该优化到多少
jerrymouse - 96后
赞同来自:
fying99
赞同来自: