ES集群性能问题，求大神帮忙优化下es写入性能

目前有3台服务器,分别都装了logstash和es。通过filebeat来给logstash抓数据，logstash给es。
 
3台服务器做es集群。其中两台同时部署了logstash来做数据过滤。2台filebeat不在这3台里。
      补充下3台服务器（虚拟机）的配置，都是8核、32G。es内存设置maxmas都是8G。logstash也是设置的8G。
 
单独的filebeat给logstash的性能测试了，2个filebeat，一共每秒200条数据给两个节点的logstash，输出到文件，没什么压力。
但是同样的，2个filebeat，一共每秒20条(没写错，就是20每秒条)数据给两个节点的logstash，输出到es。前几分钟发现还可以，没有积压的队列什么的。过了周末2天后，今天来了一看，原本两天应该入es里800多万数据的，结果查了下es里现在才250万左右，并且logstash一直在报
[WARN ] 2018-06-25 09:12:59.849 [Ruby-0-Thread-12@[main]>worker3: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384] elasticsearch - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elastic:xxxxxx@10.14.125.5:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://elastic:xxxxxx@10.14.125.5:9200/, :error_message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@10.14.125.5:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
 
[ERROR] 2018-06-25 09:12:59.841 [Ruby-0-Thread-10@[main]>worker1: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384] elasticsearch - Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@10.14.125.4:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
 
虽然在报上述的错误，但是时不时也会打印几条成功的日志，间断性给es传输。
查了下logstash的events事件，其中一台为：
 
curl -s localhost:9600/_node/stats/events?pretty=true
{
  "host" : "elk2",
  "version" : "6.2.2",
  "http_address" : "127.0.0.1:9600",
  "id" : "e2582e3-b96-4b7aae0e-5ea0c3be8c3",
  "name" : "elk2",
  "events" : {
    "in" : 993336,
    "filtered" : 991304,
    "out" : 990304,
    "duration_in_millis" : 1490312865,
    "queue_push_duration_in_millis" : 1027666576
  }
 
持续监控这个events事件，发现in的数据随着 out的增加，而增加。
 
上面的这些，基本应该可以断定是es的写入性能赶不上logstash和日志量。
下面是我es的配置，和index的索引模板设置。
 
es的配置:
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

http.cors.enabled: true
http.cors.allow-origin: "*"
#http.cors.allow-headers: "Authorization"

network.host: 0.0.0.0
http.port: 9200

cluster.name: icelk
node.name: node4

node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["10.14.125.4","10.14.125.5","10.14.125.3"]
discovery.zen.minimum_master_nodes: 2

action.auto_create_index : .security,.monitoring*,.watches,.triggered_watches,.watcher-history*,*.iclog,*.info,*.*
xpack.security.enabled : true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.monitoring.enabled : true
xpack.graph.enabled : false
xpack.watcher.enabled : true
xpack.ml.enabled : false
 
 
index的索引模板:
{
"index_patterns": "index-visitpage*",
"settins": {
"index.merge.policy.max_merged_segment": "2gb",
"index.merge.policy.segments_per_tier": "24",
"index.number_of_replicas": "1",
"index.number_of_shards": "5",
"index.optimize_auto_generated_id": "true",
"index.refresh_interval": "300s",
"index.routing.allocation.total_shards_per_node": "-1",
"index.unassigned.node_left.delayed_timeout": "20m"
},
"mappings": {
"doc": {
"properties": {
"app_version": {
"type": "keyword"
},
"appname": {
"type": "keyword"
},
"cpu_core": {
"type": "keyword"
},
"cpu_speed": {
"type": "keyword"
},
"device_brand": {
"type": "keyword"
},
"device_id": {
"type": "keyword"
},
"device_model": {
"type": "keyword"
},
"device_resolution": {
"type": "keyword"
},
"device_root": {
"type": "keyword"
},
"device_system_version": {
"type": "keyword"
},
"device_type": {
"type": "keyword"
},
"document_type": {
"type": "keyword"
},
"network_operator": {
"type": "keyword"
},
"network_version": {
"type": "keyword"
},
"offset": {
"type": "long"
},
"ram_size": {
"type": "keyword"
},
"visit_sessionid": {
"type": "keyword"
}
}
}
}
}
 
 
各位大神帮忙优化下es提高写入性能，
 
补充下es运行这两天的图：
 

 
各位大神帮忙优化下es提高写入性能，
 
各位大神帮忙优化下es提高写入性能，
 
各位大神帮忙优化下es提高写入性能，
 
 补充下：3台es配置的都是可为主节点、可数据节点。logstash的output配置的三个es的ip都写成数组了，都会连。
 
 补充下3台服务器的IO，我不会看！！！！！请大神帮忙看下是哪遇到瓶颈了！