提问要多花一点心思哦

什么情况会集群uuid不一致导致加入集群一直失败

Elasticsearch | 作者 Sagittar | 发布于2020年12月28日 | 阅读数：4754
分享到：QQ空间新浪微博微信QQ好友印象笔记有道云笔记
软件版本：7.3.1
上下文：项目中使用ES，项目升级后会重启ES（3个ES节点）（ES不升级只是重启），升级过程不同步，有些快有些慢。升级完成后进行重启操作。
 
这几个月也遇到了几次
 node1 [node_2409:8086:8110:3f0::2d]
node2 [node_2409:8086:8110:3f0::2e]
node3 [node_2409:8086:8110:3f0::2f]

node1


[2020-12-23T02:18:36,155][INFO ][o.e.e.NodeEnvironment    ] [node_2409:8086:8110:3f0::2f] using [1] data paths, mounts [[/opt/oss (/dev/mapper/vg_ext-lv_opt)]], net usable_space [33gb], net total_space [39.1gb], types [ext4]

[2020-12-23T02:18:36,158][INFO ][o.e.e.NodeEnvironment    ] [node_2409:8086:8110:3f0::2f] heap size [989.8mb], compressed ordinary object pointers [true]

[2020-12-23T02:18:36,160][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] node name [node_2409:8086:8110:3f0::2f], node ID [2dFgJYXoTyWJ12es9CuiCQ], cluster name [Eam_ES_Cluster_Product]

[2020-12-23T02:18:36,161][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] version[7.3.1], pid[20151], build[oss/tar/4749ba6/2019-08-19T20:19:25.651794Z], OS[Linux/3.10.0-862.14.1.5.h442.eulerosv2r7.x86_64/amd64], JVM[Huawei Technologies Co., Ltd/OpenJDK 64-Bit Server VM/1.8.0_242/25.242-b08]

[2020-12-23T02:18:36,162][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] JVM home [/opt/oss/rtsp/jre-2.40.16-euler_amd64]

[2020-12-23T02:18:36,162][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] JVM arguments [-Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-6609433216513750091, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Djava.security.policy=/opt/oss/envs/Product-DVEamElasticsearchService/20201223021711464/es/config/security.policy, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/oss/SOP/DVEamElasticsearchService/logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=20m, -Des.insecure.allow.root=false, -Dprocname=dveamelasticsearchservice-3-0, -DNFW=dveamelasticsearchservice-3-0, -DTOMCAT_LOG_DIR=/var/log/oss/SOP/DVEamElasticsearchService/dveamelasticsearchservice-3-0/tomcatlog, -DTOMCAT_WORK_DIR=/var/log/oss/SOP/DVEamElasticsearchService/dveamelasticsearchservice-3-0/tomcatlog, -Djna.tmpdir=/var/log/oss/SOP/DVEamElasticsearchService/tmp, -Dio.netty.allocator.type=unpooled, -Des.path.home=/opt/oss/envs/Product-DVEamElasticsearchService/20201223021711464/es, -Des.path.conf=/opt/oss/envs/Product-DVEamElasticsearchService/20201223021711464/es/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=false]

[2020-12-23T02:18:37,955][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [aggs-matrix-stats]

[2020-12-23T02:18:37,956][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [analysis-common]

[2020-12-23T02:18:37,957][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [ingest-common]

[2020-12-23T02:18:37,957][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [ingest-geoip]

[2020-12-23T02:18:37,957][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [ingest-user-agent]

[2020-12-23T02:18:37,959][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [lang-expression]

[2020-12-23T02:18:37,959][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [lang-mustache]

[2020-12-23T02:18:37,959][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [lang-painless]

[2020-12-23T02:18:37,959][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [mapper-extras]

[2020-12-23T02:18:37,960][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [parent-join]

[2020-12-23T02:18:37,960][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [percolator]

[2020-12-23T02:18:37,960][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [rank-eval]

[2020-12-23T02:18:37,960][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [reindex]

[2020-12-23T02:18:37,961][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [repository-url]

[2020-12-23T02:18:37,961][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [transport-netty4]

[2020-12-23T02:18:37,962][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded plugin [digitalview-ssl]

[2020-12-23T02:18:42,785][INFO ][o.e.d.DiscoveryModule    ] [node_2409:8086:8110:3f0::2f] using discovery type [zen] and seed hosts providers [settings]

[2020-12-23T02:18:43,350][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] initialized

[2020-12-23T02:18:43,350][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] starting ...

[2020-12-23T02:18:43,661][INFO ][o.e.t.TransportService   ] [node_2409:8086:8110:3f0::2f] publish_address {[2409:8086:8110:3f0::2f]:9300}, bound_addresses {[2409:8086:8110:3f0::2f]:9300}

[2020-12-23T02:18:43,678][INFO ][o.e.b.BootstrapChecks    ] [node_2409:8086:8110:3f0::2f] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2020-12-23T02:18:45,245][INFO ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2f] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-node_2409:8086:8110:3f0::2d,2dFgJYXoTyWJ12es9CuiCQ,2H9c-7LTT-eha7ICDCCqcA}

[2020-12-23T02:18:45,667][INFO ][o.e.c.s.MasterService    ] [node_2409:8086:8110:3f0::2f] elected-as-master ([2] nodes joined)[{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim} elect leader, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}]}, added {{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim},}

[2020-12-23T02:18:45,783][INFO ][o.e.c.c.CoordinationState] [node_2409:8086:8110:3f0::2f] cluster UUID set to [_4xxif5aQ_qY2qjy4VzOKw]

[2020-12-23T02:18:45,789][WARN ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2f] failed to validate incoming join request from node [{node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{AYTQ-JPtTGS-WrvVt2fNIA}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]

org.elasticsearch.transport.RemoteTransportException: [node_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _4xxif5aQ_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS_aqA, rejecting

at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

[2020-12-23T02:18:45,819][INFO ][o.e.c.s.ClusterApplierService] [node_2409:8086:8110:3f0::2f] master node changed {previous [], current [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}]}, added {{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim},}, term: 1, version: 1, reason: Publication{term=1, version=1}

[2020-12-23T02:18:45,851][INFO ][o.e.h.AbstractHttpServerTransport] [node_2409:8086:8110:3f0::2f] publish_address {[2409:8086:8110:3f0::2f]:9200}, bound_addresses {[2409:8086:8110:3f0::2f]:9200}

[2020-12-23T02:18:45,853][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] started

[2020-12-23T02:18:45,930][INFO ][o.e.g.GatewayService     ] [node_2409:8086:8110:3f0::2f] recovered [0] indices into cluster_state

[2020-12-23T02:18:46,485][WARN ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2f] failed to validate incoming join request from node [{node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{AYTQ-JPtTGS-WrvVt2fNIA}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]

org.elasticsearch.transport.RemoteTransportException: [node_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _4xxif5aQ_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS_aqA, rejecting

at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

[2020-12-23T02:18:47,514][WARN ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2f] failed to validate incoming join request from node [{node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{AYTQ-JPtTGS-WrvVt2fNIA}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]

org.elasticsearch.transport.RemoteTransportException: [node_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _4xxif5aQ_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS_aqA, rejecting

at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]


node2：
[2020-12-23T02:18:28,219][INFO ][o.e.e.NodeEnvironment    ] [node_2409:8086:8110:3f0::2e] using [1] data paths, mounts [[/opt/oss (/dev/mapper/vg_ext-lv_opt)]], net usable_space [32.6gb], net total_space [39.1gb], types [ext4]

[2020-12-23T02:18:28,220][INFO ][o.e.e.NodeEnvironment    ] [node_2409:8086:8110:3f0::2e] heap size [989.8mb], compressed ordinary object pointers [true]

[2020-12-23T02:18:28,232][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2e] node name [node_2409:8086:8110:3f0::2e], node ID [2H9c-7LTT-eha7ICDCCqcA], cluster name [Eam_ES_Cluster_Product]

[2020-12-23T02:18:28,233][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2e] version[7.3.1], pid[50576], build[oss/tar/4749ba6/2019-08-19T20:19:25.651794Z], OS[Linux/3.10.0-862.14.1.5.h442.eulerosv2r7.x86_64/amd64], JVM[Huawei Technologies Co., Ltd/OpenJDK 64-Bit Server VM/1.8.0_242/25.242-b08]

[2020-12-23T02:18:28,233][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2e] JVM home [/opt/oss/rtsp/jre-2.40.16-euler_amd64]

[2020-12-23T02:18:28,234][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2e] JVM arguments [-Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-1806468132634021252, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Djava.security.policy=/opt/oss/envs/Product-DVEamElasticsearchService/20201223021711464/es/config/security.policy, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/oss/SOP/DVEamElasticsearchService/logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=20m, -Des.insecure.allow.root=false, -Dprocname=dveamelasticsearchservice-2-0, -DNFW=dveamelasticsearchservice-2-0, -DTOMCAT_LOG_DIR=/var/log/oss/SOP/DVEamElasticsearchService/dveamelasticsearchservice-2-0/tomcatlog, -DTOMCAT_WORK_DIR=/var/log/oss/SOP/DVEamElasticsearchService/dveamelasticsearchservice-2-0/tomcatlog, -Djna.tmpdir=/var/log/oss/SOP/DVEamElasticsearchService/tmp, -Dio.netty.allocator.type=unpooled, -Des.path.home=/opt/oss/envs/Product-DVEamElasticsearchService/20201223021711464/es, -Des.path.conf=/opt/oss/envs/Product-DVEamElasticsearchService/20201223021711464/es/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=false]

[2020-12-23T02:18:31,145][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [aggs-matrix-stats]

[2020-12-23T02:18:31,145][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [analysis-common]

[2020-12-23T02:18:31,146][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [ingest-common]

[2020-12-23T02:18:31,146][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [ingest-geoip]

[2020-12-23T02:18:31,153][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [ingest-user-agent]

[2020-12-23T02:18:31,155][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [lang-expression]

[2020-12-23T02:18:31,155][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [lang-mustache]

[2020-12-23T02:18:31,155][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [lang-painless]

[2020-12-23T02:18:31,155][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [mapper-extras]

[2020-12-23T02:18:31,156][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [parent-join]

[2020-12-23T02:18:31,156][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [percolator]

[2020-12-23T02:18:31,156][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [rank-eval]

[2020-12-23T02:18:31,156][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [reindex]

[2020-12-23T02:18:31,157][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [repository-url]

[2020-12-23T02:18:31,157][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded module [transport-netty4]

[2020-12-23T02:18:31,159][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2e] loaded plugin [digitalview-ssl]

[2020-12-23T02:18:37,810][INFO ][o.e.d.DiscoveryModule    ] [node_2409:8086:8110:3f0::2e] using discovery type [zen] and seed hosts providers [settings]

[2020-12-23T02:18:38,422][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2e] initialized

[2020-12-23T02:18:38,423][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2e] starting ...

[2020-12-23T02:18:38,609][INFO ][o.e.t.TransportService   ] [node_2409:8086:8110:3f0::2e] publish_address {[2409:8086:8110:3f0::2e]:9300}, bound_addresses {[2409:8086:8110:3f0::2e]:9300}

[2020-12-23T02:18:38,620][INFO ][o.e.b.BootstrapChecks    ] [node_2409:8086:8110:3f0::2e] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2020-12-23T02:18:40,373][INFO ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2e] setting initial configuration to VotingConfiguration{44C3jmp9RNis66QaYeJj5A,{bootstrap-placeholder}-node_2409:8086:8110:3f0::2f,2H9c-7LTT-eha7ICDCCqcA}

[2020-12-23T02:18:45,786][INFO ][o.e.c.c.CoordinationState] [node_2409:8086:8110:3f0::2e] cluster UUID set to [_4xxif5aQ_qY2qjy4VzOKw]

[2020-12-23T02:18:45,806][INFO ][o.e.c.s.ClusterApplierService] [node_2409:8086:8110:3f0::2e] master node changed {previous [], current [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}]}, added {{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim},}, term: 1, version: 1, reason: ApplyCommitRequest{term=1, version=1, sourceNode={node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}}

[2020-12-23T02:18:45,832][INFO ][o.e.h.AbstractHttpServerTransport] [node_2409:8086:8110:3f0::2e] publish_address {[2409:8086:8110:3f0::2e]:9200}, bound_addresses {[2409:8086:8110:3f0::2e]:9200}

[2020-12-23T02:18:45,832][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2e] started

[2020-12-23T02:36:18,042][INFO ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2e] master node [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}] failed, restarting discovery

org.elasticsearch.transport.NodeDisconnectedException: [node_2409:8086:8110:3f0::2f][[2409:8086:8110:3f0::2f]:9300][disconnected] disconnected

[2020-12-23T02:36:18,050][INFO ][o.e.c.s.ClusterApplierService] [node_2409:8086:8110:3f0::2e] master node changed {previous [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}], current []}, term: 1, version: 57, reason: becoming candidate: onLeaderFailure

[2020-12-23T02:36:18,084][WARN ][o.e.c.NodeConnectionsService] [node_2409:8086:8110:3f0::2e] failed to connect to {node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim} (tried [1] times)

org.elasticsearch.transport.ConnectTransportException: [node_2409:8086:8110:3f0::2f][[2409:8086:8110:3f0::2f]:9300] connect_exception

	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:957) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.3.1.jar:7.3.1]

	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_242]

	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_242]

	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_242]

	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_242]

	at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.3.1.jar:7.3.1]

	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]

	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]

	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]

	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]

	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]

	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 2409:8086:8110:3f0:0:0:0:2f/2409:8086:8110:3f0:0:0:0:2f:9300

Caused by: java.net.ConnectException: Connection refused

	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]

	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714) ~[?:?]

	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]

	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]

	... 7 more

[2020-12-23T02:36:28,050][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}, {node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{AYTQ-JPtTGS-WrvVt2fNIA}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:36:38,051][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}, {node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{AYTQ-JPtTGS-WrvVt2fNIA}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:36:48,053][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}, {node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{AYTQ-JPtTGS-WrvVt2fNIA}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:36:58,054][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:37:08,056][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:37:08,679][WARN ][o.e.c.NodeConnectionsService] [node_2409:8086:8110:3f0::2e] failed to connect to {node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim} (tried [7] times)

org.elasticsearch.transport.ConnectTransportException: [node_2409:8086:8110:3f0::2f][[2409:8086:8110:3f0::2f]:9300] connect_exception

	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:957) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.3.1.jar:7.3.1]

	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_242]

	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_242]

	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_242]

	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_242]

	at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.3.1.jar:7.3.1]

	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]

	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]

	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]

	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]

	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]

	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 2409:8086:8110:3f0:0:0:0:2f/2409:8086:8110:3f0:0:0:0:2f:9300

Caused by: java.net.ConnectException: Connection refused

	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]

	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714) ~[?:?]

	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]

	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]

	... 7 more

[2020-12-23T02:37:18,060][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:37:28,062][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:37:38,063][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:37:48,064][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:37:58,065][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:38:08,067][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:38:09,001][WARN ][o.e.c.NodeConnectionsService] [node_2409:8086:8110:3f0::2e] failed to connect to {node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim} (tried [13] times)

org.elasticsearch.transport.ConnectTransportException: [node_2409:8086:8110:3f0::2f][[2409:8086:8110:3f0::2f]:9300] connect_exception

	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:957) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$3(ActionListener.java:161) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.3.1.jar:7.3.1]

	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_242]

	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_242]

	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_242]

	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_242]

	at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.3.1.jar:7.3.1]

	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:68) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]

	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]

	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]

	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]

	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]

	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]

	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]

	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 2409:8086:8110:3f0:0:0:0:2f/2409:8086:8110:3f0:0:0:0:2f:9300

Caused by: java.net.ConnectException: Connection refused

	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]

	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714) ~[?:?]

	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]

	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]

	... 7 more

[2020-12-23T02:38:18,068][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1

[2020-12-23T02:38:28,069][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node_2409:8086:8110:3f0::2e] master not discovered or elected yet, an election requires 2 nodes with ids [2dFgJYXoTyWJ12es9CuiCQ, 2H9c-7LTT-eha7ICDCCqcA], have discovered [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}, {node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{HdogZVSKQtu2V61zBCdNeQ}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}] which is not a quorum; discovery will continue using [[2409:8086:8110:3f0::2d]:9300, [2409:8086:8110:3f0::2f]:9300] from hosts providers and [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}] from last-known cluster state; node term 1, last-accepted version 57 in term 1
 
[2020-12-23T02:38:57,372][INFO ][o.e.c.s.ClusterApplierService] [node_2409:8086:8110:3f0::2e] master node changed {previous , current [{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim}]}, removed {{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim},}, added {{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{p6QGBla8QkuMi-6-428Zgw}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim},}, term: 19, version: 58, reason: Publication{term=19, version=58}

[2020-12-23T02:38:57,393][INFO ][o.e.c.r.DelayedAllocationService] [node_2409:8086:8110:3f0::2e] scheduling reroute for delayed shards in [59.3s] (1 delayed shards)

[2020-12-23T02:38:57,450][INFO ][o.e.i.s.IndexShard ] [node_2409:8086:8110:3f0::2e] [i2000.mim][0] primary-replica resync completed with 0 operations

[2020-12-23T02:38:57,561][WARN ][o.e.c.c.Coordinator ] [node_2409:8086:8110:3f0::2e] failed to validate incoming join request from node [{node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{HdogZVSKQtu2V61zBCdNeQ}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]

org.elasticsearch.transport.RemoteTransportException: [node_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _4xxif5aQ_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS_aqA, rejecting

at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]



[2020-12-23T02:38:58,547][WARN ][o.e.c.c.Coordinator ] [node_2409:8086:8110:3f0::2e] failed to validate incoming join request from node [{node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{HdogZVSKQtu2V61zBCdNeQ}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]

org.elasticsearch.transport.RemoteTransportException: [node_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _4xxif5aQ_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS_aqA, rejecting

at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]

at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]

node3
[2020-12-23T02:18:36,155][INFO ][o.e.e.NodeEnvironment    ] [node_2409:8086:8110:3f0::2f] using [1] data paths, mounts [[/opt/oss (/dev/mapper/vg_ext-lv_opt)]], net usable_space [33gb], net total_space [39.1gb], types [ext4]

[2020-12-23T02:18:36,158][INFO ][o.e.e.NodeEnvironment    ] [node_2409:8086:8110:3f0::2f] heap size [989.8mb], compressed ordinary object pointers [true]

[2020-12-23T02:18:36,160][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] node name [node_2409:8086:8110:3f0::2f], node ID [2dFgJYXoTyWJ12es9CuiCQ], cluster name [Eam_ES_Cluster_Product]

[2020-12-23T02:18:36,161][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] version[7.3.1], pid[20151], build[oss/tar/4749ba6/2019-08-19T20:19:25.651794Z], OS[Linux/3.10.0-862.14.1.5.h442.eulerosv2r7.x86_64/amd64], JVM[Huawei Technologies Co., Ltd/OpenJDK 64-Bit Server VM/1.8.0_242/25.242-b08]

[2020-12-23T02:18:36,162][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] JVM home [/opt/oss/rtsp/jre-2.40.16-euler_amd64]

[2020-12-23T02:18:36,162][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] JVM arguments [-Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=512m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-6609433216513750091, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Djava.security.policy=/opt/oss/envs/Product-DVEamElasticsearchService/20201223021711464/es/config/security.policy, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/oss/SOP/DVEamElasticsearchService/logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=20m, -Des.insecure.allow.root=false, -Dprocname=dveamelasticsearchservice-3-0, -DNFW=dveamelasticsearchservice-3-0, -DTOMCAT_LOG_DIR=/var/log/oss/SOP/DVEamElasticsearchService/dveamelasticsearchservice-3-0/tomcatlog, -DTOMCAT_WORK_DIR=/var/log/oss/SOP/DVEamElasticsearchService/dveamelasticsearchservice-3-0/tomcatlog, -Djna.tmpdir=/var/log/oss/SOP/DVEamElasticsearchService/tmp, -Dio.netty.allocator.type=unpooled, -Des.path.home=/opt/oss/envs/Product-DVEamElasticsearchService/20201223021711464/es, -Des.path.conf=/opt/oss/envs/Product-DVEamElasticsearchService/20201223021711464/es/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=false]

[2020-12-23T02:18:37,955][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [aggs-matrix-stats]

[2020-12-23T02:18:37,956][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [analysis-common]

[2020-12-23T02:18:37,957][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [ingest-common]

[2020-12-23T02:18:37,957][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [ingest-geoip]

[2020-12-23T02:18:37,957][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [ingest-user-agent]

[2020-12-23T02:18:37,959][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [lang-expression]

[2020-12-23T02:18:37,959][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [lang-mustache]

[2020-12-23T02:18:37,959][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [lang-painless]

[2020-12-23T02:18:37,959][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [mapper-extras]

[2020-12-23T02:18:37,960][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [parent-join]

[2020-12-23T02:18:37,960][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [percolator]

[2020-12-23T02:18:37,960][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [rank-eval]

[2020-12-23T02:18:37,960][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [reindex]

[2020-12-23T02:18:37,961][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [repository-url]

[2020-12-23T02:18:37,961][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded module [transport-netty4]

[2020-12-23T02:18:37,962][INFO ][o.e.p.PluginsService     ] [node_2409:8086:8110:3f0::2f] loaded plugin [digitalview-ssl]

[2020-12-23T02:18:42,785][INFO ][o.e.d.DiscoveryModule    ] [node_2409:8086:8110:3f0::2f] using discovery type [zen] and seed hosts providers [settings]

[2020-12-23T02:18:43,350][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] initialized

[2020-12-23T02:18:43,350][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] starting ...

[2020-12-23T02:18:43,661][INFO ][o.e.t.TransportService   ] [node_2409:8086:8110:3f0::2f] publish_address {[2409:8086:8110:3f0::2f]:9300}, bound_addresses {[2409:8086:8110:3f0::2f]:9300}

[2020-12-23T02:18:43,678][INFO ][o.e.b.BootstrapChecks    ] [node_2409:8086:8110:3f0::2f] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2020-12-23T02:18:45,245][INFO ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2f] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-node_2409:8086:8110:3f0::2d,2dFgJYXoTyWJ12es9CuiCQ,2H9c-7LTT-eha7ICDCCqcA}

[2020-12-23T02:18:45,667][INFO ][o.e.c.s.MasterService    ] [node_2409:8086:8110:3f0::2f] elected-as-master ([2] nodes joined)[{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim} elect leader, {node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, reason: master node changed {previous [], current [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}]}, added {{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim},}

[2020-12-23T02:18:45,783][INFO ][o.e.c.c.CoordinationState] [node_2409:8086:8110:3f0::2f] cluster UUID set to [_4xxif5aQ_qY2qjy4VzOKw]

[2020-12-23T02:18:45,789][WARN ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2f] failed to validate incoming join request from node [{node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{AYTQ-JPtTGS-WrvVt2fNIA}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]

org.elasticsearch.transport.RemoteTransportException: [node_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _4xxif5aQ_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS_aqA, rejecting

	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]

	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

[2020-12-23T02:18:45,819][INFO ][o.e.c.s.ClusterApplierService] [node_2409:8086:8110:3f0::2f] master node changed {previous [], current [{node_2409:8086:8110:3f0::2f}{2dFgJYXoTyWJ12es9CuiCQ}{tAO43Q-sRjGypNdmlUcNQQ}{2409:8086:8110:3f0:0:0:0:2f}{[2409:8086:8110:3f0::2f]:9300}{dim}]}, added {{node_2409:8086:8110:3f0::2e}{2H9c-7LTT-eha7ICDCCqcA}{BlX_bI3bSx2uJpouoyjf2A}{2409:8086:8110:3f0:0:0:0:2e}{[2409:8086:8110:3f0::2e]:9300}{dim},}, term: 1, version: 1, reason: Publication{term=1, version=1}

[2020-12-23T02:18:45,851][INFO ][o.e.h.AbstractHttpServerTransport] [node_2409:8086:8110:3f0::2f] publish_address {[2409:8086:8110:3f0::2f]:9200}, bound_addresses {[2409:8086:8110:3f0::2f]:9200}

[2020-12-23T02:18:45,853][INFO ][o.e.n.Node               ] [node_2409:8086:8110:3f0::2f] started

[2020-12-23T02:18:45,930][INFO ][o.e.g.GatewayService     ] [node_2409:8086:8110:3f0::2f] recovered [0] indices into cluster_state

[2020-12-23T02:18:46,485][WARN ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2f] failed to validate incoming join request from node [{node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{AYTQ-JPtTGS-WrvVt2fNIA}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]

org.elasticsearch.transport.RemoteTransportException: [node_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _4xxif5aQ_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS_aqA, rejecting

	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]

	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

[2020-12-23T02:18:47,514][WARN ][o.e.c.c.Coordinator      ] [node_2409:8086:8110:3f0::2f] failed to validate incoming join request from node [{node_2409:8086:8110:3f0::2d}{44C3jmp9RNis66QaYeJj5A}{AYTQ-JPtTGS-WrvVt2fNIA}{2409:8086:8110:3f0:0:0:0:2d}{[2409:8086:8110:3f0::2d]:9300}{dim}]

org.elasticsearch.transport.RemoteTransportException: [node_2409:8086:8110:3f0::2d][[2409:8086:8110:3f0::2d]:9300][internal:cluster/coordination/join/validate]

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid _4xxif5aQ_qY2qjy4VzOKw than local cluster uuid x2h6qG3gQG-FAgdHGS_aqA, rejecting

	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:148) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.1.jar:7.3.1]

	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.1.jar:7.3.1]

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]

	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

 
2020-12-28 添加评论

没有找到相关结果

    已邀请:

    shuhaijun

    赞同来自:

    之前有遇到过操作系统不一致会导致节点的uuid无法识别
    2020-12-30 0 1
    分享

    pony_maggie - 公众号:犀牛饲养员的技术笔记

    赞同来自:

    uuid是集群的唯一标识。节点之间 cluster state不一致就可能出现，比如把一个节点从A集群移动到B集群，或者不小心误删了一个节点的cluster state。
     
    你这个问题可以考虑复位cluster uuid，看下elasticsearch-node detach-cluster命令
    2020-12-30 0 0
    分享
    为什么被折叠? 0 个回复被折叠

    要回复问题请先登录或注册

    Copyright © 2024 · CC BY-NC-SA 3.0

    本站服务器及带宽由 提供赞助