搞了一个冷节点存储空间有200T，一直报错 Too many open files

存储的index数据达到100T左右，log日志一直报错：
Caused by: java.nio.file.FileSystemException: /data02/elasticsearch/indices/1prUSj-vTza4mGfW9UOLiQ/_state: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:440) ~[?:?]
at java.nio.file.Files.newDirectoryStream(Files.java:550) ~[?:?]
at org.elasticsearch.gateway.MetadataStateFormat.findMaxGenerationId(MetadataStateFormat.java:362) ~[elasticsearch-8.1.2.jar:8.1.2]
at org.elasticsearch.gateway.MetadataStateFormat.write(MetadataStateFormat.java:222) ~[elasticsearch-8.1.2.jar:8.1.2]
... 8 more
[2024-03-23T20:48:26,720][ERROR][o.e.g.G.AsyncPersistedState] [dghpl3525-data-cold] Exception occurred when storing new meta data
java.lang.IllegalStateException: Failed to close the XContentBuilder
at org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1233) ~[elasticsearch-x-content-8.1.2.jar:8.1.2]
at org.elasticsearch.gateway.PersistedClusterStateService$Writer.writePages(PersistedClusterStateService.java:965) ~[elasticsearch-8.1.2.jar:8.1.2]
at org.elasticsearch.gateway.PersistedClusterStateService$Writer.addIndexMetadataDocuments(PersistedClusterStateService.java:929) ~[elasticsearch-8.1.2.jar:8.1.2]
at org.elasticsearch.gateway.PersistedClusterStateService$Writer.updateMetadata(PersistedClusterStateService.java:897) ~[elasticsearch-8.1.2.jar:8.1.2]
at org.elasticsearch.gateway.PersistedClusterStateService$Writer.writeIncrementalStateAndCommit(PersistedClusterStateService.java:823) ~[elasticsearch-8.1.2.jar:8.1.2]
at org.elasticsearch.gateway.GatewayMetaState$LucenePersistedState.setLastAcceptedState(GatewayMetaState.java:507) ~[elasticsearch-8.1.2.jar:8.1.2]
at org.elasticsearch.gateway.GatewayMetaState$AsyncPersistedState$1.doRun(GatewayMetaState.java:384) [elasticsearch-8.1.2.jar:8.1.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) [elasticsearch-8.1.2.jar:8.1.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.1.2.jar:8.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]



查看系统的ulimit -a 也没发现异常
[root@centosos ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3092867
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 655360
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 655360
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited