运行1-2天就挂了,先报如下错误,几分钟后就挂了
[2018-05-15T20:52:59,712][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 5,867 milliseconds ago. The last packet sent successfully to the server was 5,867 milliseconds ago."}
[2018-05-15T20:53:09,060][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 22,365 milliseconds ago. The last packet sent successfully to the server was 9,451 milliseconds ago."}
[2018-05-15T20:53:09,060][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 3,647 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago."}
[2018-05-15T20:52:59,713][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 19,578 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago."}
[2018-05-15T20:53:47,835][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 27,170 milliseconds ago. The last packet sent successfully to the server was 4,982 milliseconds ago."}
数据库连接不上十分钟后,接着就是堆内存溢出:
[2018-05-16T16:34:42,975][WARN ][logstash.inputs.jdbc ] Exception when executing JDBC query {:exception=>#<Sequel::DatabaseError: Java::JavaLang::OutOfMemoryError: Java heap space>}
[2018-05-16T16:34:43,073][ERROR][org.logstash.Logstash ] java.lang.OutOfMemoryError: Java heap space
logstash配置如下;
input {
stdin {
}
jdbc {
jdbc_connection_string => "jdbc:mysql://*.*.*.*:3306/TEST"
jdbc_user => "*******************"
jdbc_password => "*******************"
jdbc_driver_library => "/data/logstash/lib/mysql-connector-java-5.1.22.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
# 使用前验证连接是否有效
jdbc_validate_connection => true
# 多久进行连接有效验证(4小时)
jdbc_validation_timeout => 14400
# 连接失败后最大重试次数
connection_retry_attempts => 50
# 连接失败后重试时间间隔
connection_retry_attempts_wait_time => 1
#使用其他字段追踪,而不是用时间
use_column_value => true
#追踪的字段
tracking_column => FUPDATE_TIME
record_last_run => true
jdbc_default_timezone =>"Asia/Shanghai"
#上一个sql_last_value值的存放文件路径, 必须要在文件中指定字段的初始值
last_run_metadata_path => "/data/logstash/config/work/time/station_reply.txt"
#是否将 column 名称转小写
lowercase_column_names => false
jdbc_page_size => "10000"
statement_filepath => "/data/logstash/config/work/sql/reply.sql"
schedule => "* * * * *"
type => "reply"
}
...(一共16个这样的jdbc,因为有16张分表)
}
求指导,感谢
[2018-05-15T20:52:59,712][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 5,867 milliseconds ago. The last packet sent successfully to the server was 5,867 milliseconds ago."}
[2018-05-15T20:53:09,060][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 22,365 milliseconds ago. The last packet sent successfully to the server was 9,451 milliseconds ago."}
[2018-05-15T20:53:09,060][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 3,647 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago."}
[2018-05-15T20:52:59,713][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 19,578 milliseconds ago. The last packet sent successfully to the server was 0 milliseconds ago."}
[2018-05-15T20:53:47,835][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 27,170 milliseconds ago. The last packet sent successfully to the server was 4,982 milliseconds ago."}
数据库连接不上十分钟后,接着就是堆内存溢出:
[2018-05-16T16:34:42,975][WARN ][logstash.inputs.jdbc ] Exception when executing JDBC query {:exception=>#<Sequel::DatabaseError: Java::JavaLang::OutOfMemoryError: Java heap space>}
[2018-05-16T16:34:43,073][ERROR][org.logstash.Logstash ] java.lang.OutOfMemoryError: Java heap space
logstash配置如下;
input {
stdin {
}
jdbc {
jdbc_connection_string => "jdbc:mysql://*.*.*.*:3306/TEST"
jdbc_user => "*******************"
jdbc_password => "*******************"
jdbc_driver_library => "/data/logstash/lib/mysql-connector-java-5.1.22.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
# 使用前验证连接是否有效
jdbc_validate_connection => true
# 多久进行连接有效验证(4小时)
jdbc_validation_timeout => 14400
# 连接失败后最大重试次数
connection_retry_attempts => 50
# 连接失败后重试时间间隔
connection_retry_attempts_wait_time => 1
#使用其他字段追踪,而不是用时间
use_column_value => true
#追踪的字段
tracking_column => FUPDATE_TIME
record_last_run => true
jdbc_default_timezone =>"Asia/Shanghai"
#上一个sql_last_value值的存放文件路径, 必须要在文件中指定字段的初始值
last_run_metadata_path => "/data/logstash/config/work/time/station_reply.txt"
#是否将 column 名称转小写
lowercase_column_names => false
jdbc_page_size => "10000"
statement_filepath => "/data/logstash/config/work/sql/reply.sql"
schedule => "* * * * *"
type => "reply"
}
...(一共16个这样的jdbc,因为有16张分表)
}
求指导,感谢
2 个回复
rockybean - Elastic Certified Engineer, ElasticStack Fans,公众号:ElasticTalk
赞同来自:
hexiaohong
赞同来自:
原因是:logstash6.2.2对应的logstash-input-jdbc插件版本是4.3.3,从插件官方更新日志来看4.3.4还修复了内存泄露问题:
[root@localhost bin]# ./logstash-plugin update logstash-input-jdbc
Updating logstash-input-jdbc
Updated logstash-input-jdbc 4.3.3 to 4.3.9
[root@localhost bin]#
总结一下:其实我的这个最大的问题在于内存泄露,因为随着运行,占用内存越来越大,不释放,最后堆内存溢出,导致logstash挂了或者管道关闭。