使用 shuf 来打乱一个文件中的行或是选择文件中一个随机的行。

logstash input设置为kafaka时启动报错

Logstash | 作者 luoyongjie87 | 发布于2018年10月15日 | 阅读数:3936

input {
    kafka {
 topic_id => "test"
    bootstrap_servers => "localhost:9092"
    batch_size => 1
    codec => plain{
      format => "%{message}"
    }
        }
}
 
 Sending Logstash logs to D:/work/software/logstash-6.4.2/logs which is now configured via log4j2.properties
[2018-10-14T11:53:46,879][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-10-14T11:53:47,819][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2018-10-14T11:53:50,370][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"input", :name=>"kafka", :path=>"logstash/inputs/kafka", :error_message=>"\n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- /app/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25 (LoadError)", :error_class=>RuntimeError, :error_backtrace=>["D:/work/software/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/jar-dependencies-0.3.12/lib/jar_dependencies.rb:344:in `do_require'", "D:/work/software/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/jar-dependencies-0.3.12/lib/jar_dependencies.rb:253:in `block in require_jar'", "D:/work/software/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/jar-dependencies-0.3.12/lib/jar_dependencies.rb:295:in `require_jar_with_block'", "D:/work/software/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/jar-dependencies-0.3.12/lib/jar_dependencies.rb:252:in `require_jar'", "D:/work/software/logstash-6.4.2/lib/bootstrap/patches/jar_dependencies.rb:6:in `require_jar'", "D:/work/software/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.1.1/lib/logstash-input-kafka_jars.rb:5:in `<main>'", "org/jruby/RubyKernel.java:955:in `require'", "D:/work/software/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'", "D:/work/software/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.1.1/lib/logstash/inputs/kafka.rb:1:in `<main>'", "org/jruby/RubyKernel.java:955:in `require'", "D:/work/software/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/plugins/registry.rb:181:in `legacy_lookup'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/plugins/registry.rb:156:in `block in lookup'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/plugins/registry.rb:152:in `lookup'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/plugins/registry.rb:206:in `lookup_pipeline_plugin'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/plugin.rb:137:in `lookup'", "org/logstash/plugins/PluginFactoryExt.java:210:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:166:in `plugin'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:71:in `plugin'", "D:/work/software/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.1.1/lib/logstash/inputs/kafka.rb:5:in `<eval>'", "org/jruby/RubyKernel.java:994:in `eval'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/agent.rb:309:in `block in converge_state'"]}
[2018-10-14T11:53:50,400][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any input plugin named 'kafka'. Are you sure this is correct? Trying to load the kafka input plugin resulted in this error: Problems loading the requested plugin named kafka of type input. Error: RuntimeError \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- /app/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25 (LoadError)", :backtrace=>["D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/plugins/registry.rb:211:in `lookup_pipeline_plugin'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/plugin.rb:137:in `lookup'", "org/logstash/plugins/PluginFactoryExt.java:210:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:166:in `plugin'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:71:in `plugin'", "(eval):8:in `<eval>'", "org/jruby/RubyKernel.java:994:in `eval'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", "D:/work/software/logstash-6.4.2/logstash-core/lib/logstash/agent.rb:309:in `block in converge_state'"]}
[2018-10-14T11:53:50,870][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

C:\>
已邀请:

rochy - rochy_he

赞同来自:

报错原因貌似是没有找到 logstash-input-kafka 这个 jar
你可以检查一下你的 logstash 有没有安装这个插件
如果没有安装后重新尝试运行
 
此外配置项 topic_id 已经变成 topics 了

luoyongjie87

赞同来自:

出现问题前已经用logstash命令安装了logstash-input-kafka,也提示安装成功了。

要回复问题请先登录注册