我有点怀疑你在刷屏

指定配置文件启动Logstash失败

Logstash | 作者 Shinefire | 发布于2019年01月23日 | 阅读数:41649

环境:
下载的elasticsearch、Logstash、filebeat、kibana 均为最新的6.5.4版本
内网IP地址:10.0.0.179
 
问题描述:
参考这篇博客(https://www.linuxidc.com/Linux/2017-11/148222.htm)对ELK进行单机部署的过程中,在指定配置文件启动Logstash这一步操作的时候遇到了一个问题。
按照博主的方法,先和他一样配置好了filebeat.yml文件,配置文件如下:
# cat /usr/local/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /var/log/message-log
output.logstash:
hosts: ["10.0.0.179:5044"]

随后启动了filebeat进程,查看进程如下:
# ps axu | grep filebeat 
root 16923 0.3 0.1 239160 15536 pts/6 Sl+ 16:39 0:00 ./filebeat
root 16937 0.0 0.0 112708 980 pts/0 S+ 16:39 0:00 grep --color=auto filebeat

然后按博主的做法新建了一个message-log文件,内容是直接从默认的messages文件里面复制了一点内容,内容如下:
# cat /var/log/message-log 
Jan 22 20:54:16 apt systemd-logind: New session 231 of user root.
Jan 22 21:00:01 apt systemd: Started Session 232 of user root.
Jan 22 21:01:01 apt systemd: Started Session 233 of user root.
Jan 22 21:10:01 apt systemd: Started Session 234 of user root.
Jan 22 21:10:37 apt systemd: Started Session 235 of user root.
Jan 22 21:10:37 apt systemd-logind: New session 235 of user root.
Jan 22 21:13:33 apt systemd: Started Session 236 of user root.
Jan 22 21:13:33 apt systemd-logind: New session 236 of user root.

然后编辑一个启动Logstash需要指定的配置文件,也是基本参照博主的操作,配置文件内容如下:
# cat /usr/local/logstash/config/test.conf 
input {
beats {
port => "5044"
}
}
output {
elasticsearch {
hosts => "10.0.0.179:9200"
}
stdout { codec => rubydebug } # 这是将输出打印在屏幕上,可以注释掉
} 

随后进行指定配置文件启动Logstash的操作,启动后的结果如下:
[root@apt bin]# ./logstash -f ../config/test.conf 
Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2019-01-23T16:42:53,991][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-23T16:42:54,012][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-23T16:42:55,655][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 11, column 2 (byte 166) after ", :backtrace=>["/usr/local/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/local/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/local/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/local/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in `initialize'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:42:in `block in execute'", "/usr/local/logstash/logstash-core/lib/logstash/agent.rb:92:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/usr/local/logstash/logstash-core/lib/logstash/agent.rb:92:in `exclusive'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", "/usr/local/logstash/logstash-core/lib/logstash/agent.rb:317:in `block in converge_state'"]}
[2019-01-23T16:42:56,167][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[root@apt bin]#

 指定文件启动Logstash后稍等了一会儿,弹出上面的代码框内提示后就直接结束了。
 
并没有和博客里面说的那样,会开启5044端口和9600端口,9600在结束这个启动过程的时候,就会关闭了,5044端口更是没有看到出现过
 
 
另外也没有出现博客里面所说的等待一段时间后会输出的信息,毕竟我等待一段时间后直接就提示successful结束了,下图为博客所说的信息输出的截图:
123.png

 
 
--------------------------------------
第二次修改:2019年1月23日18:56:26
 
后来仔细看,发现在执行 ` ./logstash -f ../config/test.conf ` 这个命令的时候下面其实是有报错的,不过一开始没有注意到,报错内容可以直接看看上面的代码框粘贴内容
 
后来使用“ ./logstash -f ../config/test.conf -t ”检查了一下配置文件,发现有报错如下:
# ./logstash -f ../config/test.conf -t 
Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2019-01-23T17:34:00,487][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-23T17:34:01,182][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 11, column 2 (byte 155) after
[2019-01-23T17:34:01,202][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

不过我也不懂该如何修改,目前还根本没有搞明白... 关键是看了好几篇博客都是一模一样的,已经不知道到底是同一个作者在不同的地方发表了还是不同的人都在转发那一篇博客,总之我是没法按这样说的操作,于是换了个博客参考,另外写了个conf文件,使用-t检查是没有error了,但是发现直接指定文件启动的话还是一样会报错,有点小绝望... 
 

----------------------------------------
第三次修改:2019年1月28日16:36:13
 
filebeat配置文件如下:
# cat /usr/local/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /var/log/message-log
tags: ["test"]
type: test

output.logstash:
hosts: ["127.0.0.1:5044"]
index: "test-filebeat"

按照评论替我解答的大佬们说的手动又敲了一遍启动Logstash需要指定的配置文件:
# vim test2.conf
input {
beats {
port => "5044"
}
}

output {
elasticsearch {
hosts => "127.0.0.1:9200"
}
stdount { codec => rubydebug }
}

 
在启动了filebeat之后,使用 -t 参数测试了一下Logstash,报错内容如下:
# /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/test2.conf  -t 
Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2019-01-28T16:32:41,194][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-28T16:32:43,242][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/outputs/stdount>, :path=>"logstash/outputs/stdount", :type=>"output", :name=>"stdount"}
[2019-01-28T16:32:43,266][FATAL][logstash.runner ] The given configuration is invalid. Reason: Couldn't find any output plugin named 'stdount'. Are you sure this is correct? Trying to load the stdount output plugin resulted in this error: no such file to load -- logstash/outputs/stdount
[2019-01-28T16:32:43,283][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

我晚点再查一下这个报错吧!之前因为这个配置文件一直不行,就另外参考了其他的博客,算是跳过了这个问题,成功的搭建起来了。

---------------------------------------- 
参照博客都检查了一下,实在是没有发现什么问题,也没有什么排错的思路... 还请各位大佬有时间的话帮忙看看,指点迷津
 
 
 
已邀请:

lianjie

赞同来自:

缩进不一致?

Ezreal

赞同来自:

手敲一遍

rochy - rochy_he

赞同来自:

把第11行的注释去掉试试

zqc0512 - andy zhou

赞同来自:

"Expected one of #, input, filter, output at line 11, column 2 (byte 166) after 
配置文件有问题。你到处COPY的吧。这玩意YARN 配置需要很严格的,你搞成UTF-8格式看看吧,修改成linux文件DOC。

sweetpotato - 90IT男

赞同来自:

一般我都是手敲一边的

zqc0512 - andy zhou

赞同来自:

看看tab和 退格内容,2个空格 有可能是这玩意问题。
 

henry0417 - 80后IT男

赞同来自:

# vim test2.conf 
这里面的 stdount { c。。。。。。是什么鬼?
 
不是stdout 吗?
 
配置文件不存在什么缩进问题,每一次的错误提示都给的很明显,只是楼主没注意看清楚而已。

yikexiaohuya

赞同来自:

我也遇到了,已经被卡一周了,卡楼主怎么解决的,求?

Karl_xy

赞同来自:

用config.test and exit测试一下

Adrian_dan - 95,IT

赞同来自:

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 3, column 10 (byte 28) after input {\r\n  tcp {\r\n    port ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:309:in `block in converge_state'"]}
 
 
请问一下楼主,解决了吗,同样的问题

要回复问题请先登录注册