Questions tagged [logstash]
logstash is a tool for collecting and distributing log events.
261
questions
52
votes
5
answers
204k
views
Failed tls handshake. Does not contain any IP SANs
I'm trying to set up logstash forwarder, but I have issues with making a proper secure channel. Trying to configure this with two ubuntu (server 14.04) machines running in virtualbox. They are 100% ...
- 555
26
votes
2
answers
17k
views
How to kill a process that never dies?
Problem
I have java process which does not die neither with SIGTERM nor SIGKILL.
logstash 2591 1 99 13:22 ? 00:01:46 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:...
- 616
16
votes
1
answer
14k
views
Scaling Logstash (with redis/elasticsearch)
Over a cluster of over 12 centos 5.8 servers, I deployed logstash using the native logstash shipper, which sends /var/log/*/*.log back to a central logstash server.
We tried using rsyslogd as the ...
- 27.5k
15
votes
10
answers
64k
views
Get logstash version
How does one get the version of Logstash?
root@elk:/usr/share/elasticsearch# bin/logstash --help
bash: bin/logstash: No such file or directory
I have Logstash running on my system. Also.
root@elk:/#...
- 1,661
12
votes
1
answer
12k
views
logstash (or graylog?) vs nxLog to collect event logs and csv logs [closed]
I am current investigating the possibility to consolidate logs from multiple servers using logstash (or graylog2).
I am still a bit confused about the difference logstash and graylog. So far I ...
- 293
12
votes
2
answers
31k
views
Configuring Logstash when installed as a service [closed]
I have installed logstash as a service using the logstash APT repository on Ubuntu 13.10.
So now I can run:
dpkg -s logstash
And it outputs:
Package: logstash
Status: install ok installed
Priority:...
- 1,018
10
votes
2
answers
15k
views
Logstash can't read files it should have access too
I've added user logstash into group adm using the command $ usermod -a -G adm logstash.
One of the files that the logstash agent is trying to read is /var/log/nginx/foo-access.log, which has the ...
- 291
9
votes
2
answers
21k
views
Logstash parsing xml document containing multiple log entries
I'm currently evaluating whether logstash and elasticsearch are useful for our use-case. What I have is a log file containing multiple entries which is of the form
<root>
<entry>
...
- 408
9
votes
1
answer
489
views
Elasticsearch dies when Logstash attempts to write data
I've got a Raspberry Pi 2 (latest Raspbian as of Apr 2015) setup that last week was running both ElasticSearch and Logstash on a test network (not a straightforward setup, but it was stable for over a ...
- 193
8
votes
1
answer
11k
views
What is the significance of the @ prefix in logstash field names?
The following logstash configuration is used to accept Windows Event Logs as json over a TCP connection and then after some filtering forward the result to Elastic search (source: https://gist.github....
- 7,917
8
votes
2
answers
1k
views
How to configure a log aggregator to authenticate data?
Background: Remote log aggregation is regarded as a way to improve security. Generally, this addresses the risk that an attacker who compromises a system can edit or delete logs to frustrate forensic ...
- 183
8
votes
4
answers
5k
views
How best to monitor logstash?
I've seen this question on the mailing list a few times but haven't had a satisfactory answer.
How best to monitor that the pipeline isn't stuck? Clients -> logstash -> elasticsearch.
Logstash and ...
- 3,002
8
votes
1
answer
1k
views
ELK Stack (Logstash, Elasticsearch and Kibana) with concurrent remote syslog server?
I'm building a log analyser service to start monitoring mainly our pfSense Firewalls, XenServer Hypervisors, FreeBSD/Linux servers and Windows servers.
There's a lot of documentation on the internet ...
- 5,700
7
votes
4
answers
12k
views
nginx error log Grok pattern
I am having trouble getting the following nginx error log message to parse in the grok debugger. I have a feeling there is a stupid trick that I should use but can't figure out what it may be.
...
- 790
7
votes
3
answers
7k
views
Secure logstash and elasticsearch
I'm considering running logstash on my prod server (simple install. http://logstash.net/docs/1.1.13/tutorials/getting-started-simple) and set kibana to access logs.
My concern is: how to secure my ...
- 193
6
votes
2
answers
891
views
logstash alert after 1000 occurences
I am trying to make Logstash to alert me only after it receives over 1000 items within 10 minutes.
I need alerts in both Hipchat and PagerDuty.
My config seems reasonable, but does not work as ...
- 63
6
votes
4
answers
11k
views
How to send Windows Performance counters to Logstash + Kibana?
I would like to setup monitoring for system resources on my Windows servers. I've noticed a common configuration in Linux is to use collectd daemon to get system metrics information. From collectd ...
- 355
6
votes
3
answers
4k
views
Logstash with journald instead of rsyslog
I'm used to sending my logs from a server to a remote Logstash using rsyslog, with a configuration file roughly as follows (usually more specific to prevent too many logs from being sent):
*.* @192....
- 163
6
votes
2
answers
3k
views
How does an administrator generalize alerting when an event doesn't happen?
Often my users require me to be just as responsible for knowing if an event hasn't happened.
I've always had to build custom and brittle solutions with cron'ed shell scripts and lots of date edge ...
- 3,002
6
votes
2
answers
1k
views
Sending Subversion logs to Logstash
My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own ...
- 713
5
votes
6
answers
10k
views
Logstash binding to a port already in use
This is the output when I try and run logstash. With Redis and ElasticSearch disabled it still says address already in use. Any suggestions? As far as I can tell this was fixed in 1.1.8 but I seem to ...
- 171
5
votes
2
answers
18k
views
Centos: yum install libevent-devel conflict with compat-libevent
I want to install Logstash and others, but when I try:
# yum install libevent-devel
I get the errors below:
Transaction Check Error:
file /usr/bin/event_rpcgen.py from install of libevent-devel-...
- 51
5
votes
3
answers
10k
views
Logstash / Elasticsearch - trasform fields to lowercase in output conf
I have a standard ELK stack currently storing numerous log outputs. I'm trying to separate my indices to be source-specific.
As part of my FileBeats config, some standard fields are always generated ...
- 51
5
votes
4
answers
50k
views
How to see if filebeat data is being sent to logstash
When I open up Kibana interface, I get an error to configure index when logstash-* is entered as a query:
kibana error: please specify a default index pattern
How can I see if filebeat is sending ...
- 171
5
votes
2
answers
4k
views
How to forward application logs from Docker containers to ELK
I'm trying to centralise logging in an environment that using multiple application technologies (Java, Rails and various DBs).
We want to developers to bring up stacks with Docker Compose, but we ...
- 3,539
5
votes
2
answers
5k
views
"Index Patterns: Please specify a default index pattern" in Kibana
I'm trying to create a simple hello world for ELK and be able to see kibana reports via the internet. I've installed kibana, logstash, nginx and elastic search. Here's my /etc/logstash/conf.d/10-...
- 53
4
votes
4
answers
18k
views
How to parse audit.log using logstash
I want to use logstash to collect a log file, and the format of the file was like this:
type=USER_START msg=audit(1404170401.294:157): user pid=29228 uid=0 auid=0 ses=7972 subj=system_u:system_r:...
- 479
4
votes
4
answers
32k
views
Nothing appearing in kibana dashboard
I have installed
Logstash
ElasticSearch
Kibana
on an EC2 instance.
I can visit http://example.com:9200 Here I get
{
"status" : 200,
"name" : "Aleta Ogord",
"version" : {
"number" : "1.1.0"...
- 153
4
votes
3
answers
18k
views
logstash failing to parse syslog input
I've configured logstash (v1.5.0), with a simple syslog input, as follows:
input {
syslog {
type => syslog
port => 5514
}
}
filter {
kv {}
}
output {
elasticsearch {
...
- 449
4
votes
1
answer
21k
views
Logstash, Kibana and email alerts
I am trying to setup email alerts using logstash. Right now it emails me EVERY time the pattern "Error" is parsed into my log file which can lead to a lot of unnecessary emails.
I'd like to create a ...
- 141
4
votes
1
answer
3k
views
Adding TTL to elasticsearch with logstash
I can't figure out how to apply this documentation http://www.elasticsearch.org/guide/reference/mapping/ttl-field/ to logstash. Specifically, i don't know which keys to use (instead of "tweet" etc).
...
- 291
4
votes
3
answers
9k
views
logstash: is there a way to view the complete running config when using multiple config files in /etc/logstash/conf.d?
I am running logstash 1.5.0.1 and I have multiple config files in my /etc/logstash/conf.d folder:
01-input-source-one.conf
02-input-source-two.conf
10-filter-one.conf
11-filter-two.conf
20-...
- 973
4
votes
1
answer
2k
views
Logstash/elasticsearch stops accepting new data
I've set up a new proof of concept logstash system
CentOS 6.6 (on Vmware 5.5) - single CPU VM with 12G RAM allocated
Elasticsearch and Logstash installed from RPMs …
# rpm -q elasticsearch logstash
...
- 4,557
4
votes
4
answers
3k
views
CentOS centralised logging, syslogd, rsyslog, syslog-ng, logstash sender?
I'm trying to figure out the best way to setup a central place to store and interrogate server logs. syslog, Apache, MySQL etc.
I've found a few different options but I'm not sure what would be best. ...
- 466
4
votes
1
answer
786
views
Transparent Proxy to Docker Network Means TCP is Broken
My logging setup is a single Docker host with UDP 514 exposed for syslog. An nginx container has its port published so when you send logs to 10.1.1.100 (in the image below) it first hits nginx, whose ...
- 420
4
votes
1
answer
216
views
logstash-forward equivalent for fluentd?
Is there something equivalent to logstash-forwarder that can ship logfiles to fluentd?
I am trying to send log files from an application to a remote fluentd but have not seen whether this is ...
- 6,965
4
votes
2
answers
3k
views
Postfix - searching emails (logstash, greylog or other solution)
We are currently having ~100 servers and all of them are using remote syslog, so we have aggregated all logs on one server.
The most questioned problem from our support team is: Has an email from .......
- 1,583
4
votes
1
answer
877
views
Logstash Date Has the Wrong Year?
I'm parsing Nginx logs into logstash with the following config:
input { stdin { type => "nginx"}}
filter {
grok {
type => nginx
pattern => "%{COMBINEDAPACHELOG}"
}
...
- 185
4
votes
2
answers
3k
views
Elasticsearch performance tuning
In a Single Node Elastic Search along with logstash, We tested with 20mb and 200mb file parsing to Elastic Search on Different types of the AWS instance i.e Medium, Large and Xlarge.
Logstash conf
...
- 41
4
votes
3
answers
10k
views
Foward slash in kibana 3 query
I'm trying to add a query that will match a request that ends with a slash, like this one:
n.n.n.n - - [16/Oct/2013:16:40:41 +0100] "GET / HTTP/1.1" 200 25058 "-" "Mozilla/5.0 (iPad; CPU OS 7_0_2 ...
- 173
3
votes
1
answer
11k
views
Logstash-forwarder is throwing SSL errors
I got this task handed over to by my colleage and this is the background.
He got ELK (Elasticsearch, Logstash and Kibana) stack working with our RHEL 6.2 servers, by using the regular method of ...
- 464
3
votes
1
answer
10k
views
Logstash S3 input plugin re-scanning all bucket objects
I am using the Logstash S3 Input plugin to process S3 access logs.
The access logs are all stored in a single bucket, and there are thousands of them. I have set up the plugin to only include S3 ...
- 3,539
3
votes
1
answer
3k
views
Logstash integration with AWS Elasticsearch Service
I am using AWS Elasticsearch service to configure Elasticsearch Cluster and there is a separate server where I have installed Logstash 2.1.0
Here is my Logstash sample configuration file :-
input {
...
3
votes
1
answer
4k
views
Why is this exclude_lines in filebeat excluding all logs?
I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-...
- 1,136
3
votes
1
answer
4k
views
Logstash output-http plugin error 500 when trying to send to slack webhook
I'm running a docker container with an ELK stack. Everythings working dandy and fine. I'd like to utilize the https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html plugin.
I ...
- 1,661
3
votes
2
answers
7k
views
getting logs from systemd unit into flat files and logstash
I'm running an application under systemd within CentOS 7. It logs to stdout and systemd is capturing that into journalctl just fine. I'd like to also:
get a rotated text log file also saved to the ...
- 283
3
votes
2
answers
3k
views
Logstash Forwarder doesn't start up with chkconfig in CentOS 5
I have set up logstash-forwarder on a CentOS 5 machine installing it from this RPM:
http://download.elasticsearch.org/logstash-forwarder/packages/logstash-forwarder-0.3.1-1.x86_64.rpm
When I went ...
- 1,014
3
votes
1
answer
3k
views
Failing forwarding rsyslog
I have a centralised rsyslog server A that receives a bunch of logs through TCP from servers X, Y, Z. It then stores the files on disk but also forwards them to logstash server B (on a different ...
- 375
3
votes
1
answer
12k
views
elk stack error "unable to fetch mapping do you have indices matching the pattern"
i am trying to setup ELK stack with collectd on Ubuntu 16.04 LTS (so pretty much latest version of stack available)
kibana is behind nginx proxy (followed this guide https://www.digitalocean.com/...
- 523
3
votes
1
answer
750
views
logstash timestamp on year rollover
We use logstash to store/search logs from our mail servers. I noticed today that we didn't have any indices from this year (2015). Quick investigation showed that current logs were being stored as ...
- 4,557