Questions tagged [elk]
a.k.a. Elasticsearch, Logstash, Kibana ; refer to elastic.co
60
questions
8
votes
1
answer
9k
views
How to forward Docker container logs to ELK?
I would like to know what is the easiest way to forward my docker container logs to an ELK server, so far the solutions I have tried after having searched the internet didn't work at all.
Basically ...
- 193
8
votes
1
answer
1k
views
ELK Stack (Logstash, Elasticsearch and Kibana) with concurrent remote syslog server?
I'm building a log analyser service to start monitoring mainly our pfSense Firewalls, XenServer Hypervisors, FreeBSD/Linux servers and Windows servers.
There's a lot of documentation on the internet ...
- 5,700
4
votes
1
answer
1k
views
Externally visible Kubernetes Service on Azure
I am deploying an ELK stack on a Kubernetes(v1.4.5) cluster on Azure. This is the configuration that creates the Kibana Service and Deployment.
# deployment.yml
---
apiVersion: v1
kind: Namespace
...
- 143
3
votes
1
answer
771
views
Monitoring and alerting tools for small businesses in 2016 [closed]
This is a kind-of-recurring question, yet the closest one I could find was asked 7 years ago, which was pretty much a different time.
I run a small business and we host multiple small-to-medium ...
- 153
3
votes
1
answer
12k
views
elk stack error "unable to fetch mapping do you have indices matching the pattern"
i am trying to setup ELK stack with collectd on Ubuntu 16.04 LTS (so pretty much latest version of stack available)
kibana is behind nginx proxy (followed this guide https://www.digitalocean.com/...
- 523
3
votes
1
answer
2k
views
Logstash event @timestamp adjustment
I have standard Windows IIS log files with event date/time stamp information and timetaken (in milliseconds).
I would like to be able to adjust the event time (@timestamp) by subtracting the "...
- 2,674
3
votes
2
answers
4k
views
How to add AWS ELB access logs to logstash with S3 input?
I'm using an ELK stack for server monitoring. My application's access logs which are from AWS ELB are stored in AWS S3. I am trying to add them to logstash with the following input:
input {
s3 {
...
- 43
2
votes
1
answer
15k
views
Filebeat can't connect to logstash on another server
Filebeat (11.11.11.11) can't connect to logstash (22.22.22.22) on another server (connection reset by peer). But filebeat services from other servers can do it.
Also I can connect from this server(11....
- 179
2
votes
1
answer
2k
views
Trouble bringing in CloudWatch data to Logstash
Just getting started with a new ELK setup (never used it before, just trying to learn it). I have Logstash 2.2.4 running on ubuntu 14.04 LTS.
After putting a yaml file down with my monitor user's ...
- 1,008
2
votes
1
answer
292
views
ELK logstash and core grok patterns
I'm evaluating the ELK stack with filebeat & logstash across a diverse range of applications/ servers.
I understand the power of customising my own grok patterns for each application/log, but to ...
- 141
2
votes
0
answers
347
views
Logstash to aggregate logs into ElasticSearch
I am trying to aggregate linux logs using rsyslog into Logstash/ElasticSearch running in EKS.
filebeat is already running in EKS to aggregate Kubernetes container logs.
I have configured rsyslog ...
- 101
2
votes
0
answers
4k
views
Parsing JSON event in Logstash
I have log in following format, it is a plain json with nested fields.
{
"level": "info",
"message": {
"req": {
"headers": {
"host": "localhost:8080",
...
- 21
2
votes
0
answers
1k
views
filebeat makes a lot of I/O
We have filebeat on few servers that is writeing to elasticsearch. We can see that it is doing a lot of writes:
PID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
353 be/3 ...
- 121
2
votes
0
answers
324
views
Is there any way to get log events from crashlytics / fabric into ELK or a SaaS metrics platform like datadog?
If you use a stack like ELK or datadog for collecting server-side logs and events, how do you integrate mobile-side metrics? Is there any way to get these out of crashlytics directly, or does this log ...
- 700
2
votes
0
answers
1k
views
How do I update Logstash data to Redshift real-time?
I am looking for a way to build a pipeline for storing the logs data (I use the ELK stack) to AWS Redshift, for analytics.
I have gone through the Output plugins for ElasticSearch, and found one for ...
- 153
1
vote
1
answer
15k
views
Auto delete elasticsearch data older than 30 days
I have setup a ELK stack to collect logs at central server. It is working perfectly. But by default it is holding elasticsearch index/data permanently. We just want to maintain the data for 30Days. ...
- 189
1
vote
2
answers
2k
views
Line breaks in PHP's stack trace in nginx error logs disturbing logstash analysis
I am using nginx with PHP-FPM and ELK as log file analysis.
When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file.
...
- 289
1
vote
2
answers
2k
views
Retrieve pfSense/freeBSD logs with elk
I am attempting to centralize logs from different systems.
I installed the Elastick Stack (Elasticsearch, Logstash, Kibana) and WAZUH OSSEC on one server (named elk).
I have installed the OSSEC ...
- 110
1
vote
1
answer
15k
views
cannot validate certificate - doesn't contain any IP SAN
I am currently in the process of installing ELK ( ElastricSearch, LogStash & Kibana) stack.
My ELK server IP address is 172.29.225.32.
Elastic Search config is ::
# -----------------------------...
- 185
1
vote
0
answers
659
views
Logstash syslog filter not applying to logs?
I'm looking through some syslog logs files in my ELK stack and noticed that all the syslog_severity fields are 'notice', when I can verify in the log files that they are not 'notice'. Seems like ...
- 171
1
vote
0
answers
407
views
Logstash filter: syslog_pri always defaulting to notice?
I'm looking through some syslog logs files in my ELK stack and noticed that all the syslog_severity fields are 'notice', when I can verify in the log files that they are not 'notice'. Seems like ...
- 171
1
vote
1
answer
503
views
Using ELK X-pack for general purpose alerts and alarms
The X-pack package from Elastic is fully integrated with Elasticsearch and Kibana to provide (among other things) an alarm reporting platform.
I believe that the regular use case is to build that ...
- 169
1
vote
0
answers
44
views
How does Docker Daemon handle large log output?
I have a number of server applications running in Docker. The output is configured to go to an elk stack.
I've had a number of troubles with the elk stack and am considering going back to vanilla ...
- 2,739
1
vote
1
answer
2k
views
elastic's snapshot and restore module repository_exception
I'm using elk-docker and trying to follow Snapshot And Restore | Elasticsearch Reference [2.4] | Elastic and getting following error:
# curl --request PUT --data '{ "type": "fs", "settings": {"...
- 13.5k
1
vote
0
answers
259
views
Visualize multiline ruby exceptions in kibana 4
I have setup the latest version of Kibana4 ElasticSearch stack. The logs are being pooled from remote app sources which are running on Ruby.
I want to search for Multi Line exceptions created by ...
- 211
1
vote
0
answers
166
views
Filebeat and downstream availability
I read here and there that a broker (like Redis) might not be required in the log pipeline (typically ELK) when Filebeat is used.
From Filebeat's official page:
[Filebeat] is intelligent enough to ...
1
vote
2
answers
9k
views
Passing JSON application log to remote LogStash via NXLog on Windows
I have been trying to pass logs from a windows application which are already formatted in JSON to logstash via NXlog.
When I have NXLOG send the file to Logstash, I get blasted with errors in the ...
- 126
0
votes
1
answer
3k
views
ELK: LogStash to read log files from remote Samba-mapped network drives
I'm new to ELK, and I would like to set up a solution to index Microsoft IIS and applicative .NET logs with ES.
I'm aware about different approaches:
1) [app servers: log files ➔ Logstash] ➔ [...
0
votes
1
answer
336
views
Parsing or Reformatting Logs before feeding them to Splunk or Elastic Search
I have very complex log messages, that I want to reduce to the most important fields in order to save quota.
The log messages are multiline and there is a lot of redundant information in them. A ...
- 131
0
votes
2
answers
2k
views
Troubleshooting rsyslog integration with ELK stack
I'm trying to configure rsyslog to send logs to logstash on CentOS. So I was following the tutorial.
However, after setting up, nothing happens. Everything started ok, not error occurred but no logs ...
- 101
0
votes
2
answers
495
views
ELK stack on AWS and web app on Digital Ocean, how and can I grab logs from the later?
I have a python website hosted on digital ocean and ELK stack - Elastic Search, Kibana, Logstash - on AWS. I'm trying to understand how to feed the logs files from my python web app on digital ocean ...
- 53
0
votes
1
answer
313
views
Stop filebeat sending copius metadata
I am sending data from local log files with filebeat to graylog and I am getting a 20x storage overhead compared to the original files. There are a large amount of metadata fields however I can't seem ...
- 23
0
votes
1
answer
3k
views
Can't access Kibana remotely - Can curl successfully on machine
I've installed Kibana on rhel and am I'm trying to access remotely on http://server-url:5601.
I've installed elasticsearch and am able to access that remotely on http://server-url:9200
I've added
...
0
votes
1
answer
51
views
Import Elasticsearch 2.X data to 6.X?
I've got some data/logs from Elasticsearch 2.4 & I've got a new Elasticsearch 6.X running.
What's the best/correct way to import data from 2.X to 6.x?
Thanks!
- 425
0
votes
1
answer
2k
views
HAProxy in front of logstash broken pipe
I'm setting up ELK stacks with loadbalancing. Haproxy works fine for ES and Kibana but I'm having issues with Logstash.
Here's the haproxy configuration:
frontend logstash
bind 0.0.0.0:5000
...
- 183
0
votes
1
answer
429
views
Can I setup logstash with windows generated certs instead of openssl?
Going through this tutorial.
They use openssl to generate certs to use with logstash ie this command:
sudo openssl req -subj '/CN=ELK_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -...
- 4,273
0
votes
1
answer
3k
views
What is "io/console not supported; tty will not be manipulated"?
What is this message and how can I get rid of it?
I am using ELK stack on windows 8. Every thing works just fine but this message which I get whenever I run logstash, buzzes me off.
This is the ...
- 117
0
votes
1
answer
3k
views
ESX performance stats in Logstash
Dose anyone know if it is possible to collect ESX Performance Stats in LogStash/ELK stack? looking to see if I can collect ESXTOP data for use in the ELK stack.
- 772
0
votes
0
answers
81
views
ECK Monitor Fleet Server Error When add TLS Certificate on Elasticsearch
When i add TLS Certificates in elasticsearch my fleet-server has error as detail below.
{"log.level":"error","@timestamp":"2024-05-03T08:02:03.862Z","log....
0
votes
0
answers
38
views
How to fetch Individual output of different match_phrase inside one request
I am able to run below query which is giving response as total of two match_phrase.
Inserting dummy data as below.
POST /mod1/_bulk
{ "index" : { } }
{ "msg": "BA1" }
{ &...
- 11
0
votes
1
answer
811
views
How to pass values to query externally
I have below similar logs.
I have created dummy index and created mapping like below in dev-tools
PUT new
{
"mappings": {
"properties": {
"@timestamp": {
...
- 11
0
votes
1
answer
1k
views
filebeat log status 30 every sec
I'm learning to use ELK and have a debian PC that runs as a test client.
every 30 sec it logs a message :
021-01-18T08:29:59.656-0500#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in ...
- 115
0
votes
1
answer
137
views
How to display the logs of a VM on ELK stack which is running on another VM?
-ELK is running on localhost & successfully getting logs.
-Both VM's are on same server.
-OS on both VM'S is Ubuntu 18-04 LTS.
- 13
0
votes
1
answer
111
views
I used elk+redis+filebeat to build the logging platform, and now after redis is replaced by codis, logstash reports？？？
I used elk+redis+filebeat to build the logging platform, and now after redis is replaced by codis, logstash reports:
# [2020-06-18T11:20:54,146][WARN ][logstash.inputs.redis] Redis connection problem {...
- 1
0
votes
2
answers
1k
views
Ship Logs from application server to ELK server with beats
I am building log analyzer for production environment. My scenario is, i need to setupelasticsearch,Logstash and kibana on one centos7 server which is going to use ELK server another one is apache ...
- 3
0
votes
1
answer
439
views
Export Google Cloud SQL slow logs to ELK stack
I stumbled upon an issue and decided to ask for advice and eventually find someone with the same business need (and problem).
Summary - we’ve recently migrated the SQL service of one of our clients ...
0
votes
0
answers
21
views
Adding context paths to ELK log analytics?
I'm looking to install the ELK stack on my application server by following the instructions here:
I can see you can then access the various ELK services on localhost via certain ports. Is there a way ...
0
votes
0
answers
2k
views
ELK - Logstash not picking up syslog events
I'm setting up a ELK cluster using Centos 8 and version 7.4 of Elasticsearch, Logstash and Kibana.
My issue is with Logstash not picking up the events coming through syslog.
Configuring Logstash to ...
- 267
0
votes
1
answer
988
views
logstash not able to upload data to elasticsearch even the pipeline started
I am using elasticsearch 7.1.1 and logstash 7.1.1. I am trying to upload a log file to elastic search using grok filter.
Pipeline is getting started, but data is not getting uploaded.
Here is my ...
- 11
0
votes
1
answer
148
views
Using Elasticsearch or Logstash output?
I have been going through a few tutorials on using beats to send data to elasticsearch.
I noticed that some tutorials prefer to use logstash as the output which then outputs to elasticsearch. Some ...
- 463