Questions tagged [fluentd]
Fluentd is an open source log aggregator.
47
questions
5
votes
1
answer
593
views
How to maximize utilization of fluentd server?
I have a fluentd server that is processing an average of 1000 req/s. This server is composed by 32 cpus. In this server are running 32 fluentd docker containers with the same configuration. The entry ...
- 53
5
votes
0
answers
2k
views
Consuming multiline json in fluentd
I am new to fluentd.
I have applications that run in Docker containers. They are Java apps that log in JSON format. The JSON messages are usually split over multiple lines.
I would like to use the ...
- 299
4
votes
1
answer
2k
views
How to specify the log tag from docker compose on google logging?
In my docker compose file, I can specify the logging driver as a "fluentd" and by default, it pushes the log to the listener of google-fluentd on port 24224
I can see the log appear on the logging ...
- 213
4
votes
1
answer
216
views
logstash-forward equivalent for fluentd?
Is there something equivalent to logstash-forwarder that can ship logfiles to fluentd?
I am trying to send log files from an application to a remote fluentd but have not seen whether this is ...
- 6,965
3
votes
2
answers
6k
views
fluent configuration not working
I have setup fluent with elasticsearch and Kibana with nginx. I followed the instruction and when I go to http:/192.168.10.25:8888 I get the following message
400 Bad Request
'json' or 'msgpack' ...
- 587
3
votes
3
answers
2k
views
Disable fluentd on on Container-Optimized OS (GCE)
I would like to disable fluentd on a VM instance running Container-Optimized OS on Google compute engine.
The reason being that it is eating up a lot of RAM for functionalities I do not require.
$ ...
google-cloud-platform
2
votes
1
answer
5k
views
Forward linux logs to fluentd on linux
On one VM I have this in /etc/rsyslog.d/50-default.conf
*.* @192.168.29.1:42185
# Default rules for rsyslog.
On the vm with fluentd I have this:
I have this in /etc/td-agent/td-agent.conf
<...
- 133
2
votes
1
answer
2k
views
GCE stackdriver logging agent (fluentd) memory leak with COS
I have a VM on GCE where I run a custom Docker image. I installed it on COS (cos-stable-74-11895-125-0) to a g1-small (1 vCPU, 1.7 GB memory) instance.
It is an Elasticsearch with locked memory ...
2
votes
2
answers
316
views
How to collect logs for statistics in site?
I want to collect logs from nginx ( several servers, 1 000 000 lines in logs for minute for all servers ) to central stat server for statistics processing.
I see 2 variants:
write logs to local log ...
- 181
2
votes
2
answers
3k
views
How to get JSON-tokenized logs into cloudwatch with fluent bit in EKS 1.22 (containerd)
I would like to get the message of my log entry into AWS with correct json tokenization from CRI application logs when running in AWS EKS (version 1.22)
My application outputs valid json, but the log ...
- 941
2
votes
0
answers
243
views
Monitoring Cassandra Server Logs?
I've done some searching and have not found any examples of people using something like FluentD to forward Cassandra log file data (system.log) to a log indexer.
Is this something any is doing? If ...
- 123
2
votes
0
answers
482
views
fluentd modular configuration
Is there any way to setup fluentd/td-agent in a way that it's configuration will be modular?
I know there is @include directive but this works only if every time I add something new I modify main td-...
- 133
1
vote
1
answer
2k
views
How to address container for logging using docker-compose
A container described in docker-compose.yml uses logging extension to send log to a fluentd container.
version: "2"
services:
fluentd:
image: fluent/fluentd:v0.14.8
container_name: ...
- 233
1
vote
2
answers
4k
views
Rsyslog - "Warning: CA certificate is not set" but TLS forwarding still works
I am configuring rsyslog to forward over TCP/TLS. I was curious why, unlike HTTPS, I had to add the certificate to my client, so I went against the README and commented out the cert config, leaving me ...
1
vote
1
answer
5k
views
How to retrieve logs from AWS EC2 and Docker containers fail-proof?
Currently, we have the following setup:
Multiple AWS EC2 machines, some which have a Docker Container running, some are dockerless.
To retrieve Syslogs, webserver logs (Apache) and application logs, ...
- 141
1
vote
1
answer
3k
views
Installing fluentd / Kibana / Elastic Search on Suse Enterprise Linux 11
I have successfully installed the recommended fluentd stack ( fluentd, Kibana, Elastic Search ) on a Centos 6 virtual machine. But our production environment is all Suse Enterprise Linux 11.
I am ...
user35861
1
vote
1
answer
863
views
Fluentd not writing apache access log to mongodb
I'm following the instructions from this link to write apache access logs to mongo db- http://docs.fluentd.org/articles/apache-to-mongodb.
But nothing is getting written to mongo DB. I'm checking ...
- 121
1
vote
1
answer
844
views
Could not contact DNS servers (fluent bit)
I am configuring the fluent bit connection with elasticsearch, I have 3 elasticsearch nodes and I want to have high availability, according to the following documentation it is done like this text, ...
1
vote
0
answers
29
views
Is there a preferred way to migrate from logcheck to fluentd?
Logcheck uses a negative filter approach, you specify the log messages that you consider harmless and you will see the rest. Looking at fluentd it is the opposite, it will filter messages that are ...
- 11
1
vote
0
answers
104
views
Routing Fluent Bit tcp event doesn't match to Fluent
i’m new to Fluent and Fluent Bit.
Fluent Bit (0.13) listens to tcp and forwards to a Fluent instance (td-agent 3.2). Fluent will push it to rabbitmq. The basic process is working.
Fluent Bit ...
- 11
1
vote
0
answers
125
views
Microsoft OMSAgent Listens on TCP port 16001
I have installed the Microsoft OMSAgent (on Linux). Now I see that it listens on TCP port 16001.
Neither the documentation nor grepping through the Sourcecode gave some hints why.
Anyone got an ...
- 443
1
vote
0
answers
149
views
fluentd missing region error
i want to fetch cloudwatch logs. I installed cloudwatch input plugin and configured the config file of td-agent. but im getting the error as
"unexpected error error_class=aws::errors::...
- 11
0
votes
3
answers
10k
views
Configuring Logrotate for Fluentd files. Necessary?
I have the following fluent.conf
<source>
type forward
</source>
<source>
type monitor_agent
port 24220
</source>
# Listen DRb for debug
<source>
type ...
- 113
0
votes
1
answer
336
views
Parsing or Reformatting Logs before feeding them to Splunk or Elastic Search
I have very complex log messages, that I want to reduce to the most important fields in order to save quota.
The log messages are multiline and there is a lot of redundant information in them. A ...
- 131
0
votes
1
answer
728
views
Parse nested JSON logs in Grafana Loki (forwarded with Fluentd)
My lab has a few Docker containers as follows:
name
Docker image
Fluentd
fluent/fluentd:v1.16-1
Fluent-bit
cr.fluentbit.io/fluent/fluent-bit
Loki
grafana/loki
Grafana
grafana/grafana-enterprise
...
- 487
0
votes
1
answer
289
views
Cannot add Kubernetes metadata as label to Loki container logs in FluentBit
I am trying to gather container logs in Kubernetes environment using FluentBit, Loki and Grafana.
I would like to append Kubernetes Pod name as label so when I query the logs in Grafana, it would be ...
0
votes
2
answers
1k
views
How to config Zabbix item read data from port and create graphs?
I am new to Zabbix. I send data from fluentd client to zabbix server port 10050. My question is How should I config Zabbix item to read this data from port and generate graph for it.
My data is like
...
- 101
0
votes
1
answer
245
views
Fluentd elasticsearch comprssion
I forward logs from app servers with fluentd (type tail) to elasticsearch server. Do not understand the following:
Can I enable for reducing traffic the compression transmitted logs with type tail ...
- 31
0
votes
0
answers
12
views
How can I resolve fluentd-plugin-slack error when starting Fluentd daemon: undefined method `unescape' for URI:Module (NoMethodError)
I'm trying to get fluentd to send messages to slack. I installed fluentd via curl and fluentd-plugin-slack via fluent-gem install fluent-plugin-slack however, I'm getting these errors when trying to ...
- 121
0
votes
1
answer
291
views
Fluent-Bit - How to extract part of a file name and append it to log_stream_name for Cloudwatch log group
New to Fluent-Bit and looking for some guidance on this.
Latest version of Fluent-Bit is installed on ec2 and successfully running and working.
I am pushing logs from /var/log/nginx... to Cloudwatch
...
0
votes
0
answers
50
views
Fluentd tail config for HashiCorp Vault audit/server logs from stdout
I'm trying to figure out a way to get both the audit and server logs from HasiCorp Vault container (both logs go to stdout of the same container and they have a different structure):
Audit example:
...
0
votes
1
answer
674
views
How can I get a list of the fluent-bit inputs supported in my system?
I've tried to use the cpu plugin and it says that 'cpu' tried to instance a plugin name that don't exists. I know that not all inputs are supported on every operating system but how can I guest a list ...
- 181
0
votes
1
answer
85
views
How to gather application logs which are generating in custom path using EFK
I created the elasticsearch and kibina setup which is running outside the kubernetes cluster and i am using fluentd to gather kubernetes logs and to send data to elasticsearch. I am running fluentd as ...
- 101
0
votes
1
answer
726
views
Fluent-bit in Kubernetes cannot input timestamp data properly
At my company, I built a K8s cluster with Terraform and configured a logging system with EFK (Elasticsearch, Fluent-bit, Kibana).
k8s and Elasticsearch use AWS's EKS and Opensearch Servcie (ES 7.10), ...
- 1
0
votes
1
answer
544
views
How to stream Kubernetes log files to a different location using fluent-bit?
I have a requirement where I want the log files of all the Kubernetes containers stored in the location /var/log/containers/ to be kept persistent even after a cluster restart or node issue. I have ...
0
votes
0
answers
121
views
Fluentd logs not sent to Elasticsearch on K8s
I have a cluster in VirtualBox to learn kubernetes. I have a deployment that contains MySQL and phpMyAdmin. I created a DemonSet that has the fluentd image and collects the logs to transmit them to ...
- 1
0
votes
0
answers
185
views
How to add configuration for fluent-plugin-prometheus in Fluentd deployed via Fleet in Rancher?
I'm using Rancher to manage my Kubernetes cluster and have added a logging system (cattle-logging-system) via Fleet. I now need to add monitoring for Fluentd using the fluent-plugin-prometheus.
Here's ...
0
votes
0
answers
309
views
Fluentd logs not sent to Elasticsearch
I have a cluster in VirtualBox to learn kubernetes. I have a deployment that contains MySQL and phpMyAdmin. I created a DemonSet that has the fluentd image and collects the logs to transmit them to ...
- 1
0
votes
1
answer
3k
views
How to enable read permission on Kubernetes mount path
I have installed FluentD on Kind-Kubernetes cluster on CentOS VM running on my laptop. I'm having issues getting FluentD to read logs as it is throwing the following error.
2021-08-29 08:26:31 +0000 [...
0
votes
2
answers
2k
views
Kubernetes DaemonSet Pod exceeds memory limit but is not killed
We use kubernetes (specifically: openshift) to run our infrastructure in our team.
One of the daemonsets, fluentd, is currently causing a lot of trouble, frequently taking full nodes down with huge ...
- 360
0
votes
1
answer
238
views
installing google-fluentd agent on docker produce an error
I am trying to install google-fluentd agent on a docker and get the following error:
Restarting google-fluentd: google-fluentd ... failed!
The OS is Debian GNU/Linux 9 (stretch)
Docker file snippet:...
0
votes
1
answer
1k
views
Send Kube logs to Cloudwatch using fluentd
My cluster is on AWS and I've used kops to build by cluster.
I'm trying to send the kubernetes logs to AWS Cloud-watch using fluentd. I was able to stand-up the fluentd pods.
I am seeing the below ...
- 27
0
votes
2
answers
195
views
systemd and launching docker containers
So I am new to docker and wanted to start up a logspout docker container on server start up so I thought i would just chuck in a systemd start up file:
[Unit]
Description=Logspout GELF Container
...
- 101
0
votes
0
answers
590
views
fluentd services die immediately after they start up
Please excuse the naivety in my question, but this is not a subject I know much about at present.
My company is currently running kubernetes-managed fluentd processes to push logs to logstash. These ...
- 111
0
votes
0
answers
1k
views
using google fluentd to log from compute engine instance in google cloud
I got a problem with the logging agent from Google Cloud.
What i wanted was to use the google-fluentd (based on fluentd) on a webserver to collect the logs and sent it to Stackdriver Logging Api.
...
0
votes
1
answer
1k
views
How to split a single record to multiple records in fluentd?
I would like to ask regarding to fluentd.
My fluentd versions is below.
td-agent-2.1.5-0.x86_64
fluentd 0.10.61
I now have a tail input plugin using multiple line format which parses multiple lines ...
- 616
0
votes
2
answers
453
views
td-agent shared key mismatch between servers
I am trying to set encrypted logs with td-agent between two machines. I have followed the examples from the official documentation of fluentd however i am stuck in a very weird situation. The client ...
- 1,894
Related Tags
kubernetes × 13
logging × 12
docker × 7
elasticsearch × 6
linux × 4
logstash × 3
nginx × 2
monitoring × 2
log-files × 2
rsyslog × 2
syslog × 2
docker-compose × 2
grafana × 2
loki × 2
mysql × 1
ssl × 1
amazon-ec2 × 1
redhat × 1
load-balancing × 1
configuration × 1
more related tags
Hot Network Questions
- Why do we care if the likelihood function is tractable?
- In By His Bootstraps (Heinlein) why is Hitler's name Schickelgruber?
- Cleaning chain a few links at a time
- Could a transparent frequency-altering material be possible?
- Tombs of Ancients
- Is it consistent with ZFC that the real line is approachable by sets with no accumulation points?
- Why is completeness (as in Gödel completeness theorem) a desirable feature?
- Could space habitats have large transparent roofs?
- How to achieve punya in our lives and get human birth in next life?
- How will the ISS be decommissioned?
- Which numbers are sums of finite numbers of reciprocal squares?
- Is it better to show fake sympathy to maintain a good atmosphere?
- "All due respect to jazz." - Does this mean the speaker likes it or dislikes it?
- Sets of algebraic integers whose differences are units
- Font shape warnings in LuaLaTeX but not XeLaTeX
- Should I accept an offer of being a teacher assistant without pay?
- Why did Geordi have his visor replaced with ocular implants between Generations and First Contact?
- Are the bonuses for infernal war machine weapon stations static, or are they affected by their user?
- Nuance of やぶさかではない
- What could explain that small planes near an airport are perceived as harassing homeowners?
- Co-authors with little contribution
- Why Owasp-crs does not allow Content-Type: application/x-www-form-urlencoded
- How are "pursed" and "rounded" synonymous?
- Would a spaceport on Ceres make sense?