My stack is on AWS Fargate. Application, grafana, loki on the same ECS cluster. My application generates logs,
Log Stream sample below:
{"host":"ip-00-3001-1a08-test.us-west-1.compute.internal","short_message":"Sent message: {\"test\":{\"test\":\"31901300\",\"}}","full_message":"Sent message: {\"test\":{\"test\":\"31901300\"}}","timestamp":1.707740449767E9,"level":6,"facility":"logstash-abctest","simpleLogger":"test-abc","logSequence":9696969696,"activity":"00000000222255544","logLevel":"INFO","logger":"test","senderType":"test","terminalID":"test300"}
Find my logConfiguration in ECS Task_definitation:
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"LabelKeys": "container_name,ecs_task_definition,source,ecs_cluster",
"Labels": "{job=\"firelens\"}",
"LineFormat": "json",
"Name": "grafana-loki",
"RemoveKeys": "container_id,ecs_task_arn",
"Url": "http://loki.endpoint:3100/loki/api/v1/push"
}
There are some fields in my Log stream such as host, short_message, full_message, timestamp, level, facility, logSequence, activity, logLevel, logger, sendType, terminalID etc.
My goal is to parse these fields as Labels in Loki data source. Suggest necessary configuration to make in AWS ECS Task Definitation to accomplish it.
FYI, Currently, I am only able see meta data of cluster (container_name, ecs_cluster, ecs_task_definition, job, source) as Lebels.
Thanks in advance.
I tried to add fields of log stream in logConfiguration of the task definitation like this:
"LabelKeys": "senderType,activity,full_message,short_message,container_name,ecs_task_definition,source,ecs_cluster",
"Labels": "{job=\"firelens\",senderType=\"${senderType}\",activity=\"${activity}\"}",
But that didn't work.