I'm using the kube-prometheus-stack Helm chart to deploy Prometheus and configure ServiceMonitors for monitoring EC2 instances with the Node Exporter. I have configured the ServiceMonitor to relabel EC2 instance tags, but the instance label is not being applied as expected.
I have verified the following:
- EC2 instances have the required tags. IAM roles and permissions
- are correctly set up. ServiceMonitor and Prometheus configurations
- include the required relabeling rules.
prometheus-node-exporter:
resources:
requests:
memory: "64Mi"
cpu: "100m"
service:
annotations:
service.kubernetes.io/topology-aware-hints: Auto
service.kubernetes.io/topology-aware-routing: Auto
extraArgs:
- --collector.ethtool
nodeSelector:
kubernetes.io/os: linux
rbac:
pspEnabled: false
serviceMonitor:
enabled: true
interval: 30s
jobLabel: node-exporter
metricRelabelings: []
relabelings:
- sourceLabels: [__meta_ec2_tag_Name]
action: replace
targetLabel: instance
- sourceLabels: [__address__]
action: replace
targetLabel: __address__
prometheus:
monitor:
relabelings:
- sourceLabels: [__meta_ec2_tag_Name]
action: replace
targetLabel: instance
- sourceLabels: [__address__]
action: replace
targetLabel: __address__
However, the relabeling for the instance label does not seem to be working. Below is the output of a dry run for the ServiceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: monitoring-prometheus-node-exporter
namespace: monitoring
labels:
helm.sh/chart: prometheus-node-exporter-4.34.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: metrics
app.kubernetes.io/part-of: prometheus-node-exporter
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: monitoring
app.kubernetes.io/version: "1.8.0"
jobLabel: node-exporter
release: monitoring
spec:
jobLabel: node-exporter
selector:
matchLabels:
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: monitoring
attachMetadata:
node: false
endpoints:
- port: http-metrics
scheme: http
relabelings:
- action: replace
sourceLabels:
- __meta_ec2_tag_Name
targetLabel: instance
- action: replace
sourceLabels:
- __address__
targetLabel: __address__
Despite this configuration, the instance label does not appear in the Prometheus UI for the targets.
I expected the instance label to be populated with the EC2 tag Name for each target monitored by the ServiceMonitor. However, this is not happening, and the label is not present in the target list in the Prometheus UI.