0

I'm using the kube-prometheus-stack Helm chart to deploy Prometheus and configure ServiceMonitors for monitoring EC2 instances with the Node Exporter. I have configured the ServiceMonitor to relabel EC2 instance tags, but the instance label is not being applied as expected.

I have verified the following:

  • EC2 instances have the required tags. IAM roles and permissions
  • are correctly set up. ServiceMonitor and Prometheus configurations
  • include the required relabeling rules.
prometheus-node-exporter:
  resources:
    requests:
      memory: "64Mi"
      cpu: "100m"
  service:
    annotations:
      service.kubernetes.io/topology-aware-hints: Auto
      service.kubernetes.io/topology-aware-routing: Auto
  extraArgs:
    - --collector.ethtool
  nodeSelector:
    kubernetes.io/os: linux
  rbac:
    pspEnabled: false
  serviceMonitor:
    enabled: true
    interval: 30s
    jobLabel: node-exporter
    metricRelabelings: []
    relabelings:
      - sourceLabels: [__meta_ec2_tag_Name]
        action: replace
        targetLabel: instance
      - sourceLabels: [__address__]
        action: replace
        targetLabel: __address__
  prometheus:
    monitor:
      relabelings:
        - sourceLabels: [__meta_ec2_tag_Name]
          action: replace
          targetLabel: instance
        - sourceLabels: [__address__]
          action: replace
          targetLabel: __address__

However, the relabeling for the instance label does not seem to be working. Below is the output of a dry run for the ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: monitoring-prometheus-node-exporter
  namespace: monitoring
  labels:
    helm.sh/chart: prometheus-node-exporter-4.34.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: metrics
    app.kubernetes.io/part-of: prometheus-node-exporter
    app.kubernetes.io/name: prometheus-node-exporter
    app.kubernetes.io/instance: monitoring
    app.kubernetes.io/version: "1.8.0"
    jobLabel: node-exporter
    release: monitoring
spec:
  jobLabel: node-exporter
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-node-exporter
      app.kubernetes.io/instance: monitoring
  attachMetadata:
    node: false
  endpoints:
    - port: http-metrics
      scheme: http
      relabelings:
        - action: replace
          sourceLabels:
          - __meta_ec2_tag_Name
          targetLabel: instance
        - action: replace
          sourceLabels:
          - __address__
          targetLabel: __address__

Despite this configuration, the instance label does not appear in the Prometheus UI for the targets.

I expected the instance label to be populated with the EC2 tag Name for each target monitored by the ServiceMonitor. However, this is not happening, and the label is not present in the target list in the Prometheus UI.

0

You must log in to answer this question.

Browse other questions tagged .