Error configuring node-exporter DaemonSet scraping for Prometheus on kubernetes - service account token missing

Question

I have set up a node-exporter DaemonSet on kubernetes as well as a service that points to these node-exporter pods IPs (I followed this tutorial). When I run kubectl get endpoints -n monitoring, I verify that the service is correctly pointing to the 3 DaemonSet pods that were created.

After that, inside the prometheus.yml file I have added this config for scraping the node-exporter metrics:

  - job_name: "node-exporter"
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
          - monitoring
    relabel_configs:
      - source_labels: [__meta_kubernetes_endpoints_name]
        regex: "node-exporter"
        action: keep

The problem is when I apply these configs and restart the prometheus.service:

> systemctl status prometheys.service --no-pager --full

● prometheus.service - PromServer
     Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-06-20 14:23:22 UTC; 12min ago
   Main PID: 1564211 (prometheus)
      Tasks: 10 (limit: 33613)
     Memory: 43.6M
        CPU: 860ms
     CGroup: /system.slice/prometheus.service
             └─1564211 /usr/local/bin/prometheus --web.enable-admin-api --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries

Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.018Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=194 maxSegment=195
Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.019Z caller=head.go:755 level=info component=tsdb msg="WAL segment loaded" segment=195 maxSegment=195
Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.019Z caller=head.go:792 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=6.863563ms wal_replay_duration=60.448333ms wbl_replay_duration=100ns total_replay_duration=69.694752ms
Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.021Z caller=main.go:1040 level=info fs_type=EXT4_SUPER_MAGIC
Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.021Z caller=main.go:1043 level=info msg="TSDB started"
Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.021Z caller=main.go:1224 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.025Z caller=manager.go:317 level=error component="discovery manager scrape" msg="Cannot create service discovery" err="open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" type=kubernetes config=node-exporter
Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.029Z caller=main.go:1261 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=7.816785ms db_storage=1.6µs remote_storage=1.3µs web_handler=400ns query_engine=901ns scrape=3.284277ms scrape_sd=295.907µs notify=67.202µs notify_sd=18.7µs rules=3.682588ms tracing=6.5µs
Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.029Z caller=main.go:1004 level=info msg="Server is ready to receive web requests."
Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.029Z caller=manager.go:995 level=info component="rule manager" msg="Starting rule manager..."

I get this error specifically:

Jun 20 14:23:23 vm-devops-tiim-giro-prd prometheus[1564211]: ts=2024-06-20T14:23:23.025Z caller=manager.go:317 level=error component="discovery manager scrape" msg="Cannot create service discovery" err="open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" type=kubernetes config=node-exporter

A colleague suggested I disable SSL to which I did, adding to the prometheus.yml this tls_config lines, under the node-exporter job:

    tls_config:
      insecure_skip_verify: true

But sadly it didn't solve it...

Why am I getting this error since I haven't got a service account configured in any config file related to the node-exporter? Also I've checked and this /var/run/secrets/kubernetes.io/serviceaccount directory doesn't even exist. What am I missing?

Note: this question has the same context as this one, but now I am getting a different error