Mystery "Failed to locate executable"

Question

I've been setting up a monitoring solution for various servers using Promtail, Loki and Grafana, following this article. I got a monitoring machine running Loki and Grafana (on Rocky Linux 9.3) and a bunch of Ubuntu servers running Promtail, which shovels logs into Loki. That works great.

However, the last step I wanted to make was to set up Promtail for the monitoring machine as well. I follow the steps above - steps that have worked on some 20 servers - and suddenly, I keep getting "failed to locate executable":

[root@localhost ~]# systemctl status promtail.service
× promtail.service - Promtail for Loki
     Loaded: loaded (/etc/systemd/system/promtail.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2024-01-09 05:21:23 EST; 5s ago
   Duration: 22ms
    Process: 3633351 ExecStart=/usr/local/bin/promtail-linux-amd64 -config.file /etc/loki/promtail.yaml (code=exited, status=203/EXEC)
   Main PID: 3633351 (code=exited, status=203/EXEC)
        CPU: 21ms

Jan 09 05:21:23 localhost.localdomain systemd[1]: Started Promtail for Loki.
Jan 09 05:21:23 localhost.localdomain systemd[3633351]: promtail.service: Failed to locate executable /usr/local/bin/promtail-linux-amd64: Permission denied
Jan 09 05:21:23 localhost.localdomain systemd[3633351]: promtail.service: Failed at step EXEC spawning /usr/local/bin/promtail-linux-amd64: Permission denied
Jan 09 05:21:23 localhost.localdomain systemd[1]: promtail.service: Main process exited, code=exited, status=203/EXEC
Jan 09 05:21:23 localhost.localdomain systemd[1]: promtail.service: Failed with result 'exit-code'.

But the executable is in the right place, and the owner is the promtail user:

[root@localhost ~]# ls -al /usr/local/bin/
total 165048
drwxr-xr-x+  3 root     root          128 Jan  2 12:35 .
drwxr-xr-x. 12 root     root          131 May 30  2023 ..
-rwxr-xr-x+  1 loki     loki     59424768 May  3  2023 loki-linux-amd64
-rw-r--r--+  1 root     root     18930096 May 31  2023 loki-linux-amd64.zip
-rwxr-xr-x.  1 root     root          233 Nov  6 11:53 normalizer
-rwxrwxr--+  1 promtail promtail 90640576 May  3  2023 promtail-linux-amd64
drwxr-xr-x.  7 root     root         4096 Jan  4 06:47 server_heartbeat

The ACL looks like this:

[root@localhost ~]# getfacl /usr/local/bin/promtail-linux-amd64 
getfacl: Removing leading '/' from absolute path names
# file: usr/local/bin/promtail-linux-amd64
# owner: promtail
# group: promtail
user::rwx
group::r-x
other::r--

The service file specifies promtail as the user:

[Unit]

Description=Promtail for Loki

After=network.target

[Service]

Type=simple

User=promtail

ExecStart=/usr/local/bin/promtail-linux-amd64 -config.file /etc/loki/promtail.yaml

Restart=on-abort

NoNewPrivileges=true

PrivateTmp=yes

RestrictNamespaces=uts ipc pid user cgroup

ProtectKernelTunables=yes

ProtectKernelModules=yes

ProtectControlGroups=yes

#ProtectSystem=strict

#PrivateUsers=strict

#CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_DAC_READ_SEARCH

[Install]

WantedBy=multi-user.target

What am I missing?

Edit: Following up on @gerald-schneider's comment, the relevant SELinux contexts looks like this:

[root@localhost ~]# ls -Z /usr/local/bin/promtail-linux-amd64 
unconfined_u:object_r:admin_home_t:s0 /usr/local/bin/promtail-linux-amd64

[root@localhost ~]# ls -Z /etc/loki/promtail.yaml 
unconfined_u:object_r:admin_home_t:s0 /etc/loki/promtail.yaml

[root@localhost ~]# ls -Z /tmp/positions.yaml 
unconfined_u:object_r:user_tmp_t:s0 /tmp/positions.yaml

Answer 1

The permissions themselves look fine, but the . at the end of the permissions indicate that extended SELinux attributes are set.

Check the SELinux attributes using ls -Z and make sure the files have the correct SELinux context.