Prometheus Blackbox http_2xx returning 403 since webserver migrated to StackCP

Question

I had working Prometheus Blackbox Exporter http_2xx checks monitoring various web servers. Then the web hosting provider migrated from cPanel to Stack CP.

Since then all the http_2xx just return 403 (Forbidden) errors and I can't work out why as I can run successful wget and curl commands from the same Prometheus server/IP. I cannot seem to re-create a 403 error with other commands.

Can anyone make any suggestions why this might be the case or how to debug further?

I have checked StackCP config and there does not seem to be anything there which would forbid these monitoring checks.

I can't see anything in the web server logs about 403 errors.

blackbox.yml config starts:

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      method: GET

And I also tried this thinking it might be an SSL/certificate issue but this makes no difference:

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      method: GET
      tls_config:
        insecure_skip_verify: true

And syslog excerpt from the prometheus server:

Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.356Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Beginning probe" probe=http timeout_seconds=9.5
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.356Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolving target address" ip_protocol=ip6
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolving target address" ip_protocol=ip4
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Resolved target address" ip=185.151.30.208
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.371Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Making HTTP request" url=https://185.151.30.208 host=southcoastgroup.co.uk
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.406Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Received HTTP response" status_code=403
Jan 31 18:08:00 Overwatch prometheus-blackbox-exporter[3362305]: ts=2024-01-31T18:08:00.406Z caller=main.go:180 module=http_2xx target=https://southcoastgroup.co.uk level=debug msg="Invalid HTTP response status code, wanted 2xx" status_code=403

Ubuntu 22.04.3 LTS / prometheus, version 2.31.2+ds1 / blackbox_exporter, version 0.19.0

EDIT 1 as suggested by @AlexD although the outputs don't mean much to me. In the strace I can see it connecting to the webserver (185.151.30.208) but I can't see any 403 errors or any obvious reasons for the problem.

tcpdump:

12:38:35.206371 IP (tos 0x0, ttl 64, id 10246, offset 0, flags [DF], proto TCP (6), length 60)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [S], cksum 0x9a70 (incorrect -> 0xce3d), seq 709527684, win 64240, options [mss 1460,sackOK,TS val 1145455779 ecr 0,nop,wscale 7], length 0

12:38:35.212490 IP (tos 0x0, ttl 64, id 10247, offset 0, flags [DF], proto TCP (6), length 52)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xd0f0), ack 1090810164, win 502, options [nop,nop,TS val 1145455785 ecr 1774061831], length 0

12:38:35.213068 IP (tos 0x0, ttl 64, id 10248, offset 0, flags [DF], proto TCP (6), length 351)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9b93 (incorrect -> 0x362b), seq 0:299, ack 1, win 502, options [nop,nop,TS val 1145455786 ecr 1774061831], length 299

12:38:35.219896 IP (tos 0x0, ttl 64, id 10249, offset 0, flags [DF], proto TCP (6), length 52)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xc71e), ack 2207, win 496, options [nop,nop,TS val 1145455792 ecr 1774061839], length 0

12:38:35.221063 IP (tos 0x0, ttl 64, id 10250, offset 0, flags [DF], proto TCP (6), length 116)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9aa8 (incorrect -> 0x5ff2), seq 299:363, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 64

12:38:35.221177 IP (tos 0x0, ttl 64, id 10251, offset 0, flags [DF], proto TCP (6), length 138)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9abe (incorrect -> 0xe1f8), seq 363:449, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 86

12:38:35.221291 IP (tos 0x0, ttl 64, id 10252, offset 0, flags [DF], proto TCP (6), length 125)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9ab1 (incorrect -> 0xdb93), seq 449:522, ack 2207, win 501, options [nop,nop,TS val 1145455794 ecr 1774061839], length 73

12:38:35.227011 IP (tos 0x0, ttl 64, id 10253, offset 0, flags [DF], proto TCP (6), length 52)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [.], cksum 0x9a68 (incorrect -> 0xc543), ack 2439, win 501, options [nop,nop,TS val 1145455800 ecr 1774061846], length 0

12:38:35.227070 IP (tos 0x0, ttl 64, id 10254, offset 0, flags [DF], proto TCP (6), length 83)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9a87 (incorrect -> 0x9012), seq 522:553, ack 2439, win 501, options [nop,nop,TS val 1145455800 ecr 1774061846], length 31

12:38:35.229169 IP (tos 0x0, ttl 64, id 10255, offset 0, flags [DF], proto TCP (6), length 76)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [P.], cksum 0x9a80 (incorrect -> 0xa0f1), seq 553:577, ack 2614, win 501, options [nop,nop,TS val 1145455802 ecr 1774061848], length 24

12:38:35.229208 IP (tos 0x0, ttl 64, id 10256, offset 0, flags [DF], proto TCP (6), length 52)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [F.], cksum 0x9a68 (incorrect -> 0xc458), seq 577, ack 2614, win 501, options [nop,nop,TS val 1145455802 ecr 1774061848], length 0

12:38:35.234976 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b6 (correct), seq 709528262, win 0, length 0

12:38:35.235003 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b6 (correct), seq 709528262, win 0, length 0

12:38:35.235026 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)

    192.168.1.50.42780 > 185.151.30.208.443: Flags [R], cksum 0xb7b5 (correct), seq 709528263, win 0, length 0

Excerpt from sudo strace -f -o file -p 3043635:

3043960 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>

3043636 <... nanosleep resumed>NULL)    = 0

3043960 <... futex resumed>)            = 1

3043638 <... futex resumed>)            = 0

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043638 futex(0xc000046d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>

3043960 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...>

3043636 <... nanosleep resumed>NULL)    = 0

3043960 <... socket resumed>)           = 8

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043960 connect(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("185.151.30.208")}, 16) = -1 EINPROGRESS (Operation now in progress)

3043636 <... nanosleep resumed>NULL)    = 0

3043960 epoll_ctl(5, EPOLL_CTL_ADD, 8, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=2636876800, u64=140362168110080}} <unfinished ...>

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043960 <... epoll_ctl resumed>)        = 0

3043960 write(7, "\0", 1)               = 1

3043636 <... nanosleep resumed>NULL)    = 0

3043958 <... epoll_pwait resumed>[{events=EPOLLIN, data={u32=15774768, u64=15774768}}], 128, 9499, NULL, 2321375387197960) = 1

3043960 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043960 <... futex resumed>)            = 1

3043958 read(6,  <unfinished ...>

3043638 <... futex resumed>)            = 0

3043960 futex(0xc000081148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>

3043958 <... read resumed>"\0", 16)     = 1

3043638 futex(0xc000046d48, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>

3043636 <... nanosleep resumed>NULL)    = 0

3043958 epoll_pwait(5,  <unfinished ...>

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043958 <... epoll_pwait resumed>[], 128, 0, NULL, 2321375387197960) = 0

3043958 epoll_pwait(5,  <unfinished ...>

3043636 <... nanosleep resumed>NULL)    = 0

3043636 futex(0xedd178, FUTEX_WAIT_PRIVATE, 0, {tv_sec=9, tv_nsec=463217921} <unfinished ...>

3043958 <... epoll_pwait resumed>[{events=EPOLLOUT, data={u32=2636876800, u64=140362168110080}}], 128, 9463, NULL, 2321375387197773) = 1

3043958 futex(0xedd178, FUTEX_WAKE_PRIVATE, 1) = 1

3043958 getsockopt(8, SOL_SOCKET, SO_ERROR,  <unfinished ...>

3043636 <... futex resumed>)            = 0

3043958 <... getsockopt resumed>[0], [4]) = 0

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043958 getpeername(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("185.151.30.208")}, [112 => 16]) = 0

3043636 <... nanosleep resumed>NULL)    = 0

3043958 futex(0xc000046d48, FUTEX_WAKE_PRIVATE, 1) = 1

3043638 <... futex resumed>)            = 0

3043636 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

3043958 getsockname(8,  <unfinished ...>

3043638 epoll_pwait(5,  <unfinished ...>

EDIT 2 as noticed by @DazWilkin openssl commands were giving an error on my server (Verify return code: 21 (unable to verify the first certificate). After checking the site on https://www.ssllabs.com it looked like there were extra downloads in the certification paths and I was missing a couple of intermediate certificates on my server. So I fixed that and now the openssl commands work without any certificate errors.

However, Blackbox http_2xx still returns 403

Use strace and tcpdump to check what actually blackbox-exporter doing. — AlexD, Commented Feb 1 at 9:09
There's an issuer with the server's cert (unable to get local issuer certificate) which may be causing this problem curl --silent --verbose --request GET https://southcoastgroup.co.uk --write-out '%{response_code}' --output /dev/null and openssl s_client -showcerts -servername southcoastgroup.co.uk -connect southcoastgroup.co.uk:443 </dev/null — DazWilkin, Commented Feb 3 at 17:35
@DazWilkin thanks for the suggestion. I did have problems with certificates with wget/curl commands after the migration as it seemed the hosting provider also changed the certificate in some way too. I had to download and install some new CA and intermediate certificates to my Ubuntu. So currently on my server your curl command results in SSL certificate verify ok. but your openssl command does still give an error Verify return code: 21 (unable to verify the first certificate). So perhaps this is a certificate issue. I will investigate further to see if I can get openssl working — codlord, Commented Feb 4 at 7:49
@DazWilkin fixed the openssl cert errors on my server which I think were more like warnings anyway, so updated my post. All curl/wget/openssl all run without any errors on my server now. But still no closer — codlord, Commented Feb 4 at 8:40

codlord · Accepted Answer · 2024-02-04 08:55:48Z

o.k. so I finally figured it out. For some reason the migrated webserver was rejecting based on the User-Agent (or perhaps lack of a User-Agent as I am not sure what or if the http2_xx probe sends a User-Agent). But all I needed to do was add a fake browser-like User-Agent in my case like this:

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      method: GET
      headers:
        User-Agent: "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"

Stack Exchange Network

Prometheus Blackbox http_2xx returning 403 since webserver migrated to StackCP

1 Answer 1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
ubuntu
monitoring
curl
wget
prometheus
.

Hot Network Questions

Prometheus Blackbox http_2xx returning 403 since webserver migrated to StackCP

1 Answer 1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged ubuntumonitoringcurlwgetprometheus.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
ubuntu
monitoring
curl
wget
prometheus
.