nginx prox_pass: port 3000 gets redirected to 80. Same config with other ports works

Question

I have the following scenario:

Service A

Service A is available under host:8080.

I have configured a reverse proxy in nginx to resolve servicea.domain to host:8080.

Here is my config-file (Location: /etc/nginx/sites-available/servicea)

server {
    listen 80;
    listen [::]:80;

    server_name servicea.domain.com;

    location / {
        proxy_pass http://host:8080/admin/;
        include proxy_params;

    proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 90;
        proxy_set_header X-Forwarded-Proto $scheme;

    set $xforwardedssl "off";
    if ($scheme = https) {
            set $xforwardedssl "on";
    }
    }
}

Service B

I would like to do the same with Service B (Grafana). This can be reached under host:3000. My nginx-config under /etc/nginx/sites-available/serviceb looks like this:

server {
    listen 80;
    listen [::]:80;

    server_name serviceb.domain.com;

    location / {
        proxy_pass http://host:3000/;
        include proxy_params;

    proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 90;
        proxy_set_header X-Forwarded-Proto $scheme;

    set $xforwardedssl "off";
    if ($scheme = https) {
            set $xforwardedssl "on";
    }
    }
}

• Both files are symlinked to /etc/nginx/sites-enabled/.
• Nginx starts successfully and does not complain.
• Everything works when calling servicea.domain.
• Ehen calling serviceb.domain I get a 400 error code in the browser.

When I use wget to load the page, I see that it does not actually resolve to host:3000 but to host:80.

╰─$ wget serviceb.domain.com
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/home/config/.wget-hsts'. HSTS will be disabled.
--2024-04-08 12:17:00--  http://serviceb.domain.com/
Resolving serviceb.domain.com (serviceb.domain.com)... 10.25.25.34
Connecting to serviceb.domain.com (serviceb.domain.com)|10.25.25.34|:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
2024-04-08 12:17:03 ERROR 400: Bad Request.

Why is that? I have the same configuration 1:1? A little proof that the config is the same. Here is the output of diff:

╰─$ diff serviceb servicea
5c5
<     server_name servicea.domain.com;
---
>     server_name serviceb.domain.com;
8c8
<         proxy_pass http://host:8080/admin/;
---
>         proxy_pass http://host:3000/;

Can anyone give me a hint where I can find settings that override my reverse proxy or otherwise impact name resolution? Let me know, if you need further information.

Thank you in advance!

Answer 1

Nevermind. The problems have been resolved.

Some clarification here:

• Port 80 in the wget command was simply the requested port on the reverse proxy, which is how it should be.
• There was something wrong with the configuration files. However, a lot has changed in the meantime.

Here is the current configuration we use, for anyone who has made their way here through the internet.

service-a

#########################################################################################
# Service A --> /etc/nginx/conf.d/service-a.conf

server {
  server_name service-a.domain.tld;
    location / {
        proxy_pass http://host-a:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 90;
        proxy_set_header X-Forwarded-Proto $scheme;
        set $xforwardedssl "off";
        if ($scheme = https) {
            set $xforwardedssl "on";
         }
        proxy_set_header X-Forwarded-Ssl $xforwardedssl;
    }

    listen [::]:443 ssl; 
    listen 443 ssl; 
    ssl_certificate /etc/letsencrypt/live/domain.tld/fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/live/domain.tld/privkey.pem; 
    include /etc/letsencrypt/options-ssl-nginx.conf; 
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; 


}
server {
    if ($host = service-a.domain.tld:8080) {
        return 301 https://$host$request_uri;
    } 


  listen 80;
  listen [::]:80;
  server_name service-a.domain.tld;
    return 404; 
}

service-b (Grafana)

#########################################################################################
# Service B (Grafana) /etc/nginx/conf.d/service-b.conf

server {
  server_name service-b.domain.tld;
    location / {
        proxy_pass http://host-a:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 90;
        proxy_set_header X-Forwarded-Proto $scheme;
        set $xforwardedssl "off";
        if ($scheme = https) {
            set $xforwardedssl "on";
         }
        proxy_set_header X-Forwarded-Ssl $xforwardedssl;
    }

    listen [::]:443 ssl; 
    listen 443 ssl; 
    ssl_certificate /etc/letsencrypt/live/domain.tld/fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/live/domain.tld/privkey.pem; 
    include /etc/letsencrypt/options-ssl-nginx.conf; 
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; 


}
server {
    if ($host = service-b.domain.tld:3000) {
        return 301 https://$host$request_uri;
    } 


  listen 80;
  listen [::]:80;
  server_name service-b.domain.tld;
    return 404; 
}

and here the changed parts in /etc/grafana/grafana.ini:

#################################### Server ####################################
[server]
# Protocol (http, https, h2, socket)
protocol = https

# The ip address to bind to, empty will bind to all interfaces
http_addr =

# The http port  to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = domain.tld

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
;root_url = %(protocol)s://%(domain)s:%(http_port)s/
root_url = https://subdomain.domain.tld:3000


# https certs & key file
cert_file = /etc/grafana/grafana.crt
cert_key = /etc/grafana/grafana.key

For further information regarding Grafana, see the original doku.