telnet to port 80 (nginx), with forward to apache2 at port 8080, fails with 'connection closed by foreign host' and no explanations seemingly anywhere

Question

i'm running an Kubuntu 22.04 server, and want to shuttle LAN-only traffic from port 80, served as far as i can tell by nginx, to port 8080 served by apache2 (for PHP purposes).

but i'm getting this "connection closed by foreign host" (presumably from nginx), that's proving a complete show-stopper at port 80. i can still access the PHP and code filetree that i need to access by addressing port 8080 directly in the browser, but i do wonder what's causing this.

any help would be much appreciated. :)

root@parakeet:/var/log/nginx# netstat -tulpn | grep 80
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      3001930/nginx: mast 
tcp6       0      0 :::80                   :::*                    LISTEN      3001930/nginx: mast 
tcp6       0      0 :::8080                 :::*                    LISTEN      3001981/apache2  

root@parakeet:/var/log/nginx# cat error.log
root@parakeet:/var/log/nginx# ls -al | grep error.log
-rw-r-----  1 www-data adm          0 jul  2 21:09 error.log

root@parakeet:/etc/nginx/sites-enabled# cat 000-localhost.conf 
#
# Note: This file must be loaded before other virtual host config files.
# HTTPS
server {
  listen 80 http2;
  listen [::]:80 http2;

  server_name 192.168.178.29;
  root /var/www/192.168.178.29;
    
  add_header 'Access-Control-Allow-Origin' 'https://fiddle.jshell.net' always;
  add_header 'Access-Control-Allow-Credentials' 'true' always;
  add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
  add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With' always;

  location / {
    proxy_pass https://192.168.178.29:8080/;

    proxy_redirect off;
    proxy_buffering off;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    #proxy_set_header X-Forwarded-Ssl on;

    proxy_connect_timeout 159s;
    proxy_send_timeout   60;
    proxy_read_timeout   60;
    send_timeout 60;
    resolver_timeout 60;
  }
}

root@parakeet:/etc/apache2/sites-enabled# cat 000-localhost.conf 
<VirtualHost 192.168.178.29:8080>
        # The ServerName directive sets the request scheme, hostname and port that
        # the server uses to identify itself. This is used when creating
        # redirection URLs. In the context of virtual hosts, the ServerName
        # specifies what hostname must appear in the request's Host: header to
        # match this virtual host. For the default virtual host (this file) this
        # value is not decisive as it is used as a last resort host regardless.
        # However, you must set it for any further virtual host explicitly.
        ServerName 192.168.178.29

        ServerAdmin [email protected]
        DocumentRoot /var/www/192.168.178.29

        # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
        # error, crit, alert, emerg.
        # It is also possible to configure the loglevel for particular
        # modules, e.g.
        #LogLevel info ssl:warn
        #LogLevel info ssl:warn
        LogLevel info ssl:warn

        SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded
        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
        LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded
        ErrorLog ${APACHE_LOG_DIR}/error.8080.log
        CustomLog ${APACHE_LOG_DIR}/access.8080.log combined env=!forwarded
        CustomLog ${APACHE_LOG_DIR}/access.8080.log forwarded env=forwarded

        # For most configuration files from conf-available/, which are
        # enabled or disabled at a global level, it is possible to
        # include a line for only one particular virtual host. For example the
        # following line enables the CGI configuration for this host only
        # after it has been globally disabled with "a2disconf".

        <Directory /var/www/192.168.178.29>
                Options -Indexes +FollowSymLinks
                AllowOverride All
                Require all granted
        </Directory>
</VirtualHost>

Answer 1

HTTP/2 is usually supported by web browsers in combination with encryption so not something you would enable on the plain HTTP port, port 80. So you nginx configuration:

server {
   listen 80 http2;

should omit the http2 directive:

server {
   listen 80;

As to why currently you can't telnet to port 80: Unlike plain HTTP 1.1 which you can easily emulate by entering HTTP commands with telnet or HTTPS/1.1 which you can similarly test with for example openssl s_client -connect host:443 ; is HTTP/2 more difficult to test.

HTTP/2 is no longer a text based protocol, but rather a more efficient binary protocol so telnet doesn't provide the correct handshake to successfully establish a connection. See: https://blog.cloudflare.com/tools-for-debugging-testing-and-using-http-2/ for some examples to use instead.

---

Typically you allow Apache to listen either to all IP-addresses and interfaces on a system with Listen 8080 OR in the configuration you describe one binds Apache httpd specifically to localhost so it cannot be accessed remotely with Listen 127.0.0.1:8080. Either way rather than: VirtualHost 192.168.178.29:8080 you'd typically use <VirtualHost *:8080>.

Second, the apache VirtualHost definition you post doesn't appear to support HTTPS and I assume that when "addressing port 8080 directly in the browser" you also use a clear text HTTP URI.

Then using trying to use proxy_pass https://192.168.178.29:8080/; ; with HTTPS in your nginx reverse proxy is doomed to fail. That should also be plain HTTP: proxy_pass http://192.168.178.29:8080/; and when apache and nginx run on the same server even: proxy_pass http://127.0.0.1:8080/;

Answer 2

I'm not a big expert in this field, but I'll just assume that the problem may be that you need to enable the ssl directive for nginx.

listen 80 ssl http2;

You also need to make sure that you have enabled the ssl module for apache.

I would start looking for a problem by replacing https with http and making sure it works over this protocol.