nginx proxy cannot connect to upstream

Question

I have 2 spring apps running on port 8080 and 8081. I wanted to have nginx proxy for forwarding https request for either of them so I have this nginx

$cat /etc/nginx/conf.d/spring-apps.conf

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name papaluz.com;
    return 301 https://$host$request_uri;
}

# Handle SSL for the App A (default 443 for https)
server {
    listen 443 ssl;
    server_name papaluz.com;

    ssl_certificate /etc/letsencrypt/live/papaluz.com/fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/live/papaluz.com/privkey.pem; 

    # this was suggested by chatgpt, I dont know what this is so commented it out
    #ssl_protocols TLSv1.2 TLSv1.3;
    #ssl_prefer_server_ciphers off;
    #ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';
    
    location / {
        proxy_pass http://localhost:8080/; # terminating https for just http communication with upstream server
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Authorization $http_authorization;
    }
}

# Handle SSL for the app B with custom port 9000, so giving request to `https://papaluz.com:9000`
server {
    listen 9000 ssl; 
    server_name papaluz.com;

    ssl_certificate /etc/letsencrypt/live/papaluz.com/fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/live/papaluz.com/privkey.pem;

    # same as above
    #ssl_protocols TLSv1.2 TLSv1.3;
    #ssl_prefer_server_ciphers off;
    #ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';

    location / {
        proxy_pass http://localhost:8081/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Authorization $http_authorization;
    }
}

You can see that both upstream servers are on the same domain, just different port. Now accessing the index https://papaluz.com works, however, the application frontend is using axios which when calling https://papaluz.com/some_endpoind is getting 403. Now I know that the problem is nginx related, because the server is well tested and worked before I decided to have nginx as a proxy for managing more services (so its not spring-related but having to do with the proxy itself). Here is the error.log:

2024/05/19 18:26:44 [error] 1388#1388: *1 connect() failed (111: Unknown error) while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://[::1]:8080/", host: "papaluz.com"
2024/05/19 18:26:44 [error] 1388#1388: *1 connect() failed (111: Unknown error) while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "papaluz.com"
2024/05/19 18:26:49 [error] 1388#1388: *1 no live upstreams while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com"
2024/05/19 18:28:25 [error] 1388#1388: *4 connect() failed (111: Unknown error) while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "papaluz.com"
2024/05/19 18:28:25 [error] 1388#1388: *4 connect() failed (111: Unknown error) while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://[::1]:8080/", host: "papaluz.com"
2024/05/19 18:28:30 [error] 1388#1388: *4 no live upstreams while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com"
2024/05/19 18:28:33 [error] 1388#1388: *4 no live upstreams while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com"
2024/05/19 18:29:32 [error] 1388#1388: *4 connect() failed (111: Unknown error) while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "papaluz.com"
2024/05/19 18:29:32 [error] 1388#1388: *4 connect() failed (111: Unknown error) while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://[::1]:8080/", host: "papaluz.com"
2024/05/19 18:40:03 [error] 1388#1388: *9 connect() failed (111: Unknown error) while connecting to upstream, client: 154.212.141.136, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "78.47.245.20:443"
2024/05/19 18:40:03 [error] 1388#1388: *9 connect() failed (111: Unknown error) while connecting to upstream, client: 154.212.141.136, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://[::1]:8080/", host: "78.47.245.20:443"
2024/05/19 18:42:42 [crit] 1388#1388: *25 SSL_do_handshake() failed (SSL: error:0A00006C:SSL routines::bad key share) while SSL handshaking, client: 154.212.141.136, server: 0.0.0.0:443
2024/05/19 18:51:16 [error] 1388#1388: *30 connect() failed (111: Unknown error) while connecting to upstream, client: 115.238.44.234, server: papaluz.com, request: "GET / HTTP/1.0", upstream: "http://127.0.0.1:8080/"
2024/05/19 18:51:16 [error] 1388#1388: *30 connect() failed (111: Unknown error) while connecting to upstream, client: 115.238.44.234, server: papaluz.com, request: "GET / HTTP/1.0", upstream: "http://[::1]:8080/"
2024/05/19 18:51:16 [error] 1388#1388: *34 no live upstreams while connecting to upstream, client: 43.130.32.224, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com", referrer: "http://papaluz.com"
2024/05/19 19:05:32 [error] 1388#1388: *38 connect() failed (111: Unknown error) while connecting to upstream, client: 142.111.165.220, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "www.papaluz.com"
2024/05/19 19:05:32 [error] 1388#1388: *38 connect() failed (111: Unknown error) while connecting to upstream, client: 142.111.165.220, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://[::1]:8080/", host: "www.papaluz.com"
2024/05/19 19:05:34 [error] 1388#1388: *41 no live upstreams while connecting to upstream, client: 104.171.155.196, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com"
2024/05/19 19:10:29 [error] 1388#1388: *42 connect() failed (111: Unknown error) while connecting to upstream, client: 142.111.200.113, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "www.papaluz.com"
2024/05/19 19:10:29 [error] 1388#1388: *42 connect() failed (111: Unknown error) while connecting to upstream, client: 142.111.200.113, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://[::1]:8080/", host: "www.papaluz.com"
2024/05/19 19:10:35 [error] 1388#1388: *45 no live upstreams while connecting to upstream, client: 23.230.25.156, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com"
2024/05/19 20:32:17 [error] 1388#1388: *76 connect() failed (111: Unknown error) while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET /products HTTP/1.1", upstream: "http://[::1]:8080/products", host: "papaluz.com", referrer: "https://papaluz.com/"
2024/05/19 20:32:17 [error] 1388#1388: *76 connect() failed (111: Unknown error) while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET /products HTTP/1.1", upstream: "http://127.0.0.1:8080/products", host: "papaluz.com", referrer: "https://papaluz.com/"
2024/05/19 20:32:18 [error] 1388#1388: *79 no live upstreams while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET /static/js/main.367ae53b.js.map HTTP/1.1", upstream: "http://localhost/static/js/main.367ae53b.js.map", host: "papaluz.com"
2024/05/19 20:32:20 [error] 1388#1388: *76 no live upstreams while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com"
2024/05/19 20:32:23 [error] 1388#1388: *76 no live upstreams while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com"
2024/05/19 20:32:24 [error] 1388#1388: *76 no live upstreams while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com"
2024/05/19 20:32:24 [error] 1388#1388: *76 no live upstreams while connecting to upstream, client: 195.47.49.180, server: papaluz.com, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "papaluz.com"

its just saying "unable to connect to upstream, unknown error", not really helpful, but sometimes it says SSL: error:0A00006C:SSL routines::bad key share, which is maybe why chatgpt suggested the ssl_protocols, ssl_prefer_server_ciphers and ssl_ciphers, but even when those being applied (uncommented), it gave the same errors. Now I have no idea how to fix bad key share in ttls handshake and I also dont know whether it is related to the failure of connecting to the upstream (but I suspect it does). So how to solve it?

reuqest made from button onClick:

async function handleSubmit(e) {
    e.preventDefault();
    try {
      const res = await ctx.axios.post('/sign-in', {
        email,
        password: pwd
      });
      console.log(res.data);
      setCtx(prev => ({ ...prev, accessToken: res.data }));
      resetEmail();
      setPwd('');
      navigate(from, { replace: true });
    } catch (err) {
      if (!err.response) {
        setErrMsg('No Server Response');
        console.log(err);
      } else if (err.response.status === 401 || err.response.status === 403) {
        setErrMsg('unauthorized');
      } else {
        setErrMsg('login failure');
      }
    }
  }

this is request from react frontend sent to backend. It is sending the Authorization header becuase there are request interceptors (the full code is in the github for frontend and github for backend). Now the backend is catching the header in a authFilter:

@Component
@AllArgsConstructor
public class JwtAuthFilter extends OncePerRequestFilter {
    private JwtService jwtService;
    private UserDetailsServiceImpl userDetailsService;

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws ServletException, IOException {
        try {
            String token = null;
            String email = null;
            String authHeader = request.getHeader("Authorization");
            if (authHeader != null && authHeader.startsWith("Bearer ")) {
                token = authHeader.substring(7);
                if(token.isEmpty()) {
                    return;
                }
                email = jwtService.extractEmail(token);
            }
            if (email != null) {
                UserDetails userDetails = userDetailsService.loadUserByUsername(email);
                if (jwtService.validateToken(token, userDetails)) {
                    UsernamePasswordAuthenticationToken authToken =
                            new UsernamePasswordAuthenticationToken(userDetails, null, userDetails.getAuthorities());
                    authToken.setDetails(new WebAuthenticationDetailsSource().buildDetails(request));
                    SecurityContextHolder.getContext().setAuthentication(authToken);
                }
            }
            filterChain.doFilter(request, response);
        } catch (ExpiredJwtException e) {
            response.getWriter().print(new ObjectMapper().writeValueAsString(Collections.singletonMap("message", e.getMessage())));
            response.setStatus(HttpStatus.UNAUTHORIZED.value());
        }
    }
}

again, this well worked before using nginx. But I set in the nginx config to not drop the Authorization header. Now here is SecurityConfig in relation to 403 errors:

@Configuration
public class SecurityConfig implements WebMvcConfigurer {
    @Autowired
    private JwtAuthFilter authFilter;
    @Autowired
    private JwtLogoutHandler logoutHandler;

    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        http
                .authorizeHttpRequests(req -> req
                                .requestMatchers("/static/**").permitAll()
                                .requestMatchers("/assets/**").permitAll()
                                .requestMatchers("/images/**").permitAll()
                                .requestMatchers("/", "/index.html").permitAll()
                                .requestMatchers("/sign-up").permitAll()
                                .requestMatchers("/sign-in").permitAll()
                                .requestMatchers("/email-verified").permitAll()
                                .requestMatchers("/refresh").permitAll()
                                .requestMatchers("/logout").permitAll()

                                .requestMatchers("/spreadsheet/**").hasAuthority("USER")
                                .requestMatchers("/construction-work/**").hasAuthority("USER")

                                .anyRequest().authenticated()
                )
                .csrf(AbstractHttpConfigurer::disable)
                .cors(Customizer.withDefaults())
                .addFilterBefore(authFilter, UsernamePasswordAuthenticationFilter.class)
                .sessionManagement(session -> session.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
                .authenticationProvider(authenticationProvider());
        // https://stackoverflow.com/questions/36354405/spring-security-disable-logout-redirect
        http.logout().logoutSuccessHandler(logoutHandler);
        return http.build();
    }

    @Bean
    public PasswordEncoder passwordEncoder() {
        return new BCryptPasswordEncoder();
    }

    @Bean
    UserDetailsService userDetailsService() {
        return new UserDetailsServiceImpl();
    }

    @Bean
    public AuthenticationProvider authenticationProvider() {
        DaoAuthenticationProvider authenticationProvider = new DaoAuthenticationProvider();
        authenticationProvider.setUserDetailsService(userDetailsService());
        authenticationProvider.setPasswordEncoder(passwordEncoder());
        return authenticationProvider;
    }

    @Bean
    public AuthenticationManager authenticationManager(AuthenticationConfiguration configuration) throws Exception {
        return configuration.getAuthenticationManager();
    }

    @Override
    public void addCorsMappings(CorsRegistry registry) {
        registry.addMapping("/**")
                .allowedOrigins("http://localhost:3000", "https://papaluz.com:9000")
                .allowedMethods("GET", "POST", "PUT", "DELETE")
                .allowCredentials(true);
    }
}

for more code snippets see the gihub, links above. My assumption is: I was using the app before nginx. I set up ssl from letsencrypt and everything worked fine. I decided to have another service on another port, not using different domain for it, so thats the decision for the reverse proxy and nginx. Then, I can get to index page (home page), becuase there is no authroziation request. But then when I want to login (and send the post request with Authorization header), thats when the frontend cannot connect to backend. But I suspect nginx for this.