Kubernetes frontend service latency much higher than Docker Compose-based setup

Question

I am experiencing a latency issue with my Kubernetes setup compared to a Docker Compose setup. When making a request to the frontend service on Kubernetes, it takes close to 10 seconds to respond, while the same request against a Docker Compose instance of the exact same image takes about 200ms.

Kubernetes Node Components:

• Nginx server
• API service
• Postgres
• Redis
• Web frontend service

Network Configuration:

• Using Calico's Tigera operator with the machine on subnet 192.168.88.0/24 and the node on 10.10.0.0/16
• No other customizations apart from the subnet.
• Followed the Calico quickstart guide: Calico Quickstart Guide

Additionally:

• None of the system's deployments specify resource limits.
• No Ingress resource specified. Instead, a NodePort is declared on the nginx service. However, I cannot access it externally using the hostname nginx.my-namespace.svc.cluster.local (nslookup reports no such hostname). Since this suggests some sort of DNS resolution issue, I'm wondering if this could be related to my issue?
• The latency issue is consistent even when testing inside the frontend pod with curl http://localhost....
• There is no difference in latency between static pages or pages with dynamically generated content.

I'm convinced it's not an application issue and suspect it to be related to the network.

What could be causing this significant latency in my Kubernetes setup, and what steps can I take to diagnose and resolve this issue?

• Machine: bare-metal Ubuntu 22.04 8 Core 32GiB

• Kube: registry.k8s.io/kube-apiserver:v1.28.11

• CNI: docker.io/calico/apiserver:v3.28.0

• CRI: containerd containerd.io 1.6.33

Answer 1

If you run curl http://localhost from the pod itself and have the same latency issue, it is not network related. Well, 99% of the time. It could be that localhost doesn’t actually point to 127.0.0.1. Try http://127.0.0.1 from the pod. If that sees the same latency issue, this is not network related from this perspective as no actual network is involved before the curl and the web server you are accessing. In this case some possible issues are related to either the application or some dependency within the web server or application that tries to check something on the internet or network before service the pages. So

1. Application issue
2. Issue where curl -> webserver/application -> some network resource dependency -> response, and that dependency may be the reason, in which case it is the network

To isolate the issue, first ensure you have a simple web server and the page does not have any external dependencies to operate and try that with 127.0.0.1 curl. If that has latency, it’s the web server. If it doesn’t, there is the application dependency.

But overall, the curl http://127.0.0.1 in the pod removes a network from your test between curl and the server so that’s not the problem.