How can i get cilium to pass the failing connectivity test

Question

I am trying to deploy cilium to my eks cluster, for context, this cluster is a private cluster running behind a private subnet, and routed to the internet through a NAT gateway and then an internet gateway. I have been able to follow the cilium installation guid here. my nodes are tainted and i have patched the daemonset as the documentation asked.

When i run cilium status, i can see it is ok

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet         cilium             Desired: 3, Ready: 3/3, Available: 3/3
Containers:       cilium             Running: 3
                  cilium-operator    Running: 2
Cluster Pods:     2/2 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.15.0@sha256:9cfd6a0a3a964780e73a11159f93cc363e616f7d9783608f62af6cfdf3759619: 3
                  cilium-operator    quay.io/cilium/operator-aws:v1.15.0@sha256:cf45167a8bb336c763046553c6a97c0d7f12f7e2a498dfb2340fa27832a81b3a: 2

However when i run cilium connectivity test, not all the test passes. the error is as shown below.

❌ 4/42 tests failed (30/321 actions), 13 tests skipped, 1 scenarios skipped:
Test [no-policies]:
  ❌ no-policies/pod-to-host/ping-ipv4-1: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> <NODE_IP> (<NODE_IP>:0)
  ❌ no-policies/pod-to-host/ping-ipv4-3: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> <NODE_IP> (<NODE_IP>:0)
  ❌ no-policies/pod-to-host/ping-ipv4-5: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> <NODE_IP> (<NODE_IP>:0)
  ❌ no-policies/pod-to-host/ping-ipv4-7: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> <NODE_IP> (<NODE_IP>:0)
  ❌ no-policies/pod-to-host/ping-ipv4-9: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> <NODE_IP> (<NODE_IP>:0)
  ❌ no-policies/pod-to-host/ping-ipv4-11: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> <NODE_IP> (<NODE_IP>:0)
Test [no-policies-extra]:
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-0: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-1: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-2: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> cilium-test/echo-same-node (echo-same-node:8080)
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-3: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> cilium-test/echo-same-node (echo-same-node:8080)
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-4: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-5: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-6: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> cilium-test/echo-same-node (echo-same-node:8080)
  ❌ no-policies-extra/pod-to-remote-nodeport/curl-7: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> cilium-test/echo-same-node (echo-same-node:8080)
  ❌ no-policies-extra/pod-to-local-nodeport/curl-0: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-local-nodeport/curl-1: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> cilium-test/echo-same-node (echo-same-node:8080)
  ❌ no-policies-extra/pod-to-local-nodeport/curl-2: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> cilium-test/echo-other-node (echo-other-node:8080)
  ❌ no-policies-extra/pod-to-local-nodeport/curl-3: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> cilium-test/echo-same-node (echo-same-node:8080)
Test [allow-all-except-world]:
  ❌ allow-all-except-world/pod-to-host/ping-ipv4-1: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> 18.130.173.145 (<NODE_IP>:0)
  ❌ allow-all-except-world/pod-to-host/ping-ipv4-3: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> 18.171.241.88 (<NODE_IP>:0)
  ❌ allow-all-except-world/pod-to-host/ping-ipv4-5: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> 13.40.120.114 (<NODE_IP>:0)
  ❌ allow-all-except-world/pod-to-host/ping-ipv4-7: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> 18.130.173.145 (<NODE_IP>:0)
  ❌ allow-all-except-world/pod-to-host/ping-ipv4-9: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> 18.171.241.88 (<NODE_IP>:0)
  ❌ allow-all-except-world/pod-to-host/ping-ipv4-11: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> 13.40.120.114 (<NODE_IP>:0)
Test [host-entity]:
  ❌ host-entity/pod-to-host/ping-ipv4-1: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> <NODE_IP> (<NODE_IP>:0)
  ❌ host-entity/pod-to-host/ping-ipv4-3: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> <NODE_IP> (<NODE_IP>:0)
  ❌ host-entity/pod-to-host/ping-ipv4-5: cilium-test/client-846d67868c-mpfrc (10.0.1.217) -> <NODE_IP> (<NODE_IP>:0)
  ❌ host-entity/pod-to-host/ping-ipv4-7: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> <NODE_IP> (<NODE_IP>:0)
  ❌ host-entity/pod-to-host/ping-ipv4-9: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> <NODE_IP> (<NODE_IP>:0)
  ❌ host-entity/pod-to-host/ping-ipv4-11: cilium-test/client2-865b7d7b6f-469vq (10.0.1.178) -> <NODE_IP> (<NODE_IP>:0)
connectivity test failed: 4 tests failed

Question

How can i resolve this and get cilium running.

PS I just swapped out the node ip addresses for the variable <NODE_IP> for the purpose of posting this question.

Answer 1

The solution is disable assigning public IP addresses to EKS nodes.

You can configure that in the Network Interfaces in the Launch Template that the EC2 instances are using. Set the "Auto-assign public IP" to Disable.