Packet flow of kubernetes cilium CNI

Question

I have a kubernete cluster with cilium replacing kube-proxy. The nat iptables looks like the following:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
CILIUM_PRE_nat  all  --  anywhere             anywhere             /* cilium-feeder: CILIUM_PRE_nat */
LIMADNS    all  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
CILIUM_OUTPUT_nat  all  --  anywhere             anywhere             /* cilium-feeder: CILIUM_OUTPUT_nat */
LIMADNS    all  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
CILIUM_POST_nat  all  --  anywhere             anywhere             /* cilium-feeder: CILIUM_POST_nat */

Chain CILIUM_OUTPUT_nat (1 references)
target     prot opt source               destination         

Chain CILIUM_POST_nat (1 references)
target     prot opt source               destination         
MASQUERADE  all  --  10.96.0.0/24        !10.96.0.0/24         /* cilium masquerade non-cluster */
ACCEPT     all  --  anywhere             anywhere             mark match 0xa00/0xe00 /* exclude proxy return traffic from masquerade */
SNAT       all  -- !10.96.0.0/24        !10.96.0.0/24         /* cilium host->cluster masquerade */ to:10.96.0.186
SNAT       all  --  localhost            anywhere             /* cilium host->cluster from 127.0.0.1 masquerade */ to:10.96.0.186
SNAT       all  --  anywhere             anywhere             mark match 0xf00/0xf00 ctstate DNAT /* hairpin traffic that originated from a local pod */ to:10.96.0.186

Chain CILIUM_PRE_nat (1 references)
target     prot opt source               destination         

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination         

Chain LIMADNS (2 references)
target     prot opt source               destination

The route table is as follows:

default via 192.168.5.2 dev eth0 metric 202 
default via 192.168.64.1 dev lima0 metric 203 
10.96.0.0/24 via 10.96.0.186 dev cilium_host proto kernel src 10.96.0.186 
10.96.0.186 dev cilium_host proto kernel scope link 
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15 
192.168.64.0/24 dev lima0 proto kernel scope link src 192.168.64.2

Network addresses and interfaces:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:55:55:12:3b:2d brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.15/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5055:55ff:fe12:3b2d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: lima0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:55:55:48:66:6c brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.2/24 scope global lima0
       valid_lft forever preferred_lft forever
    inet6 fdcf:c9d7:1ae3:ce75:5055:55ff:fe48:666c/64 scope global dynamic mngtmpaddr proto kernel_ra 
       valid_lft 2591943sec preferred_lft 604743sec
    inet6 fe80::5055:55ff:fe48:666c/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8a:00:f7:e0:b0:27 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8800:f7ff:fee0:b027/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
5: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:1c:89:3d:96:31 brd ff:ff:ff:ff:ff:ff
    inet 10.96.0.186/32 scope global cilium_host
       valid_lft forever preferred_lft forever
    inet6 fe80::d41c:89ff:fe3d:9631/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
6: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ba:e7:75:cf:fc:c2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b8e7:75ff:fecf:fcc2/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
8: lxc_health@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 72:36:74:34:e8:8f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::7036:74ff:fe34:e88f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
10: lxc776c2c6e943e@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:ed:29:11:47:fb brd ff:ff:ff:ff:ff:ff link-netns cni-f029fabb-528a-d9ca-574b-592e5b9649ba
    inet6 fe80::a8ed:29ff:fe11:47fb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
26: lxc241137719f81@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9a:b2:f6:16:b9:eb brd ff:ff:ff:ff:ff:ff link-netns cni-cc7da8bc-85ef-a166-3a46-95638712a22d
    inet6 fe80::98b2:f6ff:fe16:b9eb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
28: lxcc5bafb630e13@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ba:d8:b5:63:31:5c brd ff:ff:ff:ff:ff:ff link-netns cni-a5059383-d15b-0ff0-ace5-d1ab1f761e42
    inet6 fe80::b8d8:b5ff:fe63:315c/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft foreve

When I use the nginx ingress controller to access the pod using command: curl -H 'Host: ang.local' 'http://127.0.0.1:30080', how can it reach the ingress controller which has the pod address of 10.96.0.95?

Here's my take:

the command send the http packet
the packet reaches the output chain which has no rules
evaluate the routing rules for the packet, which it should match the local rule, which will forward the packet to the lo device ingress
the packet source address is nat to 10.96.0.186

And what then? the packet is now have source address of 10.96.0.186 and destination address of 127.0.0.1, how could it reach the cluster internal ip 10.96.0.95?

Send an HTTP packet using ‘curl’ command to ‘127.0.0.1: 300080.Then the packet is routed to the ‘CILIUM_POST_nat’ chain which will change the source IP address to ‘10.96.0.186’. Then the packet will be forwarded to the pod’s IP address ‘10.96.0.95’. It may resolve your issue. — Kapil Sakhare, Commented Jun 9 at 13:07
@KapilSakhare Thanks for the reply, but the part where I did not understand happens to be the "forwarded to the pod’s IP address". As I mentioned, when the packet reaches the eth0 egress, the destination address is still 127.0.0.1. How could it be forwarded to the pod? It does not make sense. — William, Commented Jun 10 at 8:12

Stack Exchange Network

Packet flow of kubernetes cilium CNI

0

You must log in to answer this question.

Browse other questions tagged
kubernetes
cilium
kubernetes-cni
.

Hot Network Questions

Packet flow of kubernetes cilium CNI

0

You must log in to answer this question.

Browse other questions tagged kubernetesciliumkubernetes-cni.

Related

Hot Network Questions

Browse other questions tagged
kubernetes
cilium
kubernetes-cni
.