Kubernetes Node NotReady (cni plugin not initialized)

Question

I have installed a K8s cluster. The Working Node has successfully joined the cluster. However, it is not ready:

a@front:~$ kubectl get nodes
NAME    STATUS     ROLES           AGE   VERSION
front   Ready      control-plane   49m   v1.27.1
wn1     NotReady   <none>          40m   v1.27.1

This is due to: NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

a@front:~$ kubectl describe node wn1
Name:               wn1
Roles:              <none>
...
Taints:             node.kubernetes.io/not-ready:NoExecute
                    node.cilium.io/agent-not-ready:NoSchedule
                    node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
...
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 24 Oct 2023 21:26:10 +0000   Tue, 24 Oct 2023 21:05:34 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 24 Oct 2023 21:26:10 +0000   Tue, 24 Oct 2023 21:05:34 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Tue, 24 Oct 2023 21:26:10 +0000   Tue, 24 Oct 2023 21:05:34 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Tue, 24 Oct 2023 21:26:10 +0000   Tue, 24 Oct 2023 21:05:34 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Indeed, the Pod is not working:

a@front:~$ kubectl get pods -o wide --all-namespaces
NAMESPACE     NAME                               READY   STATUS              RESTARTS   AGE   IP              NODE    NOMINATED NODE   READINESS GATES
kube-system   cilium-drrww                       1/1     Running             0          20m   10.10.17.2      front   <none>           <none>
kube-system   cilium-gj6h6                       0/1     Init:0/6            0          10m   10.10.17.3      wn1     <none>           <none>
kube-system   cilium-operator-788c7d7585-htnr6   0/1     ContainerCreating   0          48m   10.10.17.3      wn1     <none>           <none>
kube-system   cilium-operator-788c7d7585-trmrk   1/1     Running             0          48m   10.10.17.2      front   <none>           <none>
...
kube-system   kube-proxy-7tsfd                   0/1     ContainerCreating   0          44m   10.10.17.3      wn1     <none>           <none>
kube-system   kube-scheduler-front               1/1     Running             4          52m   10.10.17.2      front   <none>           <none>

I can delete it, so it is recreated, but the issue is still the same.

This site mentions a potential problem with CIDR, which does not seem the case:

a@front:~$ kubectl cluster-info dump | grep cidr
                            "--allocate-node-cidrs=true",
                            "--cluster-cidr=10.10.17.0/16",

What can be happening or how can I research the issue further?

---

In the Working Node I basically installed kubeadm, kubelet and kubectl and joined the cluster, but I did not installed anything related to the CNI plugin (I did that only on the CP). Kubernetes is supposed to take care of that from the CP, but the Network plugin returns error: cni plugin not initialized puzzles me.

Answer 1

Can you describe the pods and see if there's any events, just like you did with the nodes? Additionally, the copy and paste of your terminal shows the pods still getting ready - how long do you wait and what happens to them after time (minutes)?

I'm coming at it from a Calico perspective (another CNI) because that's my experience and it seems like you've done the initial steps to install Kubernetes and your nodes are up and running. Are you married to Cilium? Calico has docs for "the hard way" which sounds like what you're doing and might give more success, especially as you say "I cannot share any documentation. Replication is difficult." (I don't know if that's because you're not following documentation, or it's a secret) When installing Calico, you have to configure and apply custom resource definitions before you install the CNI - have you done something similar and can you verify that's correct? Are you absolutely certain the CIDR you're using doesn't overlap with anything else? Both Calico and Cilium have a CLI - have you installed this? Can you use that tool to get the status and then get any errors?

TLDR my suggestions:

• Describe the pods or get pod logs and look for errors/events
• Check any CDRs and ensure config is correct
• Install and use CNI CLI tools to check the status
• Try a different CNI/follow instructions to allow reproduction/debugging