How do Fargate pods locate CoreDNS?

Question

How do Kubernetes serverless pods (EKS Fargate) know the IP address to access the cluster's DNS server (the CoreDNS service deployment)?

I recently updated a Kubernetes cluster to set up serverless hosting for the Cluster-Autoscaler (and/or Karpenter) deployment. This was to eliminate the deadlock hazard, of this controller potentially being rendered unschedulable when it is evicted whilst rolling the worker node fleet for security updates, which can prevent it from instantiating the new replacement worker nodes that are needed for itself and other cluster workloads. (Note there is no EKS add-on to instead run this critical controller within the managed control plane or master nodes.) To accomplish this I terraformed an aws_eks_fargate_profile resource (with the typical associated IAM role which permits the AWS Fargate service to pull container images from AWS ECR registries). This was initially unsuccessful until we resolved two issues: firstly, the control plane needs to be configured to associate various groups (ClusterRoles) of k8s RBAC permissions to that new IAM role. (This can be done either through the new EKS "access entries" AWS API if enabled, or via the old "aws-auth" ConfigMap. Apparently AWS amends the latter automatically, but our terraform was reverting the change.) Secondly, we needed to update our internal network firewalls (AWS Security Groups) to permit serverless pods to communicate with the CoreDNS service deployment (e.g., have the worker node SG accept connections from the cluster SG, which the fargate pod inherits membership of). This was required for the process contained in the pod to be able to resolve cluster-external DNS addresses (e.g. sts.amazon.com, to be able to exchange the pod's k8s service account OIDC tokens for temporary IAM credentials that this controller needed for managing the worker node fleet through cloud APIs). Note that this controller process does not otherwise make direct use of inter-pod networking (which was expected not to be functioning because "serverless" virtual machines lack the k8s daemonsets for supporting such functions, such as kube-proxy; this controller process ostensibly only needs to access API servers belonging to the cloud and the control plane).

How does the Fargate pod know to use the IP address belonging to the CoreDNS (kube-dns) Cluster-IP service, for resolving DNS queries? (I presume that Cluster-IPs are assigned randomly, and not all clusters use CoreDNS.) Is it baked in at some level or does it autodetect somehow? Can it use an external DNS server instead?

Answer 1

There is a convention that Kubernetes clusters use the .10 IP address for their DNS service. Hence:

• The ClusterIP service (named kube-dns) belonging to CoreDNS must statically specify this IP address. Note that since ClusterIP services default to using dynamically assigned IPs, there is a potential collision risk if other services get declared first.

• The kubelet on each worker node is also explicitly configured to use this address as its default DNS server. (The kubelet on the Fargate virtual worker node is apparently also configured likewise.)

In AWS there is a default public DNS server accessible via various reserved IP addresses, particularly the .2 IP address in every VPC. The worker nodes (like all EC2 instances) are configured at the system level with this DNS server address by DHCP autodiscovery.

When an arbitrary process attempts to look up a domain name, it will invariably use standard C library routines for which the resolving DNS server IP address is configured in /etc/resolv.conf. When kubelet launches a pod container it populates the /etc/resolv.conf file inside that container. Thus it passes on to the pod the cluster DNS address that kubelet was configured with (usually .10), with an exception that if the pod spec explicitly contains dnsPolicy: Default (as it does in the CoreDNS deployment) then kubelet will instead pass on the DNS address that is configured on the host node's system (usually .2).

Thus, normal pods will default to using the cluster's CoreDNS service for DNS name resolution, and if the requested domain name is external to the cluster then CoreDNS will forward that request to the VPC's default DNS server. This is why the Fargate-hosted pod tries to contact an internal cluster service to resolve even an internet DNS address (such as an AWS API public endpoint). Note also that the ClusterIP is not a physical network interface, so the traffic will go via the node (and possibly a NAT?) and so require permissive networking policies between Fargate and the worker nodes.