1

I'm running the following EKS version:

kubelet --version
Kubernetes v1.22.12-eks-ba74326

I've set the following parameter as documented here.

  "containerLogMaxSize": "100 Mi"

I then restarted the kubelet:

systemctl daemon-reload
systemctl restart kubelet

But my logs are still rotating at 10 MB.

Any ideas for how to debug or fix this?

Improve this question
6
  • I don't believe it tolerates spaces between the number and the unit; did you check the journalctl -u kubelet to see if it whined on startup?
    – mdaniel
    Commented Mar 29, 2023 at 4:47
  • @mdaniel I do not see any complaining but I do see FLAG: --container-log-max-size="10Mi", so the option is certainly being ignored or overridden somehow. I also tried removing the space between the number and the unit, but that did not do anything different.
    – merlin2011
    Commented Mar 29, 2023 at 18:54
  • Hmm, ok, then the next thing I'd check is (as you said) see if EKS's scripts are doing something underhanded; start with systemctl cat kubelet.service and chase all the files and environment mentioned. You may have some luck with grep -r container-log-max /etc but that may not be the whole story since systemd is a complicated thing
    – mdaniel
    Commented Mar 30, 2023 at 2:49
  • 1
    My current suspicion is that it's this issue, so I'm going to try changing the runtime and see if that helps: github.com/kubernetes/kubernetes/issues/…
    – merlin2011
    Commented Mar 30, 2023 at 23:24
  • @merlin2011 as you have mentioned above, can you please confirm if changing the runtime has resolved your issue.
    – Kiran Kotturi
    Commented Apr 3, 2023 at 12:55

2 Answers 2

Reset to default
0

As per the official documentation

ContainerLogMaxSize is a quantity defining the maximum size of the container log file before it is rotated. For example: "5Mi" or "256Ki". If DynamicKubeletConfig is on, when dynamically updating this field, consider that it may trigger log rotation. Default: "10Mi"

If the value mentioned in the containerLogMaxSize is not accepted then by default the log rotation is returning 10 MB of log file.Please update the value as mentioned in the above example as 100Mi (without spaces) as your requirement is 100 MB.

containerLogMaxSize: 100Mi
Improve this answer
1
  • How can I tell if it's not accepted?
    – merlin2011
    Commented Mar 29, 2023 at 17:43
0

I ran into this same issue and after some digging fixed it with the following.

Systemd was using the dropin file /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf for kubelet startup and this file pointed to a config file /var/lib/kubelet/config.yaml, this is where I made my changes.

$ sudo vi /var/lib/kubelet/config.yaml 
  added the following to the bottom of the file:  
    containerLogMaxFiles: 6  
    containerLogMaxSize: "50Mi"  

Then restarted the kubelet:
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet

Finally I used the following to check the settings, I ran this on the
kubernetes control plane server. Make sure to change NODE_SERVER_NAME to
your node server name:
$ kubectl get --raw "/api/v1/nodes/NODE_SERVER_NAME/proxy/configz"

Easier to read if you have jq installed
$ kubectl get --raw "/api/v1/nodes/NODE_SERVER_NAME/proxy/configz" | jq .
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .