Skip to main content

All Questions

Tagged with
Filter by
Sorted by
Tagged with
0 votes
0 answers
45 views

kubernetes etcd failed to publish local member to cluster through raft

Environment Background: kubernetes configured 3 master and 3 worker node Kubernetes is configured component by component following https://github.com/kelseyhightower/kubernetes-the-hard-way. ...
gcngov's user avatar
  • 101
0 votes
0 answers
308 views

RKE2 error- Waiting for API server to become available Waiting to retrieve kube-proxy configuration; server is not ready

Issue 1 In the high-availability RKE2 Kubernetes cluster, I have provisioned a single load balancer (1LB) and three master nodes. Initially, all components were functioning as expected. However, ...
user24266244's user avatar
0 votes
0 answers
813 views

context deadline exceeded when deleting a etcd member

I have an unhealthy etcd cluster because a member that failed to join. That member doesn't exist and etcd stuck in selecting a leader. I.e., there were a cluster with 3 nodes, one tried to join but ...
Igor Artamonov's user avatar
0 votes
1 answer
872 views

Kubernetes component status reports etcd as unhealthy

I have a 3 node kube control network setup with etcd, kube-apiserver, kube-control-manager and kube-scheduler running. When I check the status of the components it says etcd is unhealthy, but etcd ...
George's user avatar
  • 113
3 votes
2 answers
821 views

What exactly is Kubernetes API datastore?

THis page mentioned Kubernetes API datastore -- https://projectcalico.docs.tigera.io/getting-started/kubernetes/flannel/flannel -- and it seems to suggest that Kubernetes API datastore is a ...
Jing He's user avatar
  • 141
1 vote
3 answers
4k views

kubernetes: how to check pod IP range allocated to node

Every node in a kubernetes cluster has a dedicated range of IP addresses that it can allocate to the pods. How can one check what that IP range is?
Sergey Guzenkov's user avatar
4 votes
2 answers
2k views

All kube-system pods keep crashing, etcd receives sigterm

I'm trying to set up an on-premises Kubernetes cluster. For starters I used kubeadm to initialize my control plane on one node. But all kube-system pods constantly crash. I took a deep look into the ...
Be Ku's user avatar
  • 143
1 vote
2 answers
502 views

ETCD restore granularity

I'm studying some backup and restore solutions for Kubernetes and Openshift, typically with Velero. So, I'm not familiar with ETCD backups, but I would like to ask about the granularity level of etcd ...
Vphdreamer's user avatar
0 votes
0 answers
2k views

Changed IP addresses of etcd nodes. How to restart cluster?

I have a k8s cluster set up using 'hardway' a while ago. I've recently changed my network around and needed to set new IPs for the controller nodes (which are also the clustered etcd hosts). I've ...
spoovy's user avatar
  • 354
1 vote
0 answers
5k views

Error from server: etcdserver: request timed out - error after etcd backup and restore

I have done the etcd backup and then a restore on the same cluster and now I'm having these issues where I can list resources but I can't create or delete. It's a 1 master and 2 workers setup , ...
Tomas.R's user avatar
  • 121
1 vote
0 answers
1k views

ETCD snapshots causing etcdserver: leader changed

For a while now we've experienced regular errors from operations on kube API in AKS resulting in etcdserver: leader changed message. From what we've learned there is an ETCD snapshot performed every ...
Radek 'Goblin' Pieczonka's user avatar
0 votes
0 answers
730 views

Clean install of Kubernetes on Raspberry Pi doesn't work

I'm trying to do a clean install of Kubernetes 1.23.x on a cluster of four Raspberry Pis, each running the x64 version of Raspberry Pi OS, however I am running into a major snag as soon as I try and ...
soapergem's user avatar
  • 719
1 vote
2 answers
4k views

etcdctl endpoint health --> unhealty?

If I run etcdctl it hangs. If I use --debug=true I see: root@k8scp:~# kubectl exec -n kube-system -it etcd-k8scp sh sh-5.1# ETCDCTL_API=3 etcdctl --debug=true endpoint health {"level":&...
guettli's user avatar
  • 3,893
0 votes
1 answer
211 views

etcdctl: unknown command "save" for "etcdctl"

I entered the etcd container: kubectl -n kube-system exec -it etcd-k8scp -- sh The I try to backup the container like explained in the K8s docs ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot ...
guettli's user avatar
  • 3,893
0 votes
1 answer
104 views

How to detect traffic change and trigger GitLab pipeline?

We have two etcd clusters running in Kubernetes one act as primary(A) and the second one act as backup(B). We also have etcdctl make-mirror in place between these two clusters, now the problem is if ...
Sam's user avatar
  • 101
2 votes
0 answers
334 views

Kubernetes on Flatcar Linux will not start up kube API server after a reboot of the master node

Thank you for reading and taking your time to review this problem. I have a problem using my Kubernetes cluster. It is running Flatcar Linux, made by Kinvolk, recently acquired by Microsoft. I have ...
jonasclaes's user avatar
0 votes
0 answers
2k views

How to fix etcd within a kuberentes cluster?

I have a bare-metal (kubeadm) kubernetes cluster that's really unstable, and I traced it back to an etcd issue. From the etcd pod's description I get: Image: k8s.gcr.io/etcd:3.4.13-0 Liveness: ... #...
Antoine's user avatar
  • 301
1 vote
0 answers
651 views

How to spin up a ETCD docker container and restore a backup obtained from a kubernetes ETCD Cluster node?

I am trying to spin up a ETCD node in docker and restore a backup obtained from the Kubernetes Cluster node. ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --user=root:'root' snapshot save ...
Anjula Paulus's user avatar
2 votes
0 answers
806 views

Kubernetes V1.19.13 - kubeapi servers not able connecting to different etcd database

I have upgraded Kubernets cluster ( 3 master, 3 etcd servers database) from 1.18 to v1.19.13 and ETCD to 3.4.13. since than API servers are not stable, keep switching different etcd server, because of ...
sfgroups's user avatar
  • 243
1 vote
1 answer
1k views

ETCD Cluster configuration for Kubernetes: Which one should be considered?

I would like to know how to deploy a ETCD cluster for Kubernetes. It seems like there are two different documentation and I don't know which one must be considered or the impact of each of them. From ...
Mael Fosso's user avatar
0 votes
1 answer
6k views

Kubernetes: kubeadm join fails in private network

I'm trying to set up a HA Kubernetes cluster on Hetzner Cloud following this guide. I've created 6 servers, 3 control plane hosts and 3 workers. When trying to use kubeadm to join the second server to ...
mway-niels's user avatar
4 votes
1 answer
1k views

How can we modify the heartbeat synchronization time of the etcd cluster on Kubernetes

I want to modify heartbeat-interval of embedded etcd cluster running on kubernetes, I know while installation we can do with kubeadm extraargs but how to do in running cluster? Before modifying I want ...
ImranRazaKhan's user avatar
0 votes
2 answers
4k views

ETCD database cluster certificate renewal for Kubernets external database setup

I have deployed 3 node external ETCD database (etcdctl version: 3.4.7) cluster for my Kubernetes v1.18.6 cluster using etcdadm tool. my certificate is expring in couple of months. I believe kubeadm ...
sfgroups's user avatar
  • 243
1 vote
3 answers
13k views

Kubernetes OCI runtime exec failed - starting container process caused "exec: \"etcdctl\": executable file not found in $PATH": unknown

Background Created a fresh Kubernetes cluster using kubeadm init --config /home/kube/kubeadmn-config.yaml --upload-certs and then joining the 2nd control plane node by running the below. kubeadm join ...
PieDev's user avatar
  • 71
0 votes
1 answer
886 views

Kubernetes API: Compare and update config map key

Etcd has a concept of Atomic Compare-and-Update by comparing the key's value before executing an update. I'd like to use this feature for updating a ConfigMap in my Kubernetes cluster. I'd like to ...
xpepermint's user avatar
0 votes
2 answers
150 views

Upgrade multi etcd cluster running inside docker container

Currently my k8s cluster is on v1.16.x and I want to upgrade it to v1.17.x for which ETCD has to be upgraded to 3.4 (currently 3.3). My setup is bit complex as I'm running ETCD outside the master ...
jagatjyoti's user avatar
0 votes
0 answers
1k views

Kubernetes etcd panic: store.keyindex: put with unexpected smaller revision

After an ungraceful shutdown of our Kubernetes cluster, the API server didn't start working. After some investigation, I found the following errors in each etcd member on each node. How can I recover ...
Ahmad Ahmadi's user avatar
2 votes
2 answers
5k views

Kubernetes - kubeadm join - Connection refused after new control plane joined

Issue I am trying to join a second control-plane node to a K8S cluster. The first node has a IP of 10.1.50.4 where the second node has a IP of 10.1.50.5. There is a load balancer Virtual IP for the ...
Pie's user avatar
  • 320
3 votes
0 answers
1k views

k3s without HA: how to switch master node?

Rancher documentation on k3s is quite nice and its HA support (both with external DB or embedded etcd) look nice, but I don't want/need an HA setup. In case my master node fails, I don't mind having ...
Filipe Pina's user avatar
0 votes
1 answer
3k views

Error response from daemon: {“message”:“No such container: kubelet”}

When adding a new node to a Kubernetes cluster I end up with this error : + docker start kubelet Error response from daemon: {"message":"No such container: kubelet"} Error: failed ...
TheoV's user avatar
  • 1
0 votes
0 answers
630 views

ETCD client connection to etcd servers from k8s api server issue

We expect the following issue with etcd cluster. We generated certificate with the following configuration: { "CN": "client", "hosts": [""], "...
Andrew Striletskyi's user avatar
1 vote
1 answer
4k views

TLS handshake issues with etcd

We are using external etcd cluster for k8s cluster. We connected master to this etcd server but receive "tls: first record does not look like a TLS handshake" How to fix this issues? (for ...
Andrew Striletskyi's user avatar
1 vote
2 answers
5k views

How to recover from master failure in kubernetes

I have three nodes multi-master kubernetes(1.17.3) cluster(Stacked control plane and etcd nodes), 11.11.11.1 - master1 11.11.11.2 - master2 11.11.11.3 - master3 So before going to productions, I am ...
ImranRazaKhan's user avatar
2 votes
1 answer
457 views

Why do kubernetes kube-api server need etcd-keyfile and kubelet-client-key

As I understand kube-api server acts as a client when communicating with ETCD and Kubelet. Both ETCD and Kubelet act as servers for kube-api. With secure environment (two way SSL authentication), kube-...
srinu259's user avatar
  • 123
0 votes
1 answer
2k views

Include own/external etcd cluster in kubeadm init

I have a running etcd cluster with five members (Own etcd cluster for Kubernetes). How can I include this in the kubeadm init command? My idea is that I generate the configuration, edit it manually ...
uav's user avatar
  • 564
1 vote
2 answers
564 views

Own etcd cluster for Kubernetes

I want to build my own Kubernetes cluster across two locations (300 km distance) and integrate it into GitLab. Let me list my ideas. My question is if I have a mistake in my thinking somewhere and ...
uav's user avatar
  • 564
0 votes
1 answer
795 views

Installing kubernetes with external etcd - calico problems

I faced multiple problems during installation of k8s multimaster cluster with external etcd. I did it before twice, on other sites, successfully, but this time I need help. calico was installed from ...
Paul K.'s user avatar
  • 115
0 votes
1 answer
978 views

kube-calico bird: Mesh_172_26_178_195: Socket error: bind: Address not available

Issue appears when kube-calico service on node trying to connect master: Dec 17 12:03:41 a docker[27052]: bird: Mesh_47_105_189_58: Socket error: bind: Address not available Dec 17 12:03:41 a docker[...
aleng1106's user avatar
3 votes
1 answer
5k views

missing '=' etcd when defining service file

I'm struggling while following Kelsey Hightower's "Kubernetes the Hard Way" tutorial. I've gone off script, because I'm trying to bootstrap k8s on a local server. I've got the point where I'm ...
Baron's user avatar
  • 61
0 votes
1 answer
652 views

Is there a way to determine to which etcd server the Kubernetes-apiserver is communicating with?

In the following scenario, is there a way to determine to which etcd server the Kubernetes-apiserver is communicating with?? Let's say we have 3 Master nodes with an external load balancer and 3 etcd'...
Venkata Surya Lolla's user avatar
0 votes
1 answer
231 views

Does etcd resources are encrypted on gke master nodes?

I do not find any information on etcd database managed by google on kubernetes master nodes. I see how resources (pod/nodes/...) communicates, but I don't know if the data are encrypted before being ...
Frédéric Ferchaud's user avatar
13 votes
1 answer
2k views

Graceful shutdown of Kubernetes cluster

We have a 4 node Kubernetes cluster, installed using Juju 2.0 and the official canonical Kubernetes charm. The sole purpose of this very local and private installation is development and testing. One ...
Kamil Tamiola's user avatar
1 vote
1 answer
4k views

kubelet not able to update node status

kubernetes node status reports as below. root@csp-nso-server78:~# kubectl get nodes 10.213.5.78 NAME STATUS AGE 10.213.5.78 NotReady 1h The kubelet logs show below error E0812 11:...
Akilesh's user avatar
  • 151
0 votes
1 answer
304 views

etcd & kubernetes: No connection possible to etcd instance

currently I am following the k8s guide "from scratch" (found here: LINK, up to this point everything seems pretty clear, though I start without doing all that certificate stuff. I started kubelet on ...
Jannick's user avatar
  • 131
0 votes
1 answer
852 views

OpenShift Origin asks for credentials

I had a cluster OpenShift with a master and a node. Today, it doesn't work anymore, OpenShift continously asks me the credentials, the following is the command and the output: [root@openshift-master ~...
DarkSkull's user avatar
  • 161
1 vote
3 answers
2k views

Kubernetes, flannel and exposing services

I have a kubernetes setup running nicely, but I can't seem to expose services externally. I'm thinking my networking is not set up correctly: kubernetes services addresses: --service-cluster-ip-range=...
jaxxstorm's user avatar
  • 626
1 vote
1 answer
276 views

Kubernetes service etcd références

If someone with quite good kubernetes's services understanding could confirm me that once you require a new service to be created kubernetes push a reference of it on the etcd database and what is the ...
Dr I's user avatar
  • 975
3 votes
1 answer
2k views

PostgreSQL failover with Kubernetes

I've got master slave replication working with Kubernetes, but would now like to implement failover. I have pods running with the service=postgresql and role=master or role=slave roles. When the ...
Vincent den Boer's user avatar