0

Environment Background: kubernetes configured 3 master and 3 worker node

Kubernetes is configured component by component following https://github.com/kelseyhightower/kubernetes-the-hard-way.

Environment variables:

INTERNAL_IP=X.X.X.X
ETCD_NAME=$(hostname -s)

etcd systemd service configuration:

cat << EOF | sudo tee /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd

[Service]
Type=notify
Environment="ETCD_UNSUPPORTED_ARCH=arm64" 
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --cert-file=/etc/etcd/kube-api-server.crt \\
  --key-file=/etc/etcd/kube-api-server.key \\
  --peer-cert-file=/etc/etcd/kube-api-server.crt \\
  --peer-key-file=/etc/etcd/kube-api-server.key \\
  --trusted-ca-file=/etc/etcd/ca.crt \\
  --peer-trusted-ca-file=/etc/etcd/ca.crt \\
  --client-cert-auth \\
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-client-urls https://${INTERNAL_IP}:2379 \\
  --advertise-client-urls https://${INTERNAL_IP}:2379 \\
  --initial-cluster-token etcd-cluster-0 \\ 
  --initial-cluster controller-0=https://10.74.66.92:2380,controller-1=https://10.74.66.147:2380,controller-2=https://10.74.66.94:2380 \\
  --initial-cluster-state new \\
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

I got stock while configuring etcd version 3.4.32. After configuring systemd is reloaded, enabled and started. System journal shows below errors:

Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"warn","ts":"2024-06-10T01:58:25.434715Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_UNSUPPORTED_ARCH=amd64"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"warn","ts":"2024-06-10T01:58:25.434877Z","caller":"embed/config.go:687","msg":"Running http and grpc server on single port. This is not recommended for production."}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.434907Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/usr/local/bin/etcd","--name","controller-0","--cert-file=/etc/etcd/kube-api-server.crt","--key-file=/etc/etcd/kube-api-server.key","--peer-cert-file=/etc/etcd/kube-api-server.crt","--peer-key-file=/etc/etcd/kube-api-server.key","--trusted-ca-file=/etc/etcd/ca.crt","--peer-trusted-ca-file=/etc/etcd/ca.crt","--client-cert-auth","--initial-advertise-peer-urls","https://10.74.66.92:2380","--listen-peer-urls","https://10.74.66.92:2380","--listen-client-urls","https://10.74.66.92:2379","--advertise-client-urls","https://10.74.66.92:2379","--initial-cluster-token","etcd-cluster-0","--initial-cluster","controller-0=https://10.74.66.92:2380,controller-1=https://10.74.66.147:2380,controller-2=https://10.74.66.94:2380","--initial-cluster-state","existing","--data-dir=/var/lib/etcd"]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.435077Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"warn","ts":"2024-06-10T01:58:25.435099Z","caller":"embed/config.go:687","msg":"Running http and grpc server on single port. This is not recommended for production."}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.43511Z","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["https://10.74.66.92:2380"]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.435137Z","caller":"embed/etcd.go:495","msg":"starting with peer TLS","tls-info":"cert = /etc/etcd/kube-api-server.crt, key = /etc/etcd/kube-api-server.key, client-cert=, client-key=, trusted-ca = /etc/etcd/ca.crt, client-cert-auth = false, crl-file = ","cipher-suites":[]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.437092Z","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["https://10.74.66.92:2379"]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.43725Z","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.14","git-sha":"bf51a53a7","go-version":"go1.21.10","go-os":"linux","go-arch":"amd64","max-cpu-set":2,"max-cpu-available":2,"member-initialized":true,"name":"controller-0","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://10.74.66.92:2380"],"listen-peer-urls":["https://10.74.66.92:2380"],"advertise-client-urls":["https://10.74.66.92:2379"],"listen-client-urls":["https://10.74.66.92:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"existing","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.439371Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"1.65883ms"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.441378Z","caller":"etcdserver/server.go:532","msg":"No snapshot found. Recovering WAL from scratch!"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.441829Z","caller":"etcdserver/raft.go:530","msg":"restarting local member","cluster-id":"25a8868c6d7efd3e","local-member-id":"6207785f427e8327","commit-index":3}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.441887Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 switched to configuration voters=()"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.442028Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 became follower at term 1"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.442077Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 6207785f427e8327 [peers: [], term: 1, commit: 3, applied: 0, lastindex: 3, lastterm: 1]"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"warn","ts":"2024-06-10T01:58:25.455618Z","caller":"auth/store.go:1241","msg":"simple token is not cryptographically signed"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.458329Z","caller":"mvcc/kvstore.go:418","msg":"kvstore restored","current-rev":1}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.462525Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.481815Z","caller":"etcdserver/server.go:867","msg":"starting etcd server","local-member-id":"6207785f427e8327","local-server-version":"3.5.14","cluster-version":"to_be_decided"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.482441Z","caller":"etcdserver/server.go:767","msg":"starting initial election tick advance","election-ticks":10}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.482623Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.482689Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.482707Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.483039Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 switched to configuration voters=(1877589518635842952)"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.483127Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"25a8868c6d7efd3e","local-member-id":"6207785f427e8327","added-peer-id":"1a0e89b824493d88","added-peer-peer-urls":["https://10.74.66.94:2380"]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.483162Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"1a0e89b824493d88"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.483204Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"6207785f427e8327","remote-peer-id":"1a0e89b824493d88"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.483724Z","caller":"v3rpc/health.go:61","msg":"grpc service status changed","service":"","status":"SERVING"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.48653Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"1a0e89b824493d88"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.486605Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"6207785f427e8327","remote-peer-id":"1a0e89b824493d88","remote-peer-urls":["https://10.74.66.94:2380"]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.486925Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6207785f427e8327","remote-peer-id":"1a0e89b824493d88"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.487773Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 switched to configuration voters=(1877589518635842952 2533603267163640262)"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.487871Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"25a8868c6d7efd3e","local-member-id":"6207785f427e8327","added-peer-id":"23292abf750ed1c6","added-peer-peer-urls":["https://10.74.66.147:2380"]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.487903Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"23292abf750ed1c6"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.487937Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"6207785f427e8327","remote-peer-id":"23292abf750ed1c6"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.488437Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"23292abf750ed1c6"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.488465Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"6207785f427e8327","remote-peer-id":"23292abf750ed1c6","remote-peer-urls":["https://10.74.66.147:2380"]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.488553Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 switched to configuration voters=(1877589518635842952 2533603267163640262 7063746891086725927)"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.488601Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"25a8868c6d7efd3e","local-member-id":"6207785f427e8327","added-peer-id":"6207785f427e8327","added-peer-peer-urls":["https://10.74.66.92:2380"]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.488638Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6207785f427e8327","remote-peer-id":"1a0e89b824493d88"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.488709Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"6207785f427e8327","remote-peer-id":"1a0e89b824493d88"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.488836Z","caller":"embed/etcd.go:727","msg":"starting with client TLS","tls-info":"cert = /etc/etcd/kube-api-server.crt, key = /etc/etcd/kube-api-server.key, client-cert=, client-key=, trusted-ca = /etc/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.488946Z","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"6207785f427e8327","initial-advertise-peer-urls":["https://10.74.66.92:2380"],"listen-peer-urls":["https://10.74.66.92:2380"],"advertise-client-urls":["https://10.74.66.92:2379"],"listen-client-urls":["https://10.74.66.92:2379"],"listen-metrics-urls":[]}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.489075Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6207785f427e8327","remote-peer-id":"23292abf750ed1c6"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.48911Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6207785f427e8327","remote-peer-id":"23292abf750ed1c6"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.489136Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6207785f427e8327","remote-peer-id":"23292abf750ed1c6"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.489314Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"6207785f427e8327","remote-peer-id":"23292abf750ed1c6"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.489465Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6207785f427e8327","remote-peer-id":"1a0e89b824493d88"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.489724Z","caller":"embed/etcd.go:598","msg":"serving peer traffic","address":"10.74.66.92:2380"}
Jun 10 01:58:25 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:25.489742Z","caller":"embed/etcd.go:570","msg":"cmux::serve","address":"10.74.66.92:2380"}
Jun 10 01:58:26 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:26.543402Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 is starting a new election at term 1"}
Jun 10 01:58:26 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:26.54354Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 became pre-candidate at term 1"}
Jun 10 01:58:26 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:26.543591Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 received MsgPreVoteResp from 6207785f427e8327 at term 1"}
Jun 10 01:58:26 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:26.54364Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 1a0e89b824493d88 at term 1"}
Jun 10 01:58:26 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:26.543669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 23292abf750ed1c6 at term 1"}
Jun 10 01:58:27 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:27.643161Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 is starting a new election at term 1"}
Jun 10 01:58:27 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:27.643259Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 became pre-candidate at term 1"}
Jun 10 01:58:27 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:27.643281Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 received MsgPreVoteResp from 6207785f427e8327 at term 1"}
Jun 10 01:58:27 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:27.643304Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 1a0e89b824493d88 at term 1"}
Jun 10 01:58:27 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:27.643318Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 23292abf750ed1c6 at term 1"}
Jun 10 01:58:28 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:28.742961Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 is starting a new election at term 1"}
Jun 10 01:58:28 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:28.743047Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 became pre-candidate at term 1"}
Jun 10 01:58:28 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:28.743072Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 received MsgPreVoteResp from 6207785f427e8327 at term 1"}
Jun 10 01:58:28 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:28.743101Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 1a0e89b824493d88 at term 1"}
Jun 10 01:58:28 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:28.743118Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 23292abf750ed1c6 at term 1"}
Jun 10 01:58:29 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:29.843507Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 is starting a new election at term 1"}
Jun 10 01:58:29 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:29.843611Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 became pre-candidate at term 1"}
Jun 10 01:58:29 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:29.843637Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 received MsgPreVoteResp from 6207785f427e8327 at term 1"}
Jun 10 01:58:29 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:29.843667Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 1a0e89b824493d88 at term 1"}
Jun 10 01:58:29 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:29.843686Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 23292abf750ed1c6 at term 1"}
Jun 10 01:58:30 controller-0 etcd[3215]: {"level":"warn","ts":"2024-06-10T01:58:30.487457Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"1a0e89b824493d88","rtt":"0s","error":"dial tcp 10.74.66.94:2380: connect: connection refused"}
Jun 10 01:58:30 controller-0 etcd[3215]: {"level":"warn","ts":"2024-06-10T01:58:30.49016Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"1a0e89b824493d88","rtt":"0s","error":"dial tcp 10.74.66.94:2380: connect: connection refused"}
Jun 10 01:58:30 controller-0 etcd[3215]: {"level":"warn","ts":"2024-06-10T01:58:30.490302Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"23292abf750ed1c6","rtt":"0s","error":"dial tcp 10.74.66.147:2380: connect: connection refused"}
Jun 10 01:58:30 controller-0 etcd[3215]: {"level":"warn","ts":"2024-06-10T01:58:30.490285Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"23292abf750ed1c6","rtt":"0s","error":"dial tcp 10.74.66.147:2380: connect: connection refused"}
Jun 10 01:58:30 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:30.943532Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 is starting a new election at term 1"}
Jun 10 01:58:30 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:30.943594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 became pre-candidate at term 1"}
Jun 10 01:58:30 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:30.94361Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 received MsgPreVoteResp from 6207785f427e8327 at term 1"}
Jun 10 01:58:30 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:30.94363Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 1a0e89b824493d88 at term 1"}
Jun 10 01:58:30 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:30.943641Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 23292abf750ed1c6 at term 1"}
Jun 10 01:58:32 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:32.043509Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 is starting a new election at term 1"}
Jun 10 01:58:32 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:32.043596Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 became pre-candidate at term 1"}
Jun 10 01:58:32 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:32.043616Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 received MsgPreVoteResp from 6207785f427e8327 at term 1"}
Jun 10 01:58:32 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:32.043684Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 1a0e89b824493d88 at term 1"}
Jun 10 01:58:32 controller-0 etcd[3215]: {"level":"info","ts":"2024-06-10T01:58:32.043699Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6207785f427e8327 [logterm: 1, index: 3] sent MsgPreVote request to 23292abf750ed1c6 at term 1"}
Jun 10 01:58:32 controller-0 etcd[3215]: {"level":"warn","ts":"2024-06-10T01:58:32.482957Z","caller":"etcdserver/server.go:2139","msg":"failed to publish local member to cluster through raft","local-member-id":"6207785f427e8327","local-member-attributes":"{Name:controller-0 ClientURLs:[https://10.74.66.92:2379]}","request-path":"/0/members/6207785f427e8327/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}

In the process of configuration and troubleshoot i have set hostname, checked network between cluster using ping, curl, telnet and dns resolved too. Configured has been replicated to 3 master too.

Improve this question

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .