I have had great success using etcdadm to automate all of those steps I just described, with the bad news being that you have to build the etcdadm binary yourself, because they don't - as of this message - attach built artifacts to their releases now you can either bring the other masters back online, joining them to that new cluster until you have a new quorum, and tear down the disposable cluster, or you can walk over the disposable etcd members, removing them from the cluster one at a time until only master1 remains in your "cluster of one".join it to the new, now healthy disposable etcd cluster.reset its state ( rm -rf /var/lib/etcd, and delete the server and peer certs unless you have used the same CA for the disposable cluster - something I highly recommend but may not be possible for a variety of reasons).create a new, 3-node "disposable" etcd cluster, restoring from that snapshot.create an etcd snapshot from the surviving etcd node.This can happen that two nodes may have hardware issue in production, from my search it look like etcd issue but how i can access etcd and remove master2 and master3, i thought to do docker ps and docker exec but docker ps is not showing etcd container. How i can recover master1 in this situation? As soon as i shut master2 i have no access to kubectl and its saying apiserver is not accessible.I shutdown master3 but dont face any issue still two master accessible and i can run kubectl and do administrations.So by applying above steps all pods are running on master 1 and 2, it removes entries from kubeadm-conf configmap and also from etcd, infact i run above steps on master2 and still one master is up and running and i can run kubectl. kubectl delete node 11.11.11.3 on master3.Run kubectl drain 11.11.11.3 on master3.So before going to productions, I am trying possible failures and did below steps I have three nodes multi-master kubernetes(1.17.3) cluster(Stacked control plane and etcd nodes), 11.11.11.1 - master1
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |