| Előző változat mindkét oldalonElőző változatKövetkező változat | Előző változat |
| kubernetes:mentes-helyreallitas [2026/07/09 05:39] – riba.zoltan | kubernetes:mentes-helyreallitas [2026/07/09 18:57] (aktuális) – riba.zoltan |
|---|
| ====== Kubernetes mentés, helyreállítás ====== | ====== Kubernetes mentés, helyreállítás ====== |
| | |
| | |Dokumentum verzió|1.00| |
| | |Dokumentum kelte|2026.07.09| |
| | |Kubernetes verzió|1.34 - 1.36| |
| |
| ===== Control plane ===== | ===== Control plane ===== |
| |
| ==== Mentés ==== | A control plane gépek helyreállítása előtt győződjünk meg arról, hogy a mentések rendelkezésre állnak. A gépek konfigurációját tartalmazó mentés csak azon a gépen állítható helyre, ahol készült! |
| |
| === ETCD mentés === | ==== Konfiguráció mentése ==== |
| |
| === K8S konfiguráció mentés === | A mentés az alábbi állományokat tartalmazza: |
| |
| ==== Egy control plane kiesés ==== | * /etc/cni |
| | * /etc/containerd |
| | * /etc/kubernetes/manifests/*.yaml |
| | * /etc/kubernetes/pki |
| | * /etc/kubernetes/*.conf |
| | * /root/kubeadm-config.yaml |
| | * /var/lib/kubelet/pki |
| | * /var/lib/kubelet/config.yaml |
| | * /var/lib/kubelet/instance-config.yaml |
| | * /var/lib/kubelet/kubeadm-flags.env |
| |
| ==== Kettő control plane kiesés ==== | A ''/var/lib/kubelet/kubeadm-flags.env'' fájlban ''kubernetes v1.35'' verziótól nincs inicializálási paraméter, ezért kihagyható a mentésből. |
| | |
| ==== Három control plane kiesés ==== | |
| | |
| | |
| ===== Worker ===== | |
| |
| | A mentést ''root'' felhasználóként kell futtatni. az alábbi példa napi mentést készít a helyi gép /var/backup alkönvtárába. |
| |
| | <code> |
| | # mkdir -p /var/backup && tar -czvf /var/backup/k8s-config-$(hostname)-$(date +'%Y%m%d').tar.gz /etc/cni /etc/containerd /etc/kubernetes/manifests/*.yaml /etc/kubernetes/pki /etc/kubernetes/*.conf /root/kubeadm-config.yaml /var/lib/kubelet/pki /var/lib/kubelet/config.yaml /var/lib/kubelet/instance-config.yaml /var/lib/kubeadm-flags.env |
| | </code> |
| |
| | === ETCD mentés === |
| |
| | Az ETCD adatbázisáról egyetlen control plane gépen kell snapshot-ot készíteni. A snapshot ''root'' felhasználóként az ''etcdctl'' utasítással készíthető el. A snapshot készítése után visszaállítási tesztet kell végezni. Az ETCD 3.6-os verziójától kezdve a helyreállítási műveleteket az ''etcdutl'' utasítással tudjuk végrehajtani. |
| | |
| | |
| ===== ETCD környezet helyreállítása mentésből ===== | |
| | |
| A környezet helyreállítása előtt győződjünk meg arról, hogy minden control-plane gépen telepítve van az etcdctl és az etcdutl program. | |
| |
| <code> | <code> |
| </code> | </code> |
| |
| Megjegyzés: 3.5-ös ''etcdctl'' verzió az utolsó olyan változat, amellyel a helyreállítás elvégezhető. 3.6-os verziótól a helyreállítási műveleteket az ''etcdutl'' utasítással kell elvégezni. | ETCD snapshot készítése. |
| |
| ==== Mentés állapotának ellenőrzése ==== | <code> |
| | # mkdir -p /var/backup/ && etcdctl snapshot save /var/backup/etcd-snapshot-$(hostname)-$(date +'%Y%m%d').db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key |
| | ... |
| | Snapshot saved at /var/backup/etcd-snapshot-HOSTNAME-YYYYMMDD.db |
| | Server version 3.6.0 |
| | </code> |
| |
| === Állapot gyors ellenőrzése === | Snapshot állapot ellenőrzése. |
| |
| <code> | <code> |
| # etcdutl snapshot status /root/etcd-snapshot-20260606.db -w table | # etcdutl snapshot status /var/backup/etcd-snapshot-HOSTNAME-YYYYMMDD.db -w table |
| +----------+----------+------------+------------+---------+ | +----------+----------+------------+------------+---------+ |
| | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION | | | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION | |
| +----------+----------+------------+------------+---------+ | +----------+----------+------------+------------+---------+ |
| | e1ac0dbb | 17572986 | 749 | 24 MB | 3.6.0 | | | aac67e22 | 18385870 | 516 | 24 MB | 3.6.0 | |
| +----------+----------+------------+------------+---------+ | +----------+----------+------------+------------+---------+ |
| </code> | </code> |
| |
| === Helyreállítási teszt === | Egy átmeneti könyvtárban helyreállítási tesztet futtatva győződhetünk meg a snapshot állapotáról. |
| |
| Győződjünk meg a mentés helyreállíthatóságáról. A mentést valamelyik control-plane gépen egy átmeneti könyvtárban állítsuk helyre. | <code> |
| | # rm -rf /tmp/etcd-restore-test-$(date +'%Y%m%d') && etcdutl snapshot restore /var/backup/etcd-snapshot-$(hostname)-$(date +'%Y%m%d').db --data-dir=/tmp/etcd-restore-test-$(date +'%Y%m%d') |
| | |
| | # [ $? -eq 0 ] && echo "OK - Sikeres helyreállítás" || echo "FAILED - Sikertelen helyreállítás" |
| | OK - Sikeres helyreállítás |
| | |
| | # rm -rf /tmp/etcd-restore-test-$(date +'%Y%m%d') |
| | </code> |
| | |
| | ==== Állapot ellenőrzése ==== |
| | |
| | Az alábbi utasításokkal ellenőrizhető a klaszter állapota. |
| |
| <code> | <code> |
| # rm -rf /tmp/etcd-restore-test | # kubectl get nodes -l node-role.kubernetes.io/control-plane |
| etcdutl snapshot restore /root/etcd-snapshot-20260606.db --data-dir=/tmp/etcd-restore-test | NAME STATUS ROLES AGE VERSION |
| 2026-07-06T08:54:44+02:00 info snapshot/v3_snapshot.go:305 restoring snapshot {"path": "/root/etcd-snapshot-20260606.db", "wal-dir": "/tmp/etcd-restore-test/member/wal", "data-dir": "/tmp/etcd-restore-test", "snap-dir": "/tmp/etcd-restore-test/member/snap", "initial-memory-map-size": 10737418240} | kube01 Ready control-plane 66d v1.36.2 |
| 2026-07-06T08:54:44+02:00 info bbolt backend/backend.go:203 Opening db file (/tmp/etcd-restore-test/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc000304008} | kube02 Ready control-plane 66d v1.36.2 |
| 2026-07-06T08:54:44+02:00 info bbolt bbolt@v1.4.3/db.go:322 Opening bbolt db (/tmp/etcd-restore-test/member/snap/db) successfully | kube03 Ready control-plane 66d v1.36.2 |
| 2026-07-06T08:54:44+02:00 info schema/membership.go:138 Trimming membership information from the backend... | |
| 2026-07-06T08:54:44+02:00 info bbolt backend/backend.go:203 Opening db file (/tmp/etcd-restore-test/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc0000621e0} | |
| 2026-07-06T08:54:44+02:00 info bbolt bbolt@v1.4.3/db.go:322 Opening bbolt db (/tmp/etcd-restore-test/member/snap/db) successfully | |
| 2026-07-06T08:54:44+02:00 info membership/cluster.go:424 added member {"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"], "added-peer-is-learner": false} | |
| 2026-07-06T08:54:45+02:00 info bbolt backend/backend.go:203 Opening db file (/tmp/etcd-restore-test/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc000062228} | |
| 2026-07-06T08:54:45+02:00 info bbolt bbolt@v1.4.3/db.go:322 Opening bbolt db (/tmp/etcd-restore-test/member/snap/db) successfully | |
| 2026-07-06T08:54:45+02:00 info snapshot/v3_snapshot.go:333 restored snapshot {"path": "/root/etcd-snapshot-20260606.db", "wal-dir": "/tmp/etcd-restore-test/member/wal", "data-dir": "/tmp/etcd-restore-test", "snap-dir": "/tmp/etcd-restore-test/member/snap", "initial-memory-map-size": 10737418240} | |
| </code> | </code> |
| |
| Amennyiben sikeresen lefut a helyreállítás, akkor a teszt sikeres. | Mindhárom node estében érdemes ellenőrizni a működési feltételeket. |
| |
| ==== Jelenlegi állapot mentése ==== | <code> |
| | # for node in kube01 kube02 kube03 ; do echo "### ${node} állapot ###" ; kubectl describe node ${node} | egrep '(Ready|MemoryPressure|DiskPressure|PIDPressure|NetworkUnavailable)' | awk '{print $1": "$2}' ; echo ; done |
| | ### kube01 állapot ### |
| | NetworkUnavailable: False |
| | MemoryPressure: False |
| | DiskPressure: False |
| | PIDPressure: False |
| | Ready: True |
| |
| Kubernetes objektumok állapotának mentése egy control-plane gépen. Erre a helyreállítás után az objektumok állapotának ellenőrzésénél lesz szükség. | ### kube02 állapot ### |
| | NetworkUnavailable: False |
| | MemoryPressure: False |
| | DiskPressure: False |
| | PIDPressure: False |
| | Ready: True |
| | |
| | ### kube03 állapot ### |
| | NetworkUnavailable: False |
| | MemoryPressure: False |
| | DiskPressure: False |
| | PIDPressure: False |
| | Ready: True |
| | </code> |
| | |
| | Ellenőrizzük a POD-ok állapotát. |
| |
| <code> | <code> |
| # mkdir -p /root/k8s-backup | # kubectl -n kube-system get pods |
| # kubectl get nodes -o wide > /root/k8s-backup/nodes.before.txt | NAME READY STATUS RESTARTS AGE |
| # kubectl get pods -A -o wide > /root/k8s-backup/pods.before.txt | coredns-589f44dc88-5ppfq 1/1 Running 7 (46h ago) 66d |
| # kubectl get ns > /root/k8s-backup/namespaces.before.txt | coredns-589f44dc88-nmvss 1/1 Running 7 (46h ago) 66d |
| # kubectl get cm -A > /root/k8s-backup/configmaps.before.txt | etcd-kube01 1/1 Running 14 (47h ago) 66d |
| | etcd-kube02 1/1 Running 2 (46h ago) 21d |
| | etcd-kube03 1/1 Running 2 (46h ago) 21d |
| | kube-apiserver-kube01 1/1 Running 31 (47h ago) 21d |
| | kube-apiserver-kube02 1/1 Running 2 (46h ago) 21d |
| | kube-apiserver-kube03 1/1 Running 5 (46h ago) 21d |
| | kube-controller-manager-kube01 1/1 Running 5 (47h ago) 21d |
| | kube-controller-manager-kube02 1/1 Running 7 (46h ago) 21d |
| | kube-controller-manager-kube03 1/1 Running 4 (46h ago) 21d |
| | kube-proxy-5v48r 1/1 Running 2 (46h ago) 21d |
| | kube-proxy-hj68x 1/1 Running 2 (47h ago) 21d |
| | kube-proxy-xrk5x 1/1 Running 2 (46h ago) 21d |
| | kube-scheduler-kube01 1/1 Running 7 (47h ago) 21d |
| | kube-scheduler-kube02 1/1 Running 4 (46h ago) 21d |
| | kube-scheduler-kube03 1/1 Running 5 (46h ago) 21d |
| | metrics-server-55bf4495db-tsgtq 1/1 Running 6 (46h ago) 26d |
| </code> | </code> |
| |
| Szolgáltatások tiltása és leállítása az összes control-plane gépen. | Áttekintő ellenőrzés. |
| | |
| | <code> |
| | # kubectl get --raw='/readyz?verbose' |
| | [+]ping ok |
| | [+]log ok |
| | [+]etcd ok |
| | [+]etcd-readiness ok |
| | [+]informer-sync ok |
| | [+]poststarthook/start-apiserver-admission-initializer ok |
| | [+]poststarthook/generic-apiserver-start-informers ok |
| | [+]poststarthook/priority-and-fairness-config-consumer ok |
| | [+]poststarthook/priority-and-fairness-filter ok |
| | [+]poststarthook/storage-object-count-tracker-hook ok |
| | [+]poststarthook/start-apiextensions-informers ok |
| | [+]poststarthook/start-apiextensions-controllers ok |
| | [+]poststarthook/crd-informer-synced ok |
| | [+]poststarthook/start-system-namespaces-controller ok |
| | [+]poststarthook/peer-endpoint-reconciler-controller ok |
| | [+]poststarthook/start-cluster-authentication-info-controller ok |
| | [+]poststarthook/start-kube-apiserver-identity-lease-controller ok |
| | [+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok |
| | [+]poststarthook/storage-readiness ok |
| | [+]poststarthook/start-legacy-token-tracking-controller ok |
| | [+]poststarthook/start-service-ip-repair-controllers ok |
| | [+]poststarthook/rbac/bootstrap-roles ok |
| | [+]poststarthook/scheduling/bootstrap-system-priority-classes ok |
| | [+]poststarthook/priority-and-fairness-config-producer ok |
| | [+]poststarthook/bootstrap-controller ok |
| | [+]poststarthook/start-kubernetes-service-cidr-controller ok |
| | [+]poststarthook/aggregator-reload-proxy-client-cert ok |
| | [+]poststarthook/start-kube-aggregator-informers ok |
| | [+]poststarthook/apiservice-status-local-available-controller ok |
| | [+]poststarthook/apiservice-status-remote-available-controller ok |
| | [+]poststarthook/apiservice-registration-controller ok |
| | [+]poststarthook/apiservice-discovery-controller ok |
| | [+]poststarthook/kube-apiserver-autoregistration ok |
| | [+]autoregister-completion ok |
| | [+]poststarthook/apiservice-openapi-controller ok |
| | [+]poststarthook/apiservice-openapiv3-controller ok |
| | [+]shutdown ok |
| | readyz check passed |
| | </code> |
| | |
| | ETCD tagok listája. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list -w table |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | 1546a2d2ac61bd8d | started | kube02 | https://192.168.110.162:2380 | https://192.168.110.162:2379 | false | |
| | | 18e2fb4d02a30b2e | started | kube03 | https://192.168.110.163:2380 | https://192.168.110.163:2379 | false | |
| | | 20e2ce3086b2e277 | started | kube01 | https://192.168.110.161:2380 | https://192.168.110.161:2379 | false | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | </code> |
| | |
| | ETCD endpoint állapot. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | https://kube01:2379 | 20e2ce3086b2e277 | 3.6.8 | 3.6.0 | 24 MB | 7.2 MB | 71% | 2.1 GB | false | false | 4 | 6990447 | 6990447 | | | false | |
| | | https://kube02:2379 | 1546a2d2ac61bd8d | 3.6.8 | 3.6.0 | 24 MB | 7.2 MB | 71% | 2.1 GB | true | false | 4 | 6990447 | 6990447 | | | false | |
| | | https://kube03:2379 | 18e2fb4d02a30b2e | 3.6.8 | 3.6.0 | 24 MB | 7.2 MB | 71% | 2.1 GB | false | false | 4 | 6990447 | 6990447 | | | false | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | </code> |
| | |
| | Tanúsítványok érvényességének vizsgálata. |
| | |
| | <code> |
| | # kubeadm certs check-expiration |
| | [check-expiration] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... |
| | [check-expiration] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it. |
| | |
| | CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED |
| | admin.conf Feb 15, 2126 06:54 UTC 99y ca no |
| | apiserver Feb 15, 2126 06:54 UTC 99y ca no |
| | apiserver-etcd-client Feb 15, 2126 06:54 UTC 99y etcd-ca no |
| | apiserver-kubelet-client Feb 15, 2126 06:54 UTC 99y ca no |
| | controller-manager.conf Feb 15, 2126 06:54 UTC 99y ca no |
| | etcd-healthcheck-client Feb 15, 2126 06:54 UTC 99y etcd-ca no |
| | etcd-peer Feb 15, 2126 06:54 UTC 99y etcd-ca no |
| | etcd-server Feb 15, 2126 06:54 UTC 99y etcd-ca no |
| | front-proxy-client Feb 15, 2126 06:54 UTC 99y front-proxy-ca no |
| | scheduler.conf Feb 15, 2126 06:54 UTC 99y ca no |
| | super-admin.conf Feb 15, 2126 06:54 UTC 99y ca no |
| | |
| | CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED |
| | ca Sep 01, 2225 09:27 UTC 199y no |
| | etcd-ca Sep 01, 2225 09:27 UTC 199y no |
| | front-proxy-ca Sep 01, 2225 09:27 UTC 199y no |
| | </code> |
| | |
| | Amennyiben a fenti ellenőrzések sikeresek, akkor a klaszter control plane gépei egészségesek. Nincs további teendő. |
| | |
| | ==== Egy control plane kiesése ==== |
| | |
| | Egy control plane gép kiesését a klaszter probléma nélkül elviseli. Amennyiben a kiesés az ETCD leader-t érinti, akkor a két megmaradt tagból az egyik átveszi a leader szerepet. |
| | |
| | A leader szerepkörű ETCD tagot az alábbi utasítással kaphatjuk meg. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w json | jq -r '.[] | select(.Status.leader == .Status.header.member_id) | .Endpoint' |
| | |
| | https://kube02:2379 |
| | </code> |
| | |
| | A leader kiesésekor az ETCD ellenőrző utasítás nem tud kapcsolódni a korábbi leader-hez. A két működő tag új leader-t választ. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w json | jq -r '.[] | select(.Status.leader == .Status.header.member_id) | .Endpoint' |
| | {"level":"warn","ts":"2026-07-09T09:24:07.518541+0200","logger":"etcd-client","caller":"v3@v3.6.12/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002103c0/kube02:2379","peer":"Peer{Addr: <nil>, LocalAddr: <nil>, AuthInfo: <nil>}","method":"/etcdserverpb.Maintenance/Status","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial tcp 192.168.110.162:2379: connect: connection refused\""} |
| | Failed to get the status of endpoint https://kube02:2379 (context deadline exceeded) |
| | |
| | https://kube01:2379 |
| | </code> |
| | |
| | A gépek listázásakor egy idő után a hiányzó gép ''NotReady'' állapotba kerül. |
| | |
| | <code> |
| | # kubectl get nodes -l node-role.kubernetes.io/control-plane |
| | NAME STATUS ROLES AGE VERSION |
| | kube01 Ready control-plane 66d v1.36.2 |
| | kube02 NotReady control-plane 66d v1.36.2 |
| | kube03 Ready control-plane 66d v1.36.2 |
| | </code> |
| | |
| | ETCD endpoint állapot ellenőrzésekor hiányzik a kiesett tag. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table |
| | {"level":"warn","ts":"2026-07-09T09:34:48.926554+0200","logger":"etcd-client","caller":"v3@v3.6.12/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00027f860/kube02:2379","peer":"Peer{Addr: <nil>, LocalAddr: <nil>, AuthInfo: <nil>}","method":"/etcdserverpb.Maintenance/Status","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded while waiting for connections to become ready"} |
| | Failed to get the status of endpoint https://kube02:2379 (context deadline exceeded) |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | https://kube01:2379 | 20e2ce3086b2e277 | 3.6.8 | 3.6.0 | 24 MB | 8.0 MB | 68% | 2.1 GB | true | false | 5 | 6996663 | 6996663 | | | false | |
| | | https://kube03:2379 | 18e2fb4d02a30b2e | 3.6.8 | 3.6.0 | 24 MB | 8.0 MB | 67% | 2.1 GB | false | false | 5 | 6996681 | 6996678 | | | false | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | </code> |
| | |
| | === Rövid idejű kiesés === |
| | |
| | Rövid ideig tartó kiesés esetén a hiba megszűnése után érdemes ellenőrizni az állapotokat. |
| | |
| | Gépek listája. |
| | |
| | <code> |
| | # kubectl get nodes -l node-role.kubernetes.io/control-plane |
| | NAME STATUS ROLES AGE VERSION |
| | kube01 Ready control-plane 66d v1.36.2 |
| | kube02 Ready control-plane 66d v1.36.2 |
| | kube03 Ready control-plane 66d v1.36.2 |
| | </code> |
| | |
| | ETCD endpoint állapot. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | https://kube01:2379 | 20e2ce3086b2e277 | 3.6.8 | 3.6.0 | 24 MB | 8.4 MB | 66% | 2.1 GB | true | false | 5 | 6996862 | 6996862 | | | false | |
| | | https://kube02:2379 | 1546a2d2ac61bd8d | 3.6.8 | 3.6.0 | 24 MB | 8.0 MB | 68% | 2.1 GB | false | false | 5 | 6996863 | 6996861 | | | false | |
| | | https://kube03:2379 | 18e2fb4d02a30b2e | 3.6.8 | 3.6.0 | 24 MB | 8.3 MB | 66% | 2.1 GB | false | false | 5 | 6996864 | 6996863 | | | false | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | </code> |
| | |
| | Újabb ellenőrzéssel meggyőződhetünk a RAFT indexek állapotáról. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | https://kube01:2379 | 20e2ce3086b2e277 | 3.6.8 | 3.6.0 | 24 MB | 9.8 MB | 60% | 2.1 GB | true | false | 5 | 6998333 | 6998333 | | | false | |
| | | https://kube02:2379 | 1546a2d2ac61bd8d | 3.6.8 | 3.6.0 | 24 MB | 9.6 MB | 61% | 2.1 GB | false | false | 5 | 6998333 | 6998333 | | | false | |
| | | https://kube03:2379 | 18e2fb4d02a30b2e | 3.6.8 | 3.6.0 | 24 MB | 9.8 MB | 60% | 2.1 GB | false | false | 5 | 6998333 | 6998333 | | | false | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | </code> |
| | |
| | |
| | === Tartós kiesés === |
| | |
| | Egy control plane node tartós kiesése esetén az ETCD tömöríti a RAFT naplókat, így azok alkalmazására már nincs lehetőség. Ilyenkor az alábbi lehetőségek vannak. |
| | |
| | == Automatikus helyreállás == |
| | |
| | Ebben az esetben automatikus helyreállás történik egy friss leader snapshot elküldésével. Ellenőrzéseken kívül nincs további teendő. |
| | |
| | == Helyreállítás ugyanazon a gépen == |
| | |
| | A legtisztább eljárás, hogy a két megmaradó gép egyikén töröljük az ETCD tagok és a gépek listájából a gépet. Fontos, hogy a törlegdő gépen ne fusson a ''containerd'' és a ''kubelet'' szolgáltatás valamint a kuberneteshez kapcsolódó statikus pod-ok sem. Ezt legegyszerűbben a gép leállításval tudjuk megoldani. |
| | |
| | Ellenőrizzük az ETCD tagok listáját. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key member list -w table |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | 1546a2d2ac61bd8d | started | kube02 | https://192.168.110.162:2380 | https://192.168.110.162:2379 | false | |
| | | 18e2fb4d02a30b2e | started | kube03 | https://192.168.110.163:2380 | https://192.168.110.163:2379 | false | |
| | | 20e2ce3086b2e277 | started | kube01 | https://192.168.110.161:2380 | https://192.168.110.161:2379 | false | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | </code> |
| | |
| | Távolítsuk el a hibás tagot az azonosító megadásával (ID mező). |
| | |
| | <code> |
| | # etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key member remove 18e2fb4d02a30b2e |
| | Member 18e2fb4d02a30b2e removed from cluster adeb3c2ca3243377 |
| | </code> |
| | |
| | Újra ellenőrizzük az ETCD tagok listáját. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key member list -w table |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | 1546a2d2ac61bd8d | started | kube02 | https://192.168.110.162:2380 | https://192.168.110.162:2379 | false | |
| | | 20e2ce3086b2e277 | started | kube01 | https://192.168.110.161:2380 | https://192.168.110.161:2379 | false | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | </code> |
| | |
| | Ellenőrizzük a kubernetes gépek listáját. |
| | |
| | <code> |
| | # kubectl get nodes -l node-role.kubernetes.io/control-plane |
| | NAME STATUS ROLES AGE VERSION |
| | kube01 Ready control-plane 66d v1.36.2 |
| | kube02 Ready control-plane 66d v1.36.2 |
| | kube03 NotReady control-plane 66d v1.36.2 |
| | </code> |
| | |
| | Töröljük a hibás gépet. |
| | |
| | <code> |
| | # kubectl delete node kube03 |
| | node "kube03" deleted |
| | </code> |
| | |
| | Ellenőrizzük a gépek listáját. |
| | |
| | <code> |
| | # kubectl get nodes |
| | NAME STATUS ROLES AGE VERSION |
| | kube01 Ready control-plane 66d v1.36.2 |
| | kube02 Ready control-plane 66d v1.36.2 |
| | </code> |
| | |
| | Generáljunk új tokent a gép csatlakoztatásához. |
| | |
| | <code> |
| | # kubeadm token create --print-join-command |
| | kubeadm join 192.168.110.160:6443 --token gef9jk.oafnicbu6hgwhhy7 --discovery-token-ca-cert-hash sha256:838e04036d629fbf3e152fc6606bf9a143f8e1ca0b0bc00d747fe1caba5df8f1 |
| | |
| | # kubeadm init phase upload-certs --upload-certs |
| | [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace |
| | [upload-certs] Using certificate key: |
| | 7aa4b634e4f0986669dbd19b972ec3c14a80e3ab1909a54d26ca58b0b7ec15ac |
| | </code> |
| | |
| | Ha szeretnénk előállítani a control plane csatlakoztatásának utasítását, akkor ezt a megoldást használhatjuk. Az utasítás létrehozza a tokent és a tanúsítvány kulcsot. |
| | |
| | <code> |
| | # JOIN_CMD="$(kubeadm token create --print-join-command)" |
| | # CERT_KEY="$(kubeadm init phase upload-certs --upload-certs | tail -n 1)" |
| | # echo "$JOIN_CMD --control-plane --certificate-key $CERT_KEY" |
| | kubeadm join 192.168.110.160:6443 --token 7q469t.81sbouok0em1pvjo --discovery-token-ca-cert-hash sha256:838e04036d629fbf3e152fc6606bf9a143f8e1ca0b0bc00d747fe1caba5df8f1 --control-plane --certificate-key 10babb9fd52e8ae866eabae323e7f486ae6c5fa26613ab93004738cab8432e87 |
| | </code> |
| | |
| | A snapshot-ból helyreállított gépet indítsuk el oly módon, hogy a ''containerd'' és a ''kubelet'' szolgáltatások biztosan **NE** induljanak el. Ezt a legegyszerűbben a gép boot folyamatának módosításával érjük el. A ''GRUB'' menüjében a kernel paraméterezését egészítsük ki az alábbiak szerint. |
| | |
| | ''linux ..... **systemd.mask=containerd.service systemd.mask=kubelet.service**'' |
| | |
| | Az elindult rendszerben tudjuk ellenőrizni. |
| | |
| | <code> |
| | # systemctl status containerd kubelet |
| | ○ containerd.service |
| | Loaded: masked (Reason: Unit containerd.service is masked.) |
| | Active: inactive (dead) |
| | |
| | ○ kubelet.service |
| | Loaded: masked (Reason: Unit kubelet.service is masked.) |
| | Drop-In: /usr/lib/systemd/system/kubelet.service.d |
| | └─10-kubeadm.conf |
| | Active: inactive (dead) |
| | </code> |
| | |
| | Amennyiben snapshot-ból állt helyre a gép, futtassuk a ''kubeadm reset'' utasítást. Ez törölni fogja a korábbi klaszter állományait. |
| | |
| | <code> |
| | # kubeadm reset |
| | [reset] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... |
| | [reset] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it. |
| | ... |
| | The reset process does not perform cleanup of CNI plugin configuration, |
| | network filtering rules and kubeconfig files. |
| | |
| | For information on how to perform this cleanup manually, please see: |
| | https://k8s.io/docs/reference/setup-tools/kubeadm/kubeadm-reset/ |
| | </code> |
| | |
| | Töröljük a felesleges egyéb állományokat. |
| | |
| | <code> |
| | # rm -rf /etc/cni/net.d/* /root/.kube /var/lib/cni /var/lib/etcd /var/lib/kubelet |
| | </code> |
| | |
| | Indítsuk újra a gépet. |
| | |
| | <code> |
| | # reboot |
| | </code> |
| | |
| | A gép újraindulása után csatlakoztassuk a gépet control plane szerepkörben a meglevő klaszterhez. |
| | |
| | <code> |
| | # kubeadm join 192.168.110.160:6443 --token gef9jk.oafnicbu6hgwhhy7 --discovery-token-ca-cert-hash sha256:838e04036d629fbf3e152fc6606bf9a143f8e1ca0b0bc00d747fe1caba5df8f1 --control-plane --certificate-key 7aa4b634e4f0986669dbd19b972ec3c14a80e3ab1909a54d26ca58b0b7ec15ac |
| | [preflight] Running pre-flight checks |
| | [preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... |
| | [preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it. |
| | ... |
| | |
| | This node has joined the cluster and a new control plane instance was created: |
| | |
| | * Certificate signing request was sent to apiserver and approval was received. |
| | * The Kubelet was informed of the new secure connection details. |
| | * Control plane label and taint were applied to the new node. |
| | * The Kubernetes control plane instances scaled up. |
| | * A new etcd member was added to the local/stacked etcd cluster. |
| | |
| | To start administering your cluster from this node, you need to run the following as a regular user: |
| | |
| | mkdir -p $HOME/.kube |
| | sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config |
| | sudo chown $(id -u):$(id -g) $HOME/.kube/config |
| | |
| | Run 'kubectl get nodes' to see this node join the cluster. |
| | </code> |
| | |
| | Állítsuk be a kubernetes konfigurációt a root felhasználónak. |
| | |
| | <code> |
| | # rm -rf /root/.kube/ |
| | # mkdir -p $HOME/.kube |
| | # sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config |
| | # sudo chown $(id -u):$(id -g) $HOME/.kube/config |
| | </code> |
| | |
| | Ellenőrizzük a klaszter állapotát. |
| | |
| | <code> |
| | # kubectl get nodes -l node-role.kubernetes.io/control-plane |
| | NAME STATUS ROLES AGE VERSION |
| | kube01 Ready control-plane 67d v1.36.2 |
| | kube02 Ready control-plane 67d v1.36.2 |
| | kube03 Ready control-plane 5m21s v1.36.2 |
| | </code> |
| | |
| | Ellenőrizzük az ETCD tagok listáját. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key member list -w table |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | 1546a2d2ac61bd8d | started | kube02 | https://192.168.110.162:2380 | https://192.168.110.162:2379 | false | |
| | | 20e2ce3086b2e277 | started | kube01 | https://192.168.110.161:2380 | https://192.168.110.161:2379 | false | |
| | | 23fb75a79460e3c9 | started | kube03 | https://192.168.110.163:2380 | https://192.168.110.163:2379 | false | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | </code> |
| | |
| | Ellenőrizzük az ETCD endpointok állapotát. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | https://kube01:2379 | 20e2ce3086b2e277 | 3.6.8 | 3.6.0 | 24 MB | 7.5 MB | 70% | 2.1 GB | true | false | 5 | 7031342 | 7031342 | | | false | |
| | | https://kube02:2379 | 1546a2d2ac61bd8d | 3.6.8 | 3.6.0 | 24 MB | 7.5 MB | 70% | 2.1 GB | false | false | 5 | 7031343 | 7031343 | | | false | |
| | | https://kube03:2379 | 23fb75a79460e3c9 | 3.6.8 | 3.6.0 | 24 MB | 7.5 MB | 70% | 2.1 GB | false | false | 5 | 7031343 | 7031343 | | | false | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | </code> |
| | |
| | A helyreállítás sikeres. |
| | |
| | ==== Két control plane kiesése ==== |
| | |
| | Két control plane gép kiesése esetén a kunernetes klaszter elveszíti a kvórumot. Ebben a helyzetben a worker gépeken futó pod-ok a legutolsó állapotukban maradnak. Nem működik a HPA. Nem lehet módosítani a konfiguráción. Ilyen esetben a control plane gépek mielőbbi helyreállítása szükséges. |
| | |
| | === Rövid idejű kiesés === |
| | |
| | Rövid iejű kiesés esetén amennyiben a control plane gépek fájlrendszere nem sérül, akkor a gépek újraindulásakor megtörténik a szinkron. Ilyen esetben ellenőrizni kell a klaszter állapotot. |
| | |
| | A következő példában két gép rövid időre leáll (pl.: tápellátás hiba miatt). Ebben az esetben a megszokott ellenőrzési műveletek nem futnak, mivel nincs stabil klaszter állapot. |
| | |
| | ETCD endpoint állapot vizsgálata. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key member list -w table |
| | {"level":"warn","ts":"2026-07-09T12:23:48.745371+0200","logger":"etcd-client","caller":"v3@v3.6.12/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00027e960/127.0.0.1:2379","peer":"Peer{Addr: '127.0.0.1:2379', LocalAddr: '127.0.0.1:45328', AuthInfo: 'tls'}","method":"/etcdserverpb.Cluster/MemberList","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} |
| | Error: context deadline exceeded |
| | </code> |
| | |
| | A gépek listázása időtúllépés miatt sikertelen, vagy érvénytelen választ kap. |
| | |
| | <code> |
| | # kubectl get nodes |
| | Error from server (Forbidden): nodes is forbidden: User "kubernetes-admin" cannot list resource "nodes" in API group "" at the cluster scope |
| | </code> |
| | |
| | Újra kell indítani a kiesett gépeket. Ellenőrizzük a gépek elérhetőségét. |
| | |
| | <code> |
| | [root@kube01 ~]# ping -c 5 kube02 |
| | PING kube02 (192.168.110.162) 56(84) bytes of data. |
| | 64 bytes from kube02 (192.168.110.162): icmp_seq=1 ttl=64 time=0.208 ms |
| | 64 bytes from kube02 (192.168.110.162): icmp_seq=2 ttl=64 time=0.192 ms |
| | 64 bytes from kube02 (192.168.110.162): icmp_seq=3 ttl=64 time=0.202 ms |
| | 64 bytes from kube02 (192.168.110.162): icmp_seq=4 ttl=64 time=0.204 ms |
| | 64 bytes from kube02 (192.168.110.162): icmp_seq=5 ttl=64 time=0.234 ms |
| | |
| | --- kube02 ping statistics --- |
| | 5 packets transmitted, 5 received, 0% packet loss, time 4122ms |
| | rtt min/avg/max/mdev = 0.192/0.208/0.234/0.014 ms |
| | [root@kube01 ~]# ping -c 5 kube03 |
| | PING kube03 (192.168.110.163) 56(84) bytes of data. |
| | 64 bytes from kube03 (192.168.110.163): icmp_seq=1 ttl=64 time=0.247 ms |
| | 64 bytes from kube03 (192.168.110.163): icmp_seq=2 ttl=64 time=0.295 ms |
| | 64 bytes from kube03 (192.168.110.163): icmp_seq=3 ttl=64 time=0.190 ms |
| | 64 bytes from kube03 (192.168.110.163): icmp_seq=4 ttl=64 time=0.207 ms |
| | 64 bytes from kube03 (192.168.110.163): icmp_seq=5 ttl=64 time=0.179 ms |
| | |
| | --- kube03 ping statistics --- |
| | 5 packets transmitted, 5 received, 0% packet loss, time 4084ms |
| | rtt min/avg/max/mdev = 0.179/0.223/0.295/0.042 ms |
| | </code> |
| | |
| | Ha minden gép helyreállt, akkor futtassuk az ellenőrzéseket. |
| | |
| | Control plane gépek listája. |
| | |
| | <code> |
| | # kubectl get nodes -l node-role.kubernetes.io/control-plane |
| | NAME STATUS ROLES AGE VERSION |
| | kube01 Ready control-plane 67d v1.36.2 |
| | kube02 Ready control-plane 67d v1.36.2 |
| | kube03 Ready control-plane 32m v1.36.2 |
| | </code> |
| | |
| | ETCD klaszter tagok listála. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key member list -w table |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | 1546a2d2ac61bd8d | started | kube02 | https://192.168.110.162:2380 | https://192.168.110.162:2379 | false | |
| | | 20e2ce3086b2e277 | started | kube01 | https://192.168.110.161:2380 | https://192.168.110.161:2379 | false | |
| | | 23fb75a79460e3c9 | started | kube03 | https://192.168.110.163:2380 | https://192.168.110.163:2379 | false | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | </code> |
| | |
| | ETCD endpoint állapot ellenőrzése. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | https://kube01:2379 | 20e2ce3086b2e277 | 3.6.8 | 3.6.0 | 24 MB | 12 MB | 50% | 2.1 GB | true | false | 6 | 7036311 | 7036311 | | | false | |
| | | https://kube02:2379 | 1546a2d2ac61bd8d | 3.6.8 | 3.6.0 | 24 MB | 12 MB | 50% | 2.1 GB | false | false | 6 | 7036312 | 7036312 | | | false | |
| | | https://kube03:2379 | 23fb75a79460e3c9 | 3.6.8 | 3.6.0 | 24 MB | 12 MB | 50% | 2.1 GB | false | false | 6 | 7036312 | 7036312 | | | false | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | </code> |
| | |
| | És egy utolsó ellenőrzés. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health -w table |
| | +---------------------+--------+--------------+-------+ |
| | | ENDPOINT | HEALTH | TOOK | ERROR | |
| | +---------------------+--------+--------------+-------+ |
| | | https://kube01:2379 | true | 37.207191ms | | |
| | | https://kube02:2379 | true | 39.074018ms | | |
| | | https://kube03:2379 | true | 197.541227ms | | |
| | +---------------------+--------+--------------+-------+ |
| | </code> |
| | |
| | A worker gépek állapota eltérhet a control plane gépekben tárolt állapottól. Ebben az esetben érdemes minden worker gépet újraindítani. |
| | |
| | === Végleges kiesés === |
| | |
| | Amennyiben a két kiesett gép többé már nem érhető el, akkor a meglevő node-ból egy gépes klasztert kell készíteni, és utánna csatlakoztatni a többi gépet. Az alábbi műveleteket a megmaradt gépen kell futtatni. |
| | |
| | |
| | Ellenőrizzük aze etcd állapotát. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table |
| | {"level":"warn","ts":"2026-07-09T12:59:52.835472+0200","logger":"etcd-client","caller":"v3@v3.6.12/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00027f2c0/kube02:2379","peer":"Peer{Addr: <nil>, LocalAddr: <nil>, AuthInfo: <nil>}","method":"/etcdserverpb.Maintenance/Status","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial tcp 192.168.110.162:2379: connect: no route to host\""} |
| | Failed to get the status of endpoint https://kube02:2379 (context deadline exceeded) |
| | {"level":"warn","ts":"2026-07-09T12:59:57.837654+0200","logger":"etcd-client","caller":"v3@v3.6.12/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00025e960/kube03:2379","peer":"Peer{Addr: <nil>, LocalAddr: <nil>, AuthInfo: <nil>}","method":"/etcdserverpb.Maintenance/Status","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial tcp 192.168.110.163:2379: connect: no route to host\""} |
| | Failed to get the status of endpoint https://kube03:2379 (context deadline exceeded) |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+-----------------------+--------------------------+-------------------+ |
| | | ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+-----------------------+--------------------------+-------------------+ |
| | | https://kube01:2379 | 20e2ce3086b2e277 | 3.6.8 | 3.6.0 | 24 MB | 8.2 MB | 67% | 2.1 GB | false | false | 6 | 7041707 | 7041707 | etcdserver: no leader | | false | |
| | +---------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+-----------------------+--------------------------+-------------------+ |
| | </code> |
| | |
| | Amennyiben működik az etcd, akkor snapshot-ot kell készíteni az adatbázisról. |
| | |
| | <code> |
| | # etcdctl snapshot save /root/etcd-snapshot-disaster.db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key |
| | {"level":"info","ts":"2026-07-09T13:02:05.335290+0200","caller":"snapshot/v3_snapshot.go:83","msg":"created temporary db file","path":"/root/etcd-snapshot-disaster.db.part"} |
| | {"level":"info","ts":"2026-07-09T13:02:05.341997+0200","logger":"client","caller":"v3@v3.6.12/maintenance.go:236","msg":"opened snapshot stream; downloading"} |
| | {"level":"info","ts":"2026-07-09T13:02:05.415922+0200","caller":"snapshot/v3_snapshot.go:96","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"} |
| | {"level":"info","ts":"2026-07-09T13:02:05.666551+0200","logger":"client","caller":"v3@v3.6.12/maintenance.go:302","msg":"completed snapshot read; closing"} |
| | {"level":"info","ts":"2026-07-09T13:02:05.997155+0200","caller":"snapshot/v3_snapshot.go:111","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"24 MB","took":"661.672699ms","etcd-version":"3.6.0"} |
| | {"level":"info","ts":"2026-07-09T13:02:05.997294+0200","caller":"snapshot/v3_snapshot.go:121","msg":"saved","path":"/root/etcd-snapshot-disaster.db"} |
| | Snapshot saved at /root/etcd-snapshot-disaster.db |
| | Server version 3.6.0 |
| | </code> |
| | |
| | Le kell tiltani és le kell állítani a ''kubelet'' és a ''containerd'' szolgáltatásokat. |
| |
| <code> | <code> |
| # systemctl disable kubelet containerd | |
| # systemctl stop kubelet containerd | # systemctl stop kubelet containerd |
| # systemctl reboot | # systemctl disable kubelet containerd |
| | Removed '/etc/systemd/system/multi-user.target.wants/containerd.service'. |
| | Removed '/etc/systemd/system/multi-user.target.wants/kubelet.service'. |
| | </code> |
| | |
| | Újra kell indítani a gépet. |
| <code> | <code> |
| | # reboot |
| | </code> |
| |
| Korábbi környezet mentése. | Újraindítás után a gépen nem szabad futnia a szolgáltatásoknak. |
| |
| <code> | <code> |
| # mkdir -p /root/k8s-backup/pre-restore | # systemctl is-active containerd kubelet |
| # mkdir -p /root/k8s-backup/pre-restore/etc/kubernetes | inactive |
| # mkdir -p /root/k8s-backup/pre-restore/var/lib | inactive |
| # cp -a /etc/kubernetes/manifests /root/k8s-backup/pre-restore/etc/kubernetes/ | |
| # cp -a /etc/kubernetes/pki /root/k8s-backup/pre-restore/etc/kubernetes/ | |
| # cp -a /var/lib/etcd /root/k8s-backup/pre-restore/var/lib/ | |
| </code> | </code> |
| |
| | A snapshot helyreállítását el kell végezni egy másik könyvtárban ''/var/lib/etcd-new'' |
| | |
| | <code> |
| | # etcdutl snapshot restore /root/etcd-snapshot-disaster.db --name kube01 --data-dir /var/lib/etcd-new --initial-cluster kube01=https://192.168.110.161:2380 --initial-cluster-token new-cluster --initial-advertise-peer-urls=https://192.168.110.161:2380 |
| | 2026-07-09T13:06:50+02:00 info snapshot/v3_snapshot.go:305 restoring snapshot {"path": "/root/etcd-snapshot-disaster.db", "wal-dir": "/var/lib/etcd-new/member/wal", "data-dir": "/var/lib/etcd-new", "snap-dir": "/var/lib/etcd-new/member/snap", "initial-memory-map-size": 10737418240} |
| | 2026-07-09T13:06:50+02:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-new/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc000062190} |
| | 2026-07-09T13:06:50+02:00 info bbolt bbolt@v1.4.3/db.go:322 Opening bbolt db (/var/lib/etcd-new/member/snap/db) successfully |
| | 2026-07-09T13:06:50+02:00 info schema/membership.go:138 Trimming membership information from the backend... |
| | 2026-07-09T13:06:50+02:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-new/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00011c128} |
| | 2026-07-09T13:06:50+02:00 info bbolt bbolt@v1.4.3/db.go:322 Opening bbolt db (/var/lib/etcd-new/member/snap/db) successfully |
| | 2026-07-09T13:06:50+02:00 info membership/cluster.go:424 added member {"cluster-id": "3cd4235e48cb8de9", "local-member-id": "0", "added-peer-id": "234c167f7d0f641f", "added-peer-peer-urls": ["https://192.168.110.161:2380"], "added-peer-is-learner": false} |
| | 2026-07-09T13:06:51+02:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-new/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc0000621d8} |
| | 2026-07-09T13:06:51+02:00 info bbolt bbolt@v1.4.3/db.go:322 Opening bbolt db (/var/lib/etcd-new/member/snap/db) successfully |
| | 2026-07-09T13:06:51+02:00 info snapshot/v3_snapshot.go:333 restored snapshot {"path": "/root/etcd-snapshot-disaster.db", "wal-dir": "/var/lib/etcd-new/member/wal", "data-dir": "/var/lib/etcd-new", "snap-dir": "/var/lib/etcd-new/member/snap", "initial-memory-map-size": 10737418240} |
| | </code> |
| | |
| | Sikeres helyreállítás után a korábbi könyvtárat el kell mozgatni. |
| | |
| | <code> |
| | # mv /var/lib/etcd /var/lib/etcd-old |
| | </code> |
| | |
| | A helyreállított snapshot könyvtárat a helyére kell másolni. |
| | |
| | <code> |
| | # cp -a /var/lib/etcd-new /var/lib/etcd |
| | </code> |
| | |
| | Újra engedélyezni kell a szolgáltatásokat. |
| | |
| | <code> |
| | # systemctl enable kubelet containerd |
| | Created symlink '/etc/systemd/system/multi-user.target.wants/kubelet.service' → '/usr/lib/systemd/system/kubelet.service'. |
| | Created symlink '/etc/systemd/system/multi-user.target.wants/containerd.service' → '/usr/lib/systemd/system/containerd.service'. |
| | </code> |
| | |
| | Újra kell indítani a gépet. |
| | |
| | <code> |
| | # reboot |
| | </code> |
| | |
| | Az újraindítást követően a control plane elindul. Az indulást követően ellenőrizzük a klaszer állapotát. |
| | |
| | Control plane gépek listája. |
| | |
| | <code> |
| | # kubectl get nodes -l node-role.kubernetes.io/control-plane |
| | NAME STATUS ROLES AGE VERSION |
| | kube01 Ready control-plane 67d v1.36.2 |
| | kube02 NotReady control-plane 67d v1.36.2 |
| | kube03 NotReady control-plane 90m v1.36.2 |
| | </code> |
| | |
| | ETCD endpoint lista. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table |
| | +------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED | |
| | +------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | | https://127.0.0.1:2379 | 234c167f7d0f641f | 3.6.8 | 3.6.0 | 24 MB | 13 MB | 49% | 2.1 GB | true | false | 3 | 966 | 966 | | | false | |
| | +------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+ |
| | </code> |
| | |
| | ETCD tagok listája. |
| | |
| | <code> |
| | # etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list -w table |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | | 234c167f7d0f641f | started | kube01 | https://192.168.110.161:2380 | https://192.168.110.161:2379 | false | |
| | +------------------+---------+--------+------------------------------+------------------------------+------------+ |
| | </code> |
| | |
| | Az etcd adatbázisa rendben. Új példány indult. |
| | |
| | Töröljük a két hiányzó gépet. |
| | |
| | <code> |
| | # kubectl delete node kube02 |
| | node "kube02" deleted |
| | # kubectl delete node kube03 |
| | node "kube03" deleted |
| | </code> |
| | |
| | Hozzunk létre tokent és tanúsítvány kulcsot. |
| | |
| | <code> |
| | # JOIN_CMD="$(kubeadm token create --print-join-command)" |
| | # CERT_KEY="$(kubeadm init phase upload-certs --upload-certs | tail -n 1)" |
| | # echo "$JOIN_CMD --control-plane --certificate-key $CERT_KEY" |
| | kubeadm join 192.168.110.160:6443 --token 7q469t.81sbouok0em1pvjo --discovery-token-ca-cert-hash sha256:838e04036d629fbf3e152fc6606bf9a143f8e1ca0b0bc00d747fe1caba5df8f1 --control-plane --certificate-key 10babb9fd52e8ae866eabae323e7f486ae6c5fa26613ab93004738cab8432e87 |
| | </code> |
| | |
| | Két új control plane gép telepítése után csatlakoztatjuk a gépeket a klaszterhez. |
| | |
| | <code> |
| | kubeadm join 192.168.110.160:6443 --token 7q469t.81sbouok0em1pvjo --discovery-token-ca-cert-hash sha256:838e04036d629fbf3e152fc6606bf9a143f8e1ca0b0bc00d747fe1caba5df8f1 --control-plane --certificate-key 10babb9fd52e8ae866eabae323e7f486ae6c5fa26613ab93004738cab8432e87 |
| | [preflight] Running pre-flight checks |
| | [preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... |
| | [preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it. |
| | ... |
| | |
| | This node has joined the cluster and a new control plane instance was created: |
| | |
| | * Certificate signing request was sent to apiserver and approval was received. |
| | * The Kubelet was informed of the new secure connection details. |
| | * Control plane label and taint were applied to the new node. |
| | * The Kubernetes control plane instances scaled up. |
| | * A new etcd member was added to the local/stacked etcd cluster. |
| | |
| | To start administering your cluster from this node, you need to run the following as a regular user: |
| | |
| | mkdir -p $HOME/.kube |
| | sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config |
| | sudo chown $(id -u):$(id -g) $HOME/.kube/config |
| | |
| | Run 'kubectl get nodes' to see this node join the cluster. |
| | </code> |
| | |
| | Mindkét gépen állítsuk be a kubernetes konfigurációt a root felhasználónak. |
| | |
| | <code> |
| | # rm -rf /root/.kube/ |
| | # mkdir -p $HOME/.kube |
| | # sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config |
| | # sudo chown $(id -u):$(id -g) $HOME/.kube/config |
| | </code> |
| | |
| | Ellenőrizzük a gépeket. |
| | |
| | <code> |
| | # kubectl get nodes -l node-role.kubernetes.io/control-plane |
| | NAME STATUS ROLES AGE VERSION |
| | kube01 Ready control-plane 67d v1.36.2 |
| | kube02 Ready control-plane 10m v1.36.2 |
| | kube03 Ready control-plane 10m v1.36.2 |
| | </code> |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health -w table |
| | +---------------------+--------+--------------+-------+ |
| | | ENDPOINT | HEALTH | TOOK | ERROR | |
| | +---------------------+--------+--------------+-------+ |
| | | https://kube01:2379 | true | 13.402358ms | | |
| | | https://kube03:2379 | true | 90.974734ms | | |
| | | https://kube02:2379 | true | 173.267605ms | | |
| | +---------------------+--------+--------------+-------+ |
| | </code> |
| | |
| | A helyreállítás sikeres. |
| | |
| | ==== Három control plane kiesése ==== |
| | |
| | Három control plane gép elvesztésekor a legegyszerűbb megoldás három azonos nevű és IP című gép telepítése. A telepítés után vissza kell állítani a konfiguráció mentéseket a megfelelő gépeken. |
| | |
| | Mindhárom gépre be kell másolni a mentéseket a ''/root/k8s-config-$(hostname)-YYYYMMMDD.tar.gz'' fájlként. A másolás után az összes gépek ki kell csomagolni a fájlok tartalmát. |
| | |
| | <code> |
| | # cd / |
| | # tar -xzf /root/k8s-config-$(hostname)-YYYYMMMDD.tar.gz |
| | </code> |
| | |
| | Ezzel a konfiguráció helyreállítása megtörtént. |
| | |
| | ETCD snapshot helyreállításához mindhárom gépre másoljuk be a snapshot állományt a ''/root/etcd-snapshot-$(hostname)-YYYYMMDD.db'' néven. A fájl mindhárom gépen ugyanaz legyen. |
| | |
| | <code> |
| | # (kube01) md5sum /root/etcd-snapshot-$(hostname)YYYYMMDD.db |
| | 6a66ab3ffa291dcb9d199817858b5617 /root/etcd-snapshot-$(hostname)-YYYYMMDD.db |
| | |
| | # (kube02) md5sum /root/etcd-snapshot-$(hostname)YYYYMMDD.db |
| | 6a66ab3ffa291dcb9d199817858b5617 /root/etcd-snapshot-$(hostname)-YYYYMMDD.db |
| | |
| | # (kube03) md5sum /root/etcd-snapshot-$(hostname)YYYYMMDD.db |
| | 6a66ab3ffa291dcb9d199817858b5617 /root/etcd-snapshot-$(hostname)-YYYYMMDD.db |
| | </code> |
| | |
| | Gépenként állítsuk helyre a snapshot tartalmát. |
| | |
| | == kube01 == |
| | |
| | <code> |
| | etcdutl snapshot restore /root/etcd-snapshot-$(hostname)-YYYYMMDD.db --name kube01 --data-dir /var/lib/etcd --initial-cluster kube01=https://192.168.110.161:2380,kube02=https://192.168.110.162:2380,kube03=https://192.168.110.163:2380 --initial-advertise-peer-urls https://192.168.110.161:2380 --initial-cluster-token new-cluster |
| | </code> |
| | |
| | == kube02 == |
| | |
| | <code> |
| | etcdutl snapshot restore /root/etcd-snapshot-$(hostname)-YYYYMMDD.db --name kube02 --data-dir /var/lib/etcd --initial-cluster kube01=https://192.168.110.161:2380,kube02=https://192.168.110.162:2380,kube03=https://192.168.110.163:2380 --initial-advertise-peer-urls https://192.168.110.162:2380 --initial-cluster-token new-cluster |
| | </code> |
| | |
| | == kube03 == |
| | |
| | <code> |
| | etcdutl snapshot restore /root/etcd-snapshot-$(hostname)-YYYYMMDD.db --name kube03 --data-dir /var/lib/etcd --initial-cluster kube01=https://192.168.110.161:2380,kube02=https://192.168.110.162:2380,kube03=https://192.168.110.163:2380 --initial-advertise-peer-urls https://192.168.110.163:2380 --initial-cluster-token new-cluster |
| | </code> |
| | |
| | Jogosultságok beállítása a ''/var/lib/etcd'' könyvtárra. |
| | |
| | <code> |
| | # chown -R root:root /var/lib/etcd |
| | </code> |
| | |
| | Minden gépen engedélyezni kell a ''containerd'' és a ''kubelet'' szolgáltatásokat. |
| | |
| | <code> |
| | # systemctl enable containerd kubelet |
| | Created symlink '/etc/systemd/system/multi-user.target.wants/containerd.service' → '/usr/lib/systemd/system/containerd.service'. |
| | Created symlink '/etc/systemd/system/multi-user.target.wants/kubelet.service' → '/usr/lib/systemd/system/kubelet.service'. |
| | </code> |
| | |
| | Újra kell indítani a control plane gépeket. |
| | |
| | <code> |
| | # reboot |
| | </code> |
| | |
| | Amikor a gépek elindultak ellenőrizni kell a k8s klaszter állapotát. |
| | |
| | <code> |
| | # kubectl get nodes -l node-role.kubernetes.io/control-plane |
| | NAME STATUS ROLES AGE VERSION |
| | kube01 Ready control-plane 45m v1.36.2 |
| | kube02 Ready control-plane 45m v1.36.2 |
| | kube03 Ready control-plane 45m v1.36.2 |
| | </code> |
| | |
| | <code> |
| | # etcdctl --endpoints=https://kube01:2379,https://kube02:2379,https://kube03:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health -w table |
| | +---------------------+--------+--------------+-------+ |
| | | ENDPOINT | HEALTH | TOOK | ERROR | |
| | +---------------------+--------+--------------+-------+ |
| | | https://kube01:2379 | true | 10.749263ms | | |
| | | https://kube03:2379 | true | 32.846299ms | | |
| | | https://kube02:2379 | true | 45.252051ms | | |
| | +---------------------+--------+--------------+-------+ |
| | </code> |
| | |
| | A helyreállítás sikeres. |
| | |
| | ===== Worker ===== |
| | |
| | ==== Új worker gép létrehozása ==== |
| | |
| | A worker gépek nevei és az IP beállítások egyezzenek meg a korábbi gép azonosítóival. |
| | |
| | A telepítés után valamelyik control plane gépen futtassuk a nem működő gép törlését. |
| | |
| | <code> |
| | # kubectl delete node kube04 |
| | node "kube04" deleted |
| | </code> |
| | |
| | A törlés után hozzunk létre új tokent a gép csatlakoztatásához. |
| | |
| | <code> |
| | # kubeadm token create --print-join-command |
| | kubeadm join 192.168.110.160:6443 --token sfk438.u9jyjz6cdpz1puk5 --discovery-token-ca-cert-hash sha256:838e04036d629fbf3e152fc6606bf9a143f8e1ca0b0bc00d747fe1caba5df8f1 |
| | </code> |
| | |
| | Csatlakoztassuk az új worker gépet a klaszterhez. |
| | |
| | <code> |
| | # kubeadm join 192.168.110.160:6443 --token sfk438.u9jyjz6cdpz1puk5 --discovery-token-ca-cert-hash sha256:838e04036d629fbf3e152fc6606bf9a143f8e1ca0b0bc00d747fe1caba5df8f1 |
| | [preflight] Running pre-flight checks |
| | [preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... |
| | [preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it. |
| | ... |
| | |
| | This node has joined the cluster: |
| | * Certificate signing request was sent to apiserver and a response was received. |
| | * The Kubelet was informed of the new secure connection details. |
| | |
| | Run 'kubectl get nodes' on the control-plane to see this node join the cluster. |
| | </code> |
| |
| | Minden további műveletet a telepítési leírás szerint kell végrehajtani. |
