How to take the backup of Kubernetes etcd database ? For working with etcd a client software called etcdctl must be downloaded. Below is installation script :

export RELEASE="3.3.13"
wget https://github.com/etcd-io/etcd/releases/download/v${RELEASE}/etcd-v${RELEASE}-linux-amd64.tar.gz
tar xvf etcd-v${RELEASE}-linux-amd64.tar.gz
cd etcd-v${RELEASE}-linux-amd64
sudo mv etcdctl /usr/local/bin

It is better to create a separate folder for etcd backups. And then to execute creating etcd snapshot. See below :

mkdir /etcd-backup

ETCDCTL_API=3 etcdctl --endpoints=192.168.198.147:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key snapshot save /etcd-backup/etcd-snapshot-latest.db

In the example –endpoints parameter value is <on what ip address etcd is running>:<on what port etcd is running>; –cacert is used for ssl communication; –cert is public certificate; –key is the private key; /etcd-backup/etcd-snapshot-latest.db is the path and the name of snapshot file.

Master pods details can be seen in /etc/kubernetes/manifests folder. etcd.yaml is the script for creating etcd Pod. endpoints value and all certificate values can be taken from there (values can differ from those in example).

Current information about already created etcd Pod is stored in /var/lib/etcd folder.

Note that only Api-Server communicates with etcd. There is  direct communication between Api-Server and etcd and no services need to be created for interacting with etcd. Services are required when there are multiple pods, for load balancing and request routing. But in this case there is only one etcd pod and only Api-Server can communicate.

How can previously saved data be restored from snapshot (even if /var/lib/etcd folder is also gone) ? For this, the command, examle of which is shown below, is executed :

ETCDCTL_API=3 etcdctl snapshot restore /etcd-backup/etcd-snapshot-latest.db
--initial-cluster etcd-restore=https://10.128.0.32:2380
--initial-advertise-peer-urls https://10.128.0.32:2380
--name etcd-restore
--data-dir /var/lib/etcd

Here /etcd-backup/etcd-snapshot-latest.db is the full path to snapshot data. –data-dir value points to the folder where to store information about the Pod, typically it is /var/lib/etcd. –initial-cluster ip address and port number, in the example it is https://10.128.0.32:2380, should be replaced by value from /etc/kubernetes/manifests/etcd.yaml (same as with –initial-advertise-peer-urls).