etcd is a consistent key value datastore that Kubernetes leverages to store the cluster configuration. However etcd is not a native Kubernetes component. ectd could be installed on separate dedicated instances or on master instances. In production, etcd is installed in HA (High Availability) mode i.e. etcd server is installed on more than one instance. In HA, an etcd cluster is comprised of odd number of instances - 1, 3, 5 etc. however 5 is the recommended number of etcd instances for production environments

Prerequisites:

  • Should have some understanding of Pod manifests
  • Some familiarity with the kube-system namespace

Environment:

  • Kubernetes cluster launched using kubeadm
  • We are doing this exercise on a cluster with single control plane (1 master) and 3 nodes

Let's check the details about the etcd pod running on the master

networkandcode@master:~$ kubectl get po -n kube-system -o wide | grep etcd
etcd-master                               1/1     Running   2          32h   10.128.15.226    master   <none>           <none>

The kube-apiserver is the central software block in the Kubernetes system and it acts like a server for most of the components, however it acts as a client when talking to the etcd server. So the kube-apiserver and ectd form an HTTPS client-server combination. The kube-apiserver is also running as a Pod, and there should be a way for it know the details of the etcd server such as it's IP address, it's ca certificate and so on, so that it can communicate with it. This is done with flags such as --etcd-servers of the kube-apiserver command, and this command is part of the Pod manifest

Let's check the running kube-apiserver Pod's manifest in yaml

networkandcode@master:~$ kubectl get po kube-apiserver-master -n kube-system -o yaml
apiVersion: v1
kind: Pod
spec:
  containers:
  - command:
    - kube-apiserver
    --TRUNCATED--
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
--TRUNCATED--

As the cluster was already launched using kubeadm, the kube-apiserver has the details about the etcd in it's Pod manifest, if at all we want to make changes to the Pod manifest, we could modify the configuration of the Pod saved in the path /etc/kubernetes/manifests/

networkandcode@master:~$ ls /etc/kubernetes/manifests
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

Here kube-apiserver.yaml refers to the saved Pod manifest of the kube-apiserver, and etcd.yaml is that of the ectd server

The section --etcd-servers=https://127.0.0.1:2379 says the etcd server is on the same host, note that the IP 127.0.0.1 refers to the local host itself. And, the default port on which etcd server is functioning is TCP 2379

The kube-apiserver which the etcd client now knows how to connect with the etcd server, however the etcd server should already be existing and functioning on the same IP and port. The etcd server should have started specifying the same IP and port as that given in the kube-apiserver's configuration. This is done using the flags --listen-client-urls and --advertise-client-urls of the etcd command, and this command is part of the etcd Pod's manifest. Let's check the running manifest of the etcd server

networkandcode@master:~$ kubectl get po etcd-master -n kube-system -o yaml
apiVersion: v1
kind: Pod
--TRUNCATED--
spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://10.128.15.226:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://10.128.15.226:2380
    - --initial-cluster=master=https://10.128.15.226:2380
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,https://10.128.15.226:2379
    - --listen-metrics-urls=http://127.0.0.1:2381
    - --listen-peer-urls=https://10.128.15.226:2380
    - --name=master
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
--TRUNCATED--

The --listen-client-urls flag also has an extra entry https://10.128.15.226:2379 which refers to the local IP address of the master instance where etcd is installed, and 2379 is the standard etcd server port on which the clients connect

--end-of-post---