kubernetes 1.18 安裝-day 02

August 20, 2020

Introduction

k8s 是一個用來編排容器的工具,這近幾年非常火紅。有著高可靠的架構、應用程序部署、自動部署、擴展等功能。

本篇文章已安裝為目的。

Environment

實驗環境會有一台 master 和兩台 node 而資源配置如下

192.168.134.131 | master
192.168.134.133 | node01
192.168.134.135 | node02

Configure Hostname

$ sudo hostnamectl set-hostname master
$ sudo hostnamectl set-hostname node01
$ sudo hostnamectl set-hostname node02

設定完之後,會出現 DNS 問題。此問題會讓本機無法解析,如下解決

$ sudo vim /etc/hosts
192.168.134.131 master
192.168.134.133 node01
192.168.134.133 node02

Disable Swap

所有機器應該要固定 CPU/memory,不應該使用 swap 會導致效能降低。

$ sudo swapoff -a
$ sudo swapon -s

要保持永久效果至 fstab 將 swap 項目註解

$ sudo vim /etc/fstab
...
#/dev/mapper/ubuntu--vg-swap_1 none            swap    sw              0       0

Prerequisites for Kubernetes

$ sudo apt-get update
$ sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

還需要安裝 Docker 這邊不說明。可至官方查看

Installing Kubernetes

$ sudo su -c "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -"

創建 Kubernetes 儲存庫檔案

$ sudo vim /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main

更新 cache

$ sudo apt update   
$ sudo apt-get install -y kubelet kubeadm kubectl

上述步驟三台主機都需要設置。

Initialize the Kubernetes Cluster

在 master 主機執行以下

$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.134.131 --kubernetes-version v1.18.8
# --apiserver-advertise-address: Api server 服務 IP
# --apiserver-bind-port: Api server 服務 port,預設 6443
# --pod-network-cidr: 分配 Pod IP 範圍 (CIDR)
# --kubernetes-version: 指定 Kubernetes 版本
...
[mark-control-plane] Marking the node k8smaster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 7zd2c5.w9825shkwsa7vjat
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.134.131:6443 --token av0pcs.izdosjqm3csj1par \ 
    --discovery-token-ca-cert-hash sha256:320751ce55f3163938653d85bfb4ee44104cabc092bed010d13fcab1d384042c

以一般用戶身份運行以下。讓 kubectl 可連到 Kubernetes,將設定複製到 $HOME/.kube/config 底下,kubectl 執行時會到該目錄下取得 Api server 等相關資訊。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

如果給的加入 k8s Node Token 遺失,可透過以下來獲取

$ kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
av0pcs.izdosjqm3csj1par   23h         2020-08-19T09:21:36Z   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

上述如果要重置可用 kubeadm reset 來完成,會移除相關資訊。 TLS 錯誤參考 troubleshooting-kubeadm

查看節點狀況

發現 master 的狀態沒 ready,其中影響是 DNS 元件。

$ kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
master   NotReady   master   10m   v1.18.8

前面步驟完成後,在 master 節點上,預設會在 default namespace 上,而 kubernetes 上的預設安裝元件會在 kube-system 上。

$ kubectl get namespace
NAME              STATUS   AGE
default           Active   14m
kube-node-lease   Active   14m
kube-public       Active   14m
kube-system       Active   14m

切換至 kube-system 上查看。

$ kubectl -n kube-system get pods # -n 切換 namespace
NAME                             READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-5pfbz         0/1     Pending   0          11m
coredns-66bff467f8-v4txv         0/1     Pending   0          11m
etcd-master                      1/1     Running   0          11m
kube-apiserver-master            1/1     Running   0          11m
kube-controller-manager-master   1/1     Running   0          11m
kube-proxy-fm749                 1/1     Running   0          11m
kube-scheduler-master            1/1     Running   0          11m

而 DNS 是因為系統設計需求,需要安裝 CNI 提供網路解決,這邊為官網資訊

Deploy Network for Pod

先前已經定義了一個 pod 網路,但是,應該為網路創建一個 namespace。可依照環境選擇適當的 KubernetesCNI plugin。這邊使用 FlannelFlannel 將為 Kubernetes(k8s)提供overlay network。若想要自定義的 CIDR,需更改 flannel 中的 net-conf.jsonNetwork 並用新的 CIDR init

藉由 Flannel YAML 創建 Pod 網路。

$ wget https://raw.githubusercontent.com/coreos/flannel/32a765fd19ba45b387fdc5e3812c41fff47cfd55/Documentation/kube-flannel.yml
$ kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel configured
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg unchanged
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

提供網路後,DNS 元件已經運行起來。

$ kubectl -n kube-system get pods
NAME                             READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-5pfbz         1/1     Running   0          18m
coredns-66bff467f8-v4txv         1/1     Running   0          18m
etcd-master                      1/1     Running   0          18m
kube-apiserver-master            1/1     Running   0          18m
kube-controller-manager-master   1/1     Running   0          18m
kube-flannel-ds-amd64-gljsn      1/1     Running   0          55s
kube-proxy-fm749                 1/1     Running   0          18m
kube-scheduler-master            1/1     Running   0          18m

一個叢集只能用一種 Pod 網路(除非用 multus-cni)

Verify the Nodes

檢查 nodes 狀態

$ kubectl get node
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   23m   v1.18.8

$ kubectl get node -o wide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
master   Ready    master   23m   v1.18.8   192.168.134.131   <none>        Ubuntu 20.04 LTS   5.4.0-42-generic   docker://19.3.12

要知道當前的 namespace。在建立 Kubernetes 集群時,預設下創建以下所有 namespace

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
kube-system   coredns-66bff467f8-5pfbz         1/1     Running   0          23m
kube-system   coredns-66bff467f8-v4txv         1/1     Running   0          23m
kube-system   etcd-master                      1/1     Running   0          24m
kube-system   kube-apiserver-master            1/1     Running   0          24m
kube-system   kube-controller-manager-master   1/1     Running   0          24m
kube-system   kube-flannel-ds-amd64-gljsn      1/1     Running   0          6m20s
kube-system   kube-proxy-fm749                 1/1     Running   0          23m
kube-system   kube-scheduler-master            1/1     Running   0          24m

Join Workers with K8s Master

在前面初始化時,最後給了 Token。透過 Token,將 workermaster 連接起來。以下針對每台 worker 進行加入的操作。

$ sudo kubeadm join 192.168.134.131:6443 --token av0pcs.izdosjqm3csj1par --discovery-token-ca-cert-hash sha256:320751ce55f3163938653d85bfb4ee44104cabc092bed010d13fcab1d384042c
...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

再加入節點時輸出錯誤嘗試執行以下,重新嘗試

$ sudo kubeadm reset

Verify the Nodes

狀態已 Ready 顯示表示成功

$ kubectl get nodes
NAME     STATUS   ROLES    AGE    VERSION
master   Ready    master   33m    v1.18.8
node01   Ready    <none>   4m2s   v1.18.8
node02   Ready    <none>   52s    v1.18.8

Additional information

$ sudo vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf  # 是透過 kubeadm 去架設 kubernetes cluster,所以設定檔可以置這邊修改。可參考下面"kubeadm 與 kubelet 溝通"

Ref

kubernetes 官網

flannel

create-cluster-kubeadm

troubleshooting-kubeadm

network-plugins

kubeadm 與 kubelet 溝通