Introduction
k8s 是一個用來編排容器的工具，這近幾年非常火紅。有著高可靠的架構、應用程序部署、自動部署、擴展等功能。
本篇文章已安裝為目的。
Environment
實驗環境會有一台 master 和兩台 node 而資源配置如下
- CPU
- 2 vCPU
- memory
- 2 GB
192.168.134.131 | master
192.168.134.133 | node01
192.168.134.135 | node02
Configure Hostname
$ sudo hostnamectl set-hostname master
$ sudo hostnamectl set-hostname node01
$ sudo hostnamectl set-hostname node02
設定完之後，會出現 DNS 問題。此問題會讓本機無法解析，如下解決
$ sudo vim /etc/hosts
192.168.134.131 master
192.168.134.133 node01
192.168.134.133 node02
Disable Swap
所有機器應該要固定 CPU/memory，不應該使用 swap 會導致效能降低。
$ sudo swapoff -a
$ sudo swapon -s
要保持永久效果至 fstab 將 swap 項目註解
$ sudo vim /etc/fstab
...
#/dev/mapper/ubuntu--vg-swap_1 none swap sw 0 0
Prerequisites for Kubernetes
$ sudo apt-get update
$ sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
還需要安裝 Docker 這邊不說明。可至官方查看
Installing Kubernetes
$ sudo su -c "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -"
創建 Kubernetes 儲存庫檔案
$ sudo vim /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
更新 cache
$ sudo apt update
$ sudo apt-get install -y kubelet kubeadm kubectl
- kubelet
- kubeadm
- kubectl
上述步驟三台主機都需要設置。
Initialize the Kubernetes Cluster
在 master 主機執行以下
$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.134.131 --kubernetes-version v1.18.8
# --apiserver-advertise-address: Api server 服務 IP
# --apiserver-bind-port: Api server 服務 port，預設 6443
# --pod-network-cidr: 分配 Pod IP 範圍 (CIDR)
# --kubernetes-version: 指定 Kubernetes 版本
...
[mark-control-plane] Marking the node k8smaster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 7zd2c5.w9825shkwsa7vjat
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.134.131:6443 --token av0pcs.izdosjqm3csj1par \
--discovery-token-ca-cert-hash sha256:320751ce55f3163938653d85bfb4ee44104cabc092bed010d13fcab1d384042c
以一般用戶身份運行以下。讓 kubectl 可連到 Kubernetes，將設定複製到 $HOME/.kube/config 底下，kubectl 執行時會到該目錄下取得 Api server 等相關資訊。
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
如果給的加入 k8s Node Token 遺失，可透過以下來獲取
$ kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
av0pcs.izdosjqm3csj1par 23h 2020-08-19T09:21:36Z authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
上述如果要重置可用 kubeadm reset 來完成，會移除相關資訊。
TLS 錯誤參考 troubleshooting-kubeadm
查看節點狀況
發現 master 的狀態沒 ready，其中影響是 DNS 元件。
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady master 10m v1.18.8
前面步驟完成後，在 master 節點上，預設會在 default namespace 上，而 kubernetes 上的預設安裝元件會在 kube-system 上。
$ kubectl get namespace
NAME STATUS AGE
default Active 14m
kube-node-lease Active 14m
kube-public Active 14m
kube-system Active 14m
切換至 kube-system 上查看。
$ kubectl -n kube-system get pods # -n 切換 namespace
NAME READY STATUS RESTARTS AGE
coredns-66bff467f8-5pfbz 0/1 Pending 0 11m
coredns-66bff467f8-v4txv 0/1 Pending 0 11m
etcd-master 1/1 Running 0 11m
kube-apiserver-master 1/1 Running 0 11m
kube-controller-manager-master 1/1 Running 0 11m
kube-proxy-fm749 1/1 Running 0 11m
kube-scheduler-master 1/1 Running 0 11m
而 DNS 是因為系統設計需求，需要安裝 CNI 提供網路解決，這邊為官網資訊。
Deploy Network for Pod
先前已經定義了一個 pod 網路，但是，應該為網路創建一個 namespace。可依照環境選擇適當的 Kubernetes 的 CNI plugin。這邊使用 Flannel，Flannel 將為 Kubernetes（k8s）提供overlay network。若想要自定義的 CIDR，需更改 flannel 中的 net-conf.json 的 Network 並用新的 CIDR init。
藉由 Flannel YAML 創建 Pod 網路。
$ wget https://raw.githubusercontent.com/coreos/flannel/32a765fd19ba45b387fdc5e3812c41fff47cfd55/Documentation/kube-flannel.yml
$ kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel configured
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg unchanged
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
提供網路後，DNS 元件已經運行起來。
$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
coredns-66bff467f8-5pfbz 1/1 Running 0 18m
coredns-66bff467f8-v4txv 1/1 Running 0 18m
etcd-master 1/1 Running 0 18m
kube-apiserver-master 1/1 Running 0 18m
kube-controller-manager-master 1/1 Running 0 18m
kube-flannel-ds-amd64-gljsn 1/1 Running 0 55s
kube-proxy-fm749 1/1 Running 0 18m
kube-scheduler-master 1/1 Running 0 18m
一個叢集只能用一種 Pod 網路（除非用 multus-cni）
Verify the Nodes
檢查 nodes 狀態
$ kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready master 23m v1.18.8
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready master 23m v1.18.8 192.168.134.131 <none> Ubuntu 20.04 LTS 5.4.0-42-generic docker://19.3.12
要知道當前的 namespace。在建立 Kubernetes 集群時，預設下創建以下所有 namespace。
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-5pfbz 1/1 Running 0 23m
kube-system coredns-66bff467f8-v4txv 1/1 Running 0 23m
kube-system etcd-master 1/1 Running 0 24m
kube-system kube-apiserver-master 1/1 Running 0 24m
kube-system kube-controller-manager-master 1/1 Running 0 24m
kube-system kube-flannel-ds-amd64-gljsn 1/1 Running 0 6m20s
kube-system kube-proxy-fm749 1/1 Running 0 23m
kube-system kube-scheduler-master 1/1 Running 0 24m
Join Workers with K8s Master
在前面初始化時，最後給了 Token。透過 Token，將 worker 與 master 連接起來。以下針對每台 worker 進行加入的操作。
$ sudo kubeadm join 192.168.134.131:6443 --token av0pcs.izdosjqm3csj1par --discovery-token-ca-cert-hash sha256:320751ce55f3163938653d85bfb4ee44104cabc092bed010d13fcab1d384042c
...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
再加入節點時輸出錯誤嘗試執行以下，重新嘗試
$ sudo kubeadm reset
Verify the Nodes
狀態已 Ready 顯示表示成功
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 33m v1.18.8
node01 Ready <none> 4m2s v1.18.8
node02 Ready <none> 52s v1.18.8
Additional information
$ sudo vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf # 是透過 kubeadm 去架設 kubernetes cluster，所以設定檔可以置這邊修改。可參考下面"kubeadm 與 kubelet 溝通"