有幾個潛在的issue，GitOps有其應對方式
1.手動佈署很難追蹤問題 => kubernets可用yaml檔放入Git維護
2.pipeline由GUI設定很方便，但運行的真正狀態很難由GUI全部觀察到
3.權限控管也是一難題

GitOps概念說明如下

一.GitOps三大核心概念

• Audited (稽核性) change managment of source code: 誰在何時做什麼變動均可被查到
• Declarative (宣告式) data definition of systems: 如kubernetes 的yaml佈署檔
• Control loop Configuaration managment of systems: 讓當前狀態符合預期狀態

二.GitOps四大原則

• The entire system described declaratively (宣告式)
• The canonial (典範) desired system state versioned in GitOps: 系統狀態的各版本變動也會保存在Git內供追蹤
• Approved changes that can be automatically applied to the system: 認可的改變可自動佈署至系統內
• Software agents to ensure correctness & alert an divergence: 軟體代理可確保正確性及警告、提示不一致性

三.GitOps主要流程說明

• Git Repository分為兩部份
  • 程式原始碼 => 放Application Repo
  • 設定檔 => 放Config Repo
• Git版本已不用latest而用commit hash tag取代
• 以下架構只是概念性的描述，實際依應用情境會有所調整

參考網址: https://openpracticelibrary.com/practice/gitops/

DevOps指的是Develop & Operation互相合作，主要說明如下

一.DevOps文化
1.快速佈署、快速回饋
2.試水溫

二.DevOps基礎
1.Source code管理
2.Continuous integration (CI 持續整合)
3.Continuous delivery (CD 持續整合持續佈署)
4.Monitoring & feedback (監控及回饋)
5.Rapid Innvoation (快速創新)

三.DevOps目標
1.Improved deployment frequency (提高佈署頻率)
2.Faster time to market (更快的上市時間)
3.Lower Failure rate of new releases (新版本失敗率低)
4.Shortened lead time between fixes (縮短修復間隔時間)
5.Faster mean time to recovery (更快的平均恢復時間)

四.DevOps以kubernetes(k8s)實現的可行系統架構

終於略過前一篇說明的防火牆的阻擋，今天來記錄ubuntu v21.04安裝k8s(kubernetes) v1.21.2安裝成功的步驟

ubuntu v21.04安裝k8s(kubernetes) v1.21.2遇到的問題及處理方式

1.安裝套件apt-transport-https、ca-certificates、curl

sudo apt-get update
遇到網站憑證問題，故加上--allow....設定
sudo apt-get update --allow-unauthenticated --allow-insecure-repositories
sudo apt-get install -y apt-transport-https ca-certificates curl

2.安裝kubelet kubeadm kubectl
1) 設定https://packages.cloud.google.com網站憑證

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/kubernetes-archive-keyring.gpg > /dev/null

2) 設定kubernetes安裝apt

echo "deb [arch=amd64 trusted=yes allow-insecure=yes allow-weak=yes allow-downgrade-to-insecure=yes check-valid-until=no] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
或
echo "deb [arch=amd64 trusted=yes allow-insecure=yes allow-weak=yes allow-downgrade-to-insecure=yes check-valid-until=no] https://packages.cloud.google.com/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

註:出現如下訊息

E: 無法取得 https://apt.kubernetes.io/dists/kubernetes-xenial/main/binary-amd64/Packages，Certificate verification failed: The certificate is NOT trusted. The certificate chain uses insecure algorithm. Could not handshake: Error in the certificate verification. [IP: 34.107.204.206 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.

解法: vi etc/apt/apt.conf.d/99verify-peer.conf，若為true，改為false

Acquire { https::Verify-Peer false }

3.swap改由k8s控管，故要關閉swap

swapoff -a
sed -e '/swap/ s/^#*/#/' -i /etc/fstab
free -m

4.初始master，以建立k8s cluster
1) 建置指令
sudo kubeadm init –-pod-network-cidr 10.5.0.0/16 -–v=5

終於看到成功的畫面
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.2.15:6443 –token xyovgm.p0z428rgi7flwi6a \
–discovery-token-ca-cert-hash sha256:8893a38d20d191ad63714b030926cb904bc468290710237d49d6d11f14a92b48

2)此段需執行

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

3)加node使用此token請記錄下來

kubeadm join 10.0.2.15:6443 –token xyovgm.p0z428rgi7flwi6a \
–discovery-token-ca-cert-hash sha256:8893a38d20d191ad63714b030926cb904bc468290710237d49d6d11f14a92b48

5.設定自動補齊功能

sudo apt-get install -y bash-completion
echo “source /etc/bash_completion" >> ~/.bashrc
echo “source <(kubectl completion bash)" >> ~/.bashrc
source <(kubectl completion bash)

6.若無足夠主機建置

kubectl taint nodes --all node-role.kubernetes.io/master-

收到結果: node/ubuntu-virtualbox untainted

7.列出node數
kubectl get nodes
kubectl get nodes -o wide
收到結果: (STATUS為NOT Ready)

8.列出node內容
kubectl describe node ubuntu-virtualbox

9.設定overlay網路: 用flannel
1)wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

2)vi kube-flannel.yml

修改
net-conf.json: |
{
“Network": “10.5.0.0/16″,
“Backend": {
“Type": “vxlan"
}
}

其中Network值需同kubeadm init時設定的pod-network-cidr一致

3)建立overlay網路
kubectl apply -f kube-flannel.yml
如下圖結果:

4)查看pod狀況
kubectl get pods –all-namespaces
如下圖結果:

註: flannel image因公司擋quay.io這個網址無法下載，故此步驟一直停在image Err的STATUS，後來先在外部將image拉下來，再上傳到內部server，步驟如下
外部電腦:

docker pull quay.io/coreos/flannel:v0.14.0-amd64
docker image save flannel -o flannel.tar

-- 若已run在container內可以用此方式
docker export --output="flannel.jar" flannel

內部Server:

docker load -i flannel.tar

5) 列出node數
kubectl get nodes -o wide
收到結果: (STATUS為Ready)

10.建立nginx pod

kubectl run nginx --image=nginx

註: kubernets 1.18以後請用以下指令

kubectl create deployment nginx-web --image=nginx

這樣才會連deployment及replicaset一同建立，以kubectl get all查看

kubectl get pods -o wide
如下圖結果:

11.改nodeport

kubectl get deployment -o yaml nginx > nginx.yaml
kubectl expose -f nginx.yaml --port=80 --type NodePort
或
kubectl edit svc nginx

12.查看對應的服務port
kubectl get svc
如下圖結果:

13.佈署一個nginx deployment
參考網址: https://kubernetes.io/zh/docs/tasks/run-application/run-stateless-application-deployment/

1) vi nginx_deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
– name: nginx
image: nginx:latest
ports:
– containerPort: 80

2)建立deployment
kubectl apply -f nginx_deployment.yaml

3)查看deployment、pod、svc
kubectl describe deployment nginx-deployment
kubectl get pods -o wide
kubectl get svc -o wide
如下圖結果:

4)列出deployment建立的Pods
kubectl get pods -l app=nginx
如下圖結果:

5)取得某個pod的描述
kubectl describe pod <pod-name>

14.增加nginx deployment pod由2調整成3
1)調整nginx_deployment.yaml內的replicas為3

2)重建deployment

kubectl apply -f nginx_deployment.yaml

3)查deployment的Pods數量:已變為3
kubectl get pods -l app=nginx
如下圖結果:

15.降低nginx deployment pod由3調整成2
kubectl scale deployment nginx-deployment –replicas=2
查看結果
kubectl get pods -l app=nginx
如下圖結果:

於ubuntu安裝k8s，以下是我安裝遇到的問題及我的解法，
其中包含公司防火牆無法連到某些網站我的解法說明，
因為v1.21.2還有未解問題，會降版及改CentOS再試看看，
所以相關安裝還會有新文更新試驗結果，敬請期待…

一.方法一:

在公司內部，安裝遇到公司防火牆擋連https://packages.cloud.google.com/apt/，故最後改用方法二乃至方法三
1.安裝套件apt-transport-https、ca-certificates、curl
sudo apt-get update

=> 遇到網站憑證問題，故加上–allow….設定
sudo apt-get update –allow-unauthenticated –allow-insecure-repositories

sudo apt-get install -y apt-transport-https ca-certificates curl

2.安裝kubelet kubeadm kubectl
1) 設定https://packages.cloud.google.com網站憑證
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg –dearmor | sudo tee /usr/share/keyrings/kubernetes-archivekeyring.gpg > /dev/null
2) 設定kubernetes安裝apt
echo “deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

=> 若上述指令執行仍有問題，建議在deb […]加上trusted=yes allow-insecure=yes allow-weak=yes allow-downgrade-to-insecure=yes check-valid-until=no試看看
sudo apt-get update –allow-unauthenticated –allow-insecure-repositories
sudo apt-get install -y kubelet kubeadm kubectl
=> 此步驟因防火牆擋連https://packages.cloud.google.com/apt/無法下載，故改試方法二

二.方法二:

用snap工具安裝 (若未安裝snap，要執行apt install snap)
參考網址: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/
1.安裝kubelet kubeadm kubectl
snap install kubectl –classic
kubectl version –client

snap install kubeadm –classic
kubeadm version –client

snap install kubelet –classic
kubelet version –client
kubelet

2.swap改由k8s控管，故要關閉swap
swapoff -a
sed -e ‘/swap/ s/^#*/#/’ -i /etc/fstab
free -m

3.初始master，以建立k8s cluster
kubeadm init –pod-network-cidr 10.5.0.0/16

=> 出現以下錯誤，不曉得如何解，故改用方法三
root@ubuntu-VirtualBox:~/snap# kubeadm init –pod-network-cidr 10.5.0.0/16
[init] Using Kubernetes version: v1.21.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected “cgroupfs" as the Docker cgroup driver. The recommended driver is “systemd". Please follow the guide at https://kuberne tes.io/docs/setup/cri/
[WARNING FileExisting-socat]: socat not found in system path
[WARNING Service-Kubelet]: kubelet service is not enabled, please run ‘systemctl enable kubelet.service’
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileExisting-conntrack]: conntrack not found in system path
[preflight] If you know what you are doing, you can make a check non-fatal with `–ignore-preflight-errors=…`
To see the stack trace of this error execute with –v=5 or higher
root@ubuntu-VirtualBox:~/snap# systemctl enable kubelet.service
Failed to enable unit: Unit file kubelet.service does not exist.

此項解法: [ERROR FileExisting-conntrack]: conntrack not found in system path
=> apt-get install conntrack

三.方法三:

參考網址:
http://kimiwublog.blogspot.com/2017/05/kubernetes.html
https://milexz.pixnet.net/blog/post/228096329-%E3%80%90k8s%E3%80%91kubernetes%E7%92%B0%E5%A2%83%E6%9E%B6%E8%A8%ADby-kubeadm
https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/
https://www.downloadkubernetes.com/

1.安裝kubectl
1)curl -LO “https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

=> 遇到如下錯誤 curl: (60) SSL certificate problem: EE certificate key too weak
解法: 因ubuntu 20.04將TLS 最低版本為1.2，故認證失敗
修改/etc/ssl/openssl.cnf，在 oid_section = new_oids下增加

openssl_conf = default_conf
[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
MinProtocol = TLSv1.1
CipherString = DEFAULT@SECLEVEL=1

2)curl -LO “https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256″
echo “$(<kubectl.sha256) kubectl" | sha256sum –check

3)sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version

2.安裝kubeadm
1)curl -LO “https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubeadm"

2)curl -LO “https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubeadm.sha256″
echo “$(<kubeadm.sha256) kubeadm" | sha256sum –check

3)sudo install -o root -g root -m 0755 kubeadm /usr/local/bin/kubeadm
kubeadm version

3.安裝kubelet
1)curl -LO “https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubelet"

2)curl -LO “https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubelet.sha256″
echo “$(<kubelet.sha256) kubelet" | sha256sum –check

3)sudo install -o root -g root -m 0755 kubelet /usr/local/bin/kubelet
kubelet –version

4.swap改由k8s控管，故要關閉swap

sudo swapoff -a
sudo sed -e ‘/swap/ s/^#*/#/’ -i /etc/fstab
free -m

5.初始master，以建立k8s cluster
sudo kubeadm init –pod-network-cidr 10.5.0.0/16

出現如下錯誤:
[init] Using Kubernetes version: v1.21.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected “cgroupfs" as the Docker cgroup driver. The recommended driver is “systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING FileExisting-socat]: socat not found in system path
[WARNING Service-Kubelet]: kubelet service is not enabled, please run ‘systemctl enable kubelet.service’
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileAvailable–etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
[ERROR FileAvailable–etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
[ERROR FileAvailable–etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
[ERROR FileAvailable–etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `–ignore-preflight-errors=…`
To see the stack trace of this error execute with –v=5 or higher

解法說明:

1)[WARNING IsDockerSystemdCheck]: detected “cgroupfs" as the Docker cgroup driver. The recommended driver is “systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
=> 參考如下第6項7)、8)說明

2)[WARNING Service-Kubelet]: kubelet service is not enabled, please run ‘systemctl enable kubelet.service’
=> 參考如下第6項8)說明

3)其中以下錯誤是因此環境之前有執行過，故需用kubeadm reset清掉之前的設定
[ERROR FileAvailable–etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
[ERROR FileAvailable–etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
[ERROR FileAvailable–etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
[ERROR FileAvailable–etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists

4)To see the stack trace of this error execute with –v=5 or higher

6.參考 https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/說明，先將整個環境再重新view一次
1)確認MAC address and product_uuid在每個node是unique，因第一個node尚未建，所以此步驟可略過

ifconfig -a
sudo cat /sys/class/dmi/id/product_uuid

2)Check network adapters

3)使用iptables並確認有載入bridged traffic模組

lsmod | grep br_netfilter
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sudo sysctl –system

4)確認相關ports是否有被佔用

A.ontrol-plane node(s)

Protocol Direction Port Range Purpose Used By
TCP Inbound 6443* Kubernetes API server All
TCP Inbound 2379-2380 etcd server client API kube-apiserver, etcd
TCP Inbound 10250 kubelet API Self, Control plane
TCP Inbound 10251 kube-scheduler Self
TCP Inbound 10252 kube-controller-manager Self

B.Worker node(s)

Protocol Direction Port Range Purpose Used By
TCP Inbound 10250 kubelet API Self, Control plane
TCP Inbound 30000-32767 NodePort Services? All

5)可使用以下runtime環境

• Docker /var/run/dockershim.sock
• containerd /run/containerd/containerd.sock
• CRI-O /var/run/crio/crio.sock

6)取得kubeadm設定檔
sudo kubeadm config images pull

註:kubeadm init會到到此行提醒[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’

7)將docker的cgroup driver改為systemd
參考網址: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/

A.sudo mkdir /etc/docker (若不存在此路徑才執行)

cat <<EOF | sudo tee /etc/docker/daemon.json
{
“exec-opts": [“native.cgroupdriver=systemd"],
“log-driver": “json-file",
“log-opts": {
“max-size": “100m"
},
“storage-driver": “overlay2″
}
EOF

B.重啟docker服務
systemctl daemon-reload && systemctl restart docker && systemctl enable kubelet.service

C.確認是否使用cgroup
docker info |grep “Cgroup"
=> Cgroup Driver: systemd

8)建立kubelet服務，並將其cgroup driver設定為systemd
A.sudo vi /etc/systemd/system/kubelet.service

[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/home/
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/local/bin/kubelet
–v=2 \
–cgroup-driver=systemd \
–runtime-cgroups=/systemd/system.slice \
–kubelet-cgroups=/systemd/system.slice
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

B.重啟kubelet服務
daemon-reload && systemctl restart kubelet

C.若無kubelet.service可參考設定，可下載這一份試看看
curl -sSL “https://raw.githubusercontent.com/kubernetes/release/master/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed “s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service

D.查看kubelet.service啟動的LOG，若啟動失敗可先參考此份的原因
journalctl -xeu kubelet => 看啟動Log 或 journalctl -f -u kubelet

E.若無法設定好cgroup driver可參考此

https://www.cnblogs.com/hellxz/p/kubelet-cgroup-driver-different-from-docker.html
Check on the worker nodes file /var/lib/kubelet/kubeadm-flags.env and in KUBELET_KUBEADM_ARGS if you have –cgroup-driver=cgroupfsflag. Changed it to systemd and kubelet will start working again.

9)新增或設定檔案 kubeadm-config.yaml => 但我試的結果是跟kubelet的cgroup設定為systemd沒有影響

kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
kubernetesVersion: v1.21.0
—
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

7.再次初始master，以建立k8s cluster
1)需用kubeadm reset清掉之前的設定
kubeadm reset

2)取得預設定檔
kubeadm config images pull

3)確認kubelet.service可以啟動成功
systemctl start kubelet.service 或 systemctl restart kubelet.service
systemctl status kubelet.service

4)清除所有虛擬服務
ipvsadm –clear

5)執行以下仍失敗
sudo kubeadm init –pod-network-cidr 10.5.0.0/16
sudo kubeadm init –kubernetes-version=v1.21.2 –pod-network-cidr=10.244.0.0/16 –service-cidr=10.96.0.0/12 –ignore-preflight-errors=Swap –v=5
sudo kubeadm init –kubernetes-version=v1.21.2 –pod-network-cidr=10.244.0.0/16 –service-cidr=10.96.0.0/12 –ignore-preflight-errors=all –v=5
sudo kubeadm init –config kubeadm-config.yml –v=5

出現的錯誤: Error execution phase wait-control-plane

=> 執行以下仍無果，暫時試到這
sudo vi /etc/ufw/sysctl.conf
# 2021.07.08
加上
net/bridge/bridge-nf-call-ip6tables = 1
net/bridge/bridge-nf-call-iptables = 1
net/bridge/bridge-nf-call-arptables = 1

參考網址:
https://docs.nvidia.com/datacenter/cloud-native/kubernetes/install-k8s.html
https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/kubelet-integration/
https://www.cnblogs.com/horizonli/p/10855666.html
https://www.qikqiak.com/k8s-book/docs/16.%E7%94%A8%20kubeadm%20%E6%90%AD%E5%BB%BA%E9%9B%86%E7%BE%A4%E7%8E%AF%E5%A2%83.html
https://jimmysong.io/kubernetes-handbook/cloud-native/cloud-native-local-quick-start.html

新安裝的ubuntu預設沒有ssh功能，故要手動來安裝，步驟如下
1.安裝openssh server
1) 安裝
sudo apt-get install openssh-server -y
2) 用ssh登入
ssh username@ip 或 username@hostname 或 ssh ip、ssh hostname

2.若要改ssh預設port
1) 設定開啟的port
sudo apt-get install nano -y

Port 22

Port 1337
2) 設定防火牆開放另一個port
sudo ufw allow 1337
sudo service ssh restart 或 sudo systemctl restart ssh
3) 用ssh登入
ssh username@ip -p1337 或 username@hostname -p1337 或 ssh ip -p1337、ssh hostname -p1337