008.Kubernetes二进制部署Nginx实现高可用
一 Nginx代理实现kube-apiserver高可用
1.1 Nginx实现高可用
基于 nginx 代理的 kube-apiserver 高可用方案。
控制节点的 kube-controller-manager、kube-scheduler 是多实例部署，所以只要有一个实例正常，就可以保证高可用；
集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver， kube-dns 会自动解析出多个 kube-apiserver 节点的 IP，所以也是高可用的；
在每个节点起一个 nginx 进程，后端对接多个 apiserver 实例，nginx 对它们做健康检查和负载均衡；
kubelet、kube-proxy、controller-manager、scheduler 通过本地的 nginx（监听 127.0.0.1）访问 kube-apiserver，从而实现 kube-apiserver 的高可用；
从而基于 nginx 4 层透明代理功能实现 K8S 节点( master 节点和 worker 节点)高可用访问 kube-apiserver 。
1.2 下载编译Nginx
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# wget http://nginx.org/download/nginx-1.15.3.tar.gz 3 [root@k8smaster01 work]# tar -xzvf nginx-1.15.3.tar.gz 4 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/ 5 [root@k8smaster01 nginx-1.15.3]# mkdir nginx-prefix 6 [root@k8smaster01 nginx-1.15.3]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module 7 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/ 8 [root@k8smaster01 nginx-1.15.3]# make && make install
解释：
-
–with-stream：开启 4 层透明转发(TCP Proxy)功能；
- –without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小。
- [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/
- [root@k8smaster01 nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v
1.3 验证编译后的Nginx
1 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3 2 [root@k8smaster01 nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v 3 nginx version: nginx/1.15.3 4 [root@k8smaster01 nginx-1.15.3]# ldd ./nginx-prefix/sbin/nginx #查看 nginx 动态链接的库 5 linux-vdso.so.1 => (0x00007ffdda980000) 6 libdl.so.2 => /lib64/libdl.so.2 (0x00007feb37300000) 7 libpthread.so.0 => /lib64/libpthread.so.0 (0x00007feb370e4000) 8 libc.so.6 => /lib64/libc.so.6 (0x00007feb36d17000) 9 /lib64/ld-linux-x86-64.so.2 (0x00007feb37504000)
提示：由于只开启了 4 层透明转发功能，所以除了依赖 libc 等操作系统核心 lib 库外，没有对其它 lib 的依赖(如 libz、libssl 等)，以便达到精简编译的目的。
1.4 安装和部署Nginx
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin} 7 done #创建Nginx目录 8 [root@k8smaster01 ~]# cd /opt/k8s/work 9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 10 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 11 do 12 echo ">>> ${master_ip}" 13 scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx root@${master_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx 14 ssh root@${master_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*" 15 ssh root@${master_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}" 16 done #分发Nginx二进制
1.5 配置Nginx 四层透明转发
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kube-nginx.conf <<EOF 3 worker_processes 1; 4 5 events { 6 worker_connections 1024; 7 } 8 9 stream { 10 upstream backend { 11 hash $remote_addr consistent; 12 server 172.24.8.71:6443 max_fails=3 fail_timeout=30s; 13 server 172.24.8.72:6443 max_fails=3 fail_timeout=30s; 14 server 172.24.8.73:6443 max_fails=3 fail_timeout=30s; 15 } 16 17 server { 18 listen 127.0.0.1:8443; 19 proxy_connect_timeout 1s; 20 proxy_pass backend; 21 } 22 } 23 EOF 24 [root@k8smaster01 ~]# cd /opt/k8s/work 25 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 26 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 27 do 28 echo ">>> ${master_ip}" 29 scp kube-nginx.conf root@${master_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf 30 done #分发Nginx四层透明代理配置文件
1.6 配置Nginx system
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kube-nginx.service <<EOF 3 [Unit] 4 Description=kube-apiserver nginx proxy 5 After=network.target 6 After=network-online.target 7 Wants=network-online.target 8 9 [Service] 10 Type=forking 11 ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t 12 ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx 13 ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload 14 PrivateTmp=true 15 Restart=always 16 RestartSec=5 17 StartLimitInterval=0 18 LimitNOFILE=65536 19 20 [Install] 21 WantedBy=multi-user.target 22 EOF
1.7 分发Nginx systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp kube-nginx.service root@${master_ip}:/etc/systemd/system/ 7 done
二 启动并验证
2.1 启动Nginx
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx" 7 done
2.2 检查Nginx服务
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 ssh root@${master_ip} "systemctl status kube-nginx |grep 'Active:'" 7 done
版权声明：本文为itzgr原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。