注：关于ceph、kubernetes集群的部署在此不声明，相信搜到本篇博文，你一定对ceph、kubernetes的部署环节手刃有余。

注：本篇博文牵扯到的技术点有：ceph、kubernetes、harbor、jenkins、traefik

  1. #ceph -s      //查看ceph集群状态
  2. #ceph osd pool create jenkins 128      //创建pool  建议每个pool存放的是通类应用
  3. #ceph auth get-or-create client.jenkins mon 'allow r' osd 'allow class-read, allow rwx pool=jenkins' -o ceph.client.jenkins.keyring    //创建普通用户管理对应pool

注意：ceph集群的状态要先调试成ok.

image-20200315164133224

image-20200315164335421

  1. # cat ~/.docker/config.json |base64 -w 0        //node节点访问私有仓库的认证
  2. ewoJImF1dGhzIjogewoJCSJoYXJib3IubGludXguY29tIjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NmVtbHpaV1psYVhwb2RRPT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjUgKGxpbnV4KSIKCX0KfQ==
  3. # cat secret_harbor.yaml 
  4. ##########################################################################
  5. #Author:                     zisefeizhu
  6. #QQ:                         2********0
  7. #Date:                       2020-02-19
  8. #FileName:                   secret_harbor.yaml
  9. #URL:                        https://www.cnblogs.com/zisefeizhu/
  10. #Description:                The test script
  11. #Copyright (C):              2020 All rights reserved
  12. ###########################################################################
  13. apiVersion: v1
  14. kind: Secret
  15. metadata:
  16.   name: k8s-harbor-login
  17. type: kubernetes.io/dockerconfigjson
  18. data:
  19.   .dockerconfigjson: ewoJImF1dGhzIjogewoJCSJoYXJib3IubGludXguY29tIjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NmVtbHpaV1psYVhwb2RRPT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjUgKGxpbnV4KSIKCX0KfQ==
  20. # kubectl create -f secret_harbor.yaml 
  21. secret/login created
  22. # kubectl get secret
  23. NAME                          TYPE                                  DATA   AGE
  24. ceph-admin-secret             kubernetes.io/rbd                     1      3d16h
  25. ceph-kube-secret              kubernetes.io/rbd                     1      3d16h

注意：前提是node节点可以访问到harbor，关于这部分可以参考我的这篇博文：https://www.cnblogs.com/zisefeizhu/p/12329864.html

image-20200315162612795

  1. # pwd
  2. /data/k8s/jenkins      //单个服务单个目录
  3.  
  4. # cat namespace.yaml    //单个服务单个名称空间，便于管理
  5. ##########################################################################
  6. #Author:                     zisefeizhu
  7. #QQ:                         2********0
  8. #Date:                       2020-03-09
  9. #FileName:                   namespace.yaml
  10. #URL:                        https://www.cnblogs.com/zisefeizhu/
  11. #Description:                The test script
  12. #Copyright (C):              2020 All rights reserved
  13. ###########################################################################
  14. apiVersion: v1
  15. kind: Namespace
  16. metadata:
  17.   name: jenkins
  18.   labels:
  19.     name: jenkins
  21. #kubernetes结合ceph需要使用第三方插件
  22. # cat external-storage-rbd-provisioner.yaml   
  23. ##########################################################################
  24. #Author:                     zisefeizhu
  25. #QQ:                         2********0
  26. #Date:                       2020-03-09
  27. #FileName:                   external-storage-rbd-provisioner.yaml
  28. #URL:                        https://www.cnblogs.com/zisefeizhu/
  29. #Description:                The test script
  30. #Copyright (C):              2020 All rights reserved
  31. ###########################################################################
  32. apiVersion: v1
  33. kind: ServiceAccount
  34. metadata:
  35.   name: rbd-provisioner
  36.   namespace: jenkins
  37. ---
  38. kind: ClusterRole
  39. apiVersion: rbac.authorization.k8s.io/v1
  40. metadata:
  41.   name: rbd-provisioner
  42. rules:
  43.   - apiGroups: [""]
  44.     resources: ["persistentvolumes"]
  45.     verbs: ["get", "list", "watch", "create", "delete"]
  46.   - apiGroups: [""]
  47.     resources: ["persistentvolumeclaims"]
  48.     verbs: ["get", "list", "watch", "update"]
  49.   - apiGroups: ["storage.k8s.io"]
  50.     resources: ["storageclasses"]
  51.     verbs: ["get", "list", "watch"]
  52.   - apiGroups: [""]
  53.     resources: ["events"]
  54.     verbs: ["create", "update", "patch"]
  55.   - apiGroups: [""]
  56.     resources: ["endpoints"]
  57.     verbs: ["get", "list", "watch", "create", "update", "patch"]
  58.   - apiGroups: [""]
  59.     resources: ["services"]
  60.     resourceNames: ["kube-dns"]
  61.     verbs: ["list", "get"]
  62. ---
  63. kind: ClusterRoleBinding
  64. apiVersion: rbac.authorization.k8s.io/v1
  65. metadata:
  66.   name: rbd-provisioner
  67. subjects:
  68.   - kind: ServiceAccount
  69.     name: rbd-provisioner
  70.     namespace: jenkins
  71. roleRef:
  72.   kind: ClusterRole
  73.   name: rbd-provisioner
  74.   apiGroup: rbac.authorization.k8s.io
  76. ---
  77. apiVersion: rbac.authorization.k8s.io/v1
  78. kind: Role
  79. metadata:
  80.   name: rbd-provisioner
  81.   namespace: jenkins
  82. rules:
  83. - apiGroups: [""]
  84.   resources: ["secrets"]
  85.   verbs: ["get"]
  86. ---
  87. apiVersion: rbac.authorization.k8s.io/v1
  88. kind: RoleBinding
  89. metadata:
  90.   name: rbd-provisioner
  91.   namespace: jenkins
  92. roleRef:
  93.   apiGroup: rbac.authorization.k8s.io
  94.   kind: Role
  95.   name: rbd-provisioner
  96. subjects:
  97. - kind: ServiceAccount
  98.   name: rbd-provisioner
  99.   namespace: jenkins
  101. ---
  102. apiVersion: apps/v1
  103. kind: Deployment
  104. metadata:
  105.   name: rbd-provisioner
  106.   namespace: jenkins
  107. spec:
  108.   replicas: 1
  109.   selector:
  110.     matchLabels:
  111.       app: rbd-provisioner
  112.   strategy:
  113.     type: Recreate
  114.   template:
  115.     metadata:
  116.       labels:
  117.         app: rbd-provisioner
  118.     spec:
  119.       containers:
  120.       - name: rbd-provisioner
  121.         image: "harbor.linux.com/rbd/rbd-provisioner:latest"
  122.         imagePullPolicy: IfNotPresent
  123.         env:
  124.         - name: PROVISIONER_NAME
  125.           value: ceph.com/rbd
  126.       imagePullSecrets: 
  127.         - name: k8s-harbor-login
  128.       serviceAccount: rbd-provisioner
  130. #敏感数据创建secret，这没什么可说的
  131. # cat ceph-jenkins-secret.yaml 
  132. ##########################################################################
  133. #Author:                     zisefeizhu
  134. #QQ:                         2********0
  135. #Date:                       2020-03-09
  136. #FileName:                   ceph-wordpress-secret.yaml
  137. #URL:                        https://www.cnblogs.com/zisefeizhu/
  138. #Description:                The test script
  139. #Copyright (C):              2020 All rights reserved
  140. ###########################################################################
  141. apiVersion: v1
  142. kind: Secret
  143. metadata:
  144.   name: ceph-admin-secret
  145.   namespace: jenkins
  146. data:
  147.   key: QVFBZ2pXVmVGOVJISkJBQTBTUDRoOTVZYVdHNEN6TzNaUWtIdVE9PQ==
  148. type: kubernetes.io/rbd
  149. ---
  150. apiVersion: v1
  151. kind: Secret
  152. metadata:
  153.   name: ceph-jenkins-secret
  154.   namespace: jenkins
  155. data:
  156.   key: QVFEUjRHWmVNUFJpRnhBQUQ1Zlg1UG9JRUNkMG85Qk5kVzN5SUE9PQ==
  157. type: kubernetes.io/rbd
  159. # cat ceph-jenkins-storageclass.yaml      //
  160. ##########################################################################
  161. #Author:                     zisefeizhu
  162. #QQ:                         2********0
  163. #Date:                       2020-03-09
  164. #FileName:                   ceph-wordpress-storageclass.yaml
  165. #URL:                        https://www.cnblogs.com/zisefeizhu/
  166. #Description:                The test script
  167. #Copyright (C):              2020 All rights reserved
  168. ###########################################################################
  169. apiVersion: storage.k8s.io/v1
  170. kind: StorageClass
  171. metadata:
  172.   name: ceph-jenkins
  173.   namespace: jenkins
  174.   annotations:
  175.     storageclass.kubernetes.io/is-default-class: "false"
  176. provisioner: ceph.com/rbd
  177. reclaimPolicy: Retain
  178. parameters:
  179.   monitors: 20.0.0.207:6789,20.0.0.208:6789,20.0.0.210:6789
  180.   adminId: admin
  181.   adminSecretName: ceph-admin-secret
  182.   adminSecretNamespace: jenkins
  183.   pool: jenkins
  184.   fsType: xfs
  185.   userId: jenkins
  186.   userSecretName: ceph-jenkins-secret
  187.   imageFormat: "2"
  188.   imageFeatures: "layering"
  189.   
  190. # cat jenkins-pvc.yaml 
  191. ##########################################################################
  192. #Author:                     zisefeizhu
  193. #QQ:                         2********0
  194. #Date:                       2020-03-10
  195. #FileName:                   jenkins-pvc.yaml
  196. #URL:                        https://www.cnblogs.com/zisefeizhu/
  197. #Description:                The test script
  198. #Copyright (C):              2020 All rights reserved
  199. ###########################################################################
  200. apiVersion: v1
  201. kind: PersistentVolumeClaim
  202. metadata:
  203.   name: jenkins-pvc
  204.   namespace: jenkins
  205.  # labels:
  206.  #   app: gitlab
  207. spec:
  208.   storageClassName: ceph-jenkins
  209.   accessModes:
  210.     - ReadWriteOnce
  211.   resources:
  212.     requests:
  213.       storage: 20Gi  
  214. # kubectl get pvc -n jenkins
  215. NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  216. jenkins-pvc   Bound    pvc-d386c125-5302-468a-8a94-a2570f0a4ca0   20Gi       RWO            ceph-jenkins   2d8h
  1. # pwd
  2. /data/k8s/jenkins
  3.  
  4. # cat jenkins.yaml     //核心资源清单
  5. ---
  6. apiVersion: apps/v1
  7. kind: StatefulSet
  8. metadata:
  9.   name: jenkins
  10.   namespace: jenkins
  11.   labels:
  12.     name: jenkins
  13. spec:
  14.   selector:
  15.     matchLabels:
  16.       name: jenkins
  17.   serviceName: jenkins
  18.   replicas: 1
  19.   updateStrategy:
  20.     type: RollingUpdate
  21.   template:
  22.     metadata:
  23.       name: jenkins
  24.       labels:
  25.         name: jenkins
  26.     spec:
  27.       terminationGracePeriodSeconds: 10
  28.       serviceAccountName: jenkins
  29.       #登陆私有仓库harbor认证
  30.       imagePullSecrets: 
  31.         - name: k8s-harbor-login
  32.       containers:
  33.         - name: jenkins
  34.           image: harbor.linux.com/dev/jenkins:lts
  35.           imagePullPolicy: IfNotPresent
  36.           ports:
  37.             - containerPort: 8080
  38.             - containerPort: 50000
  39.           resources:
  40.             limits:
  41.               cpu: 1
  42.               memory: 1Gi
  43.             requests:
  44.               cpu: 0.5
  45.               memory: 500Mi
  46.           env:
  47.             - name: LIMITS_MEMORY
  48.               valueFrom:
  49.                 resourceFieldRef:
  50.                   resource: limits.memory
  51.                   divisor: 1Mi
  52.             - name: JAVA_OPTS
  53.               # value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
  54.               value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
  55.           volumeMounts:
  56.             - name: jenkins-home
  57.               mountPath: /var/jenkins_home
  58.           livenessProbe:
  59.             httpGet:
  60.               path: /login
  61.               port: 8080
  62.             initialDelaySeconds: 60
  63.             timeoutSeconds: 5
  64.             failureThreshold: 12 # ~2 minutes
  65.           readinessProbe:
  66.             httpGet:
  67.               path: /login
  68.               port: 8080
  69.             initialDelaySeconds: 60
  70.             timeoutSeconds: 5
  71.             failureThreshold: 12 # ~2 minutes
  72.       securityContext:
  73.         fsGroup: 1000
  74.       volumes:
  75.       - name: jenkins-home
  76.         persistentVolumeClaim:
  77.           claimName: jenkins-pvc
  78. ---
  79. apiVersion: v1
  80. kind: Service
  81. metadata:
  82.   name: jenkins
  83.   namespace: jenkins
  84. spec:
  85.   # type: LoadBalancer
  86.   selector:
  87.     name: jenkins
  88.   # ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s >=1.7)
  89.   #externalTrafficPolicy: Local
  90.   ports:
  91.     -
  92.       name: http
  93.       port: 80     
  94.       targetPort: 8080
  95.       protocol: TCP
  96.     -
  97.       name: agent
  98.       port: 50000
  99.       protocol: TCP
  101. # cat jenkins-serviceaccount.yaml 
  102. ##########################################################################
  103. #Author:                     zisefeizhu
  104. #QQ:                         2********0
  105. #Date:                       2020-03-10
  106. #FileName:                   jenkins-serviceaccount.yaml
  107. #URL:                        https://www.cnblogs.com/zisefeizhu/
  108. #Description:                The test script
  109. #Copyright (C):              2020 All rights reserved
  110. ###########################################################################
  111. # In GKE need to get RBAC permissions first with
  112. # kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=<user-name>|--group=<group-name>]
  114. ---
  115. apiVersion: v1
  116. kind: ServiceAccount
  117. metadata:
  118.   name: jenkins
  119.   namespace: jenkins
  120. ---
  121. kind: Role
  122. apiVersion: rbac.authorization.k8s.io/v1beta1
  123. metadata:
  124.   name: jenkins
  125.   namespace: jenkins
  126. rules:
  127. - apiGroups: [""]
  128.   resources: ["pods"]
  129.   verbs: ["create","delete","get","list","patch","update","watch"]
  130. - apiGroups: [""]
  131.   resources: ["pods/exec"]
  132.   verbs: ["create","delete","get","list","patch","update","watch"]
  133. - apiGroups: [""]
  134.   resources: ["pods/log"]
  135.   verbs: ["get","list","watch"]
  136. - apiGroups: [""]
  137.   resources: ["secrets"]
  138.   verbs: ["get"]
  140. ---
  141. apiVersion: rbac.authorization.k8s.io/v1beta1
  142. kind: RoleBinding
  143. metadata:
  144.   name: jenkins
  145.   namespace: jenkins
  146. roleRef:
  147.   apiGroup: rbac.authorization.k8s.io
  148.   kind: Role
  149.   name: jenkins
  150. subjects:
  151. - kind: ServiceAccount
  152.   name: jenkins
  153.   namespace: jenkins
  154. # kubectl get pods -n jenkins
  155. NAME                               READY   STATUS        RESTARTS   AGE
  156. jenkins-0                          1/1     Running       14         30h
  157. rbd-provisioner-5c97b9d5ff-95qwj   1/1     Running       13         2d8h
  1. # cat jenkins-ingressroute.yaml 
  2. ##########################################################################
  3. #Author:                     zisefeizhu
  4. #QQ:                         2********0
  5. #Date:                       2020-03-10
  6. #FileName:                   jenkins-ingressroute.yaml
  7. #URL:                        https://www.cnblogs.com/zisefeizhu/
  8. #Description:                The test script
  9. #Copyright (C):              2020 All rights reserved
  10. ###########################################################################
  11. apiVersion: traefik.containo.us/v1alpha1
  12. kind: IngressRoute
  13. metadata:
  14.   name: jenkins
  15.   namespace: jenkins
  16. spec:
  17.   entryPoints:
  18.     - web
  19.   routes:
  20.   #登陆域名   需要自己在主机添加hosts解析  或者自建dns也行
  21.   - match: Host(`jenkins.linux.com`)
  22.     kind: Rule
  23.     services:
  24.     - name: jenkins    #和jenkins核心资源清单一致，  name是jenkins service的name
  25.       port: 80         #jenkins pod 暴漏端口

image-20200315180713459

  1. # kubectl get pods -n jenkins
  2. NAME                              READY   STATUS    RESTARTS   AGE
  3. jenkins-0                         1/1     Running   1          18m
  4. rbd-provisioner-dbc4c8b59-grfg2   1/1     Running   1          125m
  5. # kubectl logs jenkins-0  -n jenkins
  6. VM settings:
  7.     Max. Heap Size: 1.00G
  8.     Ergonomics Machine Class: server
  9.     Using VM: OpenJDK 64-Bit Server VM
  11. #注意 jenkins首次登陆密码
  12. ac4fe3940ec145fe9104eda3ca390d0a    
  14. This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
  16. *************************************************************
  17. *************************************************************
  18. *************************************************************
  20. 2020-03-10 02:44:33.446+0000 [id=39]    INFO    hudson.model.UpdateSite#updateData: Obtained the latest update center data file for UpdateSource default
  21. 2020-03-10 02:44:36.699+0000 [id=25]    INFO    hudson.model.UpdateSite#updateData: Obtained the latest update center data file for UpdateSource default
  22. 2020-03-10 02:44:42.206+0000 [id=25]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
  23. 2020-03-10 02:44:43.044+0000 [id=39]    INFO    h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
  24. 2020-03-10 02:44:43.044+0000 [id=39]    INFO    hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1
  25. 2020-03-10 02:44:43.055+0000 [id=39]    INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 81,778 ms
  26. 2020-03-10 02:44:43.414+0000 [id=19]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running

登陆密码为：

  1. ac4fe3940ec145fe9104eda3ca390d0a

image-20200315180746691

注：本篇博文完全原创，后续将发布大量有技术的原创博文，请持续关注。
注：关于ceph集群的部署、kubernetes1.17.2高可用集群的部署可以看我的历史博文。
注：关于jenkins的使用不在本篇讲解范围内(主要是截图太多了，有空再发表吧)

版权声明：本文为zisefeizhu原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://www.cnblogs.com/zisefeizhu/p/12499084.html