kubernetes实战(九)：k8s集群动态存储管理GlusterFS及容器化GlusterFS扩容

dukuan 2018-11-14 原文

kubernetes实战(九)：k8s集群动态存储管理GlusterFS及容器化GlusterFS扩容

1、准备工作

　　所有节点安装GFS客户端

yum install glusterfs glusterfs-fuse -y

　　如果不是所有节点要部署GFS管理服务，就在需要部署的节点上打上标签

[root@k8s-master01 ~]# kubectl label node k8s-node01 storagenode=glusterfs
node/k8s-node01 labeled
[root@k8s-master01 ~]# kubectl label node k8s-node02 storagenode=glusterfs
node/k8s-node02 labeled
[root@k8s-master01 ~]# kubectl label node k8s-master01 storagenode=glusterfs
node/k8s-master01 labeled

 

2、创建GFS管理服务容器集群

　　本文采用容器化方式部署GFS，公司如有GFS集群可直接使用。

　　GFS已Daemonset的方式进行部署，保证每台需要部署GFS管理服务的Node上都运行一个GFS管理服务。

　　下载相关文件：

wget https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-client-v7.0.0.linux.amd64.tar.gz

　　创建集群：

[root@k8s-master01 kubernetes]# kubectl create -f glusterfs-daemonset.json 
daemonset.extensions/glusterfs created
[root@k8s-master01 kubernetes]# pwd
/root/heketi-client/share/heketi/kubernetes

　　注意1：此时采用的为默认的挂载方式，可使用其他磁盘当做GFS的工作目录。

　　注意2：此时创建的namespace为默认的default，按需更改

　　查看pods

[root@k8s-master01 kubernetes]# kubectl get pods -l glusterfs-node=daemonset
NAME              READY     STATUS    RESTARTS   AGE
glusterfs-5npwn   1/1       Running   0          1m
glusterfs-bd5dx   1/1       Running   0          1m
...

 

3、创建Heketi服务

　　Heketi是一个提供RESTful API管理GFS卷的框架，并能够在K8S、OpenShift、OpenStack等云平台上实现动态存储资源供应，支持GFS多集群管理，便于管理员对GFS进行操作。

　　创建Heketi的ServiceAccount对象：

[root@k8s-master01 kubernetes]# cat heketi-service-account.json 
{
  "apiVersion": "v1",
  "kind": "ServiceAccount",
  "metadata": {
    "name": "heketi-service-account"
  }
}
[root@k8s-master01 kubernetes]# kubectl create -f heketi-service-account.json 
serviceaccount/heketi-service-account created
[root@k8s-master01 kubernetes]# pwd
/root/heketi-client/share/heketi/kubernetes
[root@k8s-master01 kubernetes]# kubectl get sa
NAME                     SECRETS   AGE
default                  1         13d
heketi-service-account   1         <invalid>

　　创建Heketi对应的权限和secret

[root@k8s-master01 kubernetes]# kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=edit --serviceaccount=default:heketi-service-account
clusterrolebinding.rbac.authorization.k8s.io/heketi-gluster-admin created
[root@k8s-master01 kubernetes]# kubectl create secret generic heketi-config-secret --from-file=./heketi.json
secret/heketi-config-secret created

　　初始化部署Heketi

[root@k8s-master01 kubernetes]# kubectl create -f heketi-bootstrap.json 
secret/heketi-db-backup created
service/heketi created
deployment.extensions/heketi created
[root@k8s-master01 kubernetes]# pwd
/root/heketi-client/share/heketi/kubernetes

　　

4、设置GFS集群

[root@k8s-master01 heketi-client]# cp bin/heketi-cli /usr/local/bin/
[root@k8s-master01 heketi-client]# pwd
/root/heketi-client

[root@k8s-master01 heketi-client]# heketi-cli -v
heketi-cli v7.0.0

　　修改topology-sample，manage为GFS管理服务的Node节点主机名，storage为Node节点IP，devices为Node节点上的裸设备

[root@k8s-master01 kubernetes]# cat topology-sample.json 
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "k8s-master01"
              ],
              "storage": [
                "192.168.20.20"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdc",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "k8s-node01"
              ],
              "storage": [
                "192.168.20.30"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "k8s-node02"
              ],
              "storage": [
                "192.168.20.31"
              ]
            },
            "zone": 1
          },
          "devices": [
            {
              "name": "/dev/sdb",
              "destroydata": false
            }
          ]
        }
      ]
    }
  ]
}

　　查看当前pod的ClusterIP

[root@k8s-master01 kubernetes]# kubectl get svc | grep heketi
deploy-heketi   ClusterIP   10.110.217.153   <none>        8080/TCP   26m
[root@k8s-master01 kubernetes]# export HEKETI_CLI_SERVER=http://10.110.217.153:8080

　　创建GFS集群

[root@k8s-master01 kubernetes]# heketi-cli topology load --json=topology-sample.json 
Creating cluster ... ID: a058723afae149618337299c84a1eaed
    Allowing file volumes on cluster.
    Allowing block volumes on cluster.
    Creating node k8s-master01 ... ID: 929909065ceedb59c1b9c235fc3298ec
        Adding device /dev/sdc ... OK
    Creating node k8s-node01 ... ID: 37409d82b9ef27f73ccc847853eec429
        Adding device /dev/sdb ... OK
    Creating node k8s-node02 ... ID: e3ab676be27945749bba90efb34f2eb9
        Adding device /dev/sdb ... OK

　　创建heketi持久化卷

yum install device-mapper* -y
[root@k8s-master01 kubernetes]# heketi-cli setup-openshift-heketi-storage
Saving heketi-storage.json
[root@k8s-master01 kubernetes]# ls
glusterfs-daemonset.json  heketi.json                  heketi-storage.json
heketi-bootstrap.json     heketi-service-account.json  README.md
heketi-deployment.json    heketi-start.sh              topology-sample.json
[root@k8s-master01 kubernetes]# kubectl create -f heketi-storage.json
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created

 

　　如果出现如下报错：

[root@k8s-master01 kubernetes]# heketi-cli setup-openshift-heketi-storage
Error: /usr/sbin/modprobe failed: 1
  thin: Required device-mapper target(s) not detected in your kernel.
  Run `lvcreate --help' for more information.

　　解决办法：所有节点执行modprobe dm_thin_pool

　　删除中间产物

[root@k8s-master01 kubernetes]# kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"
pod "deploy-heketi-59f8dbc97f-5rf6s" deleted
service "deploy-heketi" deleted
service "heketi" deleted
deployment.apps "deploy-heketi" deleted
replicaset.apps "deploy-heketi-59f8dbc97f" deleted
job.batch "heketi-storage-copy-job" deleted
secret "heketi-storage-secret" deleted

　　创建持久化Heketi，持久化方式也可以选用其他方法。

[root@k8s-master01 kubernetes]# kubectl create -f heketi-deployment.json
service/heketi created
deployment.extensions/heketi created

　　待pod起来后，部署完成

[root@k8s-master01 kubernetes]# kubectl get po
NAME                               READY     STATUS      RESTARTS   AGE
glusterfs-5npwn                    1/1       Running     0          3h
glusterfs-8zfzq                    1/1       Running     0          3h
glusterfs-bd5dx                    1/1       Running     0          3h
heketi-5cb5f55d9f-5mtqt            1/1       Running     0          2m

 　　查看最新部署的持久化Heketi的svc，并更改HEKETI_CLI_SERVER的值

[root@k8s-master01 kubernetes]# kubectl get svc
NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
heketi                     ClusterIP   10.111.95.240   <none>        8080/TCP   12h
heketi-storage-endpoints   ClusterIP   10.99.28.153    <none>        1/TCP      12h
kubernetes                 ClusterIP   10.96.0.1       <none>        443/TCP    14d
[root@k8s-master01 kubernetes]# export HEKETI_CLI_SERVER=http://10.111.95.240:8080
[root@k8s-master01 kubernetes]# curl http://10.111.95.240:8080/hello
Hello from Heketi

　　查看GFS信息

Hello from Heketi[root@k8s-master01 kubernetes]# heketi-cli topology info

Cluster Id: 5dec5676c731498c2bdf996e110a3e5e

    File:  true
    Block: true

    Volumes:

    Name: heketidbstorage
    Size: 2
    Id: 828dc2dfaa00b7213e831b91c6213ae4
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Mount: 192.168.20.31:heketidbstorage
    Mount Options: backup-volfile-servers=192.168.20.30,192.168.20.20
    Durability Type: replicate
    Replica: 3
    Snapshot: Disabled

        Bricks:
            Id: 16b7270d7db1b3cfe9656b64c2a3916c
            Path: /var/lib/heketi/mounts/vg_04290ec786dc7752a469b66f5e94458f/brick_16b7270d7db1b3cfe9656b64c2a3916c/brick
            Size (GiB): 2
            Node: fb181b0cef571e9af7d84d2ecf534585
            Device: 04290ec786dc7752a469b66f5e94458f

            Id: 828da093d9d78a2b1c382b13cc4da4a1
            Path: /var/lib/heketi/mounts/vg_80b61df999fcac26ebca6e28c4da8e61/brick_828da093d9d78a2b1c382b13cc4da4a1/brick
            Size (GiB): 2
            Node: d38819746cab7d567ba5f5f4fea45d91
            Device: 80b61df999fcac26ebca6e28c4da8e61

            Id: e8ef0e68ccc3a0416f73bc111cffee61
            Path: /var/lib/heketi/mounts/vg_82af8e5f2fb2e1396f7c9e9f7698a178/brick_e8ef0e68ccc3a0416f73bc111cffee61/brick
            Size (GiB): 2
            Node: 0f00835397868d3591f45432e432ba38
            Device: 82af8e5f2fb2e1396f7c9e9f7698a178


    Nodes:

    Node Id: 0f00835397868d3591f45432e432ba38
    State: online
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Zone: 1
    Management Hostnames: k8s-node02
    Storage Hostnames: 192.168.20.31
    Devices:
        Id:82af8e5f2fb2e1396f7c9e9f7698a178   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):22      Free (GiB):17      
            Bricks:
                Id:e8ef0e68ccc3a0416f73bc111cffee61   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_82af8e5f2fb2e1396f7c9e9f7698a178/brick_e8ef0e68ccc3a0416f73bc111cffee61/brick

    Node Id: d38819746cab7d567ba5f5f4fea45d91
    State: online
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Zone: 1
    Management Hostnames: k8s-node01
    Storage Hostnames: 192.168.20.30
    Devices:
        Id:80b61df999fcac26ebca6e28c4da8e61   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):22      Free (GiB):17      
            Bricks:
                Id:828da093d9d78a2b1c382b13cc4da4a1   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_80b61df999fcac26ebca6e28c4da8e61/brick_828da093d9d78a2b1c382b13cc4da4a1/brick

    Node Id: fb181b0cef571e9af7d84d2ecf534585
    State: online
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Zone: 1
    Management Hostnames: k8s-master01
    Storage Hostnames: 192.168.20.20
    Devices:
        Id:04290ec786dc7752a469b66f5e94458f   Name:/dev/sdc            State:online    Size (GiB):39      Used (GiB):22      Free (GiB):17      
            Bricks:
                Id:16b7270d7db1b3cfe9656b64c2a3916c   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_04290ec786dc7752a469b66f5e94458f/brick_16b7270d7db1b3cfe9656b64c2a3916c/brick

 

5、定义StorageClass

[root@k8s-master01 gfs]# cat storageclass-gfs-heketi.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gluster-heketi
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://10.111.95.240:8080"
  restauthenabled: "false"
[root@k8s-master01 gfs]# ku
[root@k8s-master01 gfs]# kubectl create -f storageclass-gfs-heketi.yaml 
storageclass.storage.k8s.io/gluster-heketi created

　　Provisioner参数须设置为”kubernetes.io/glusterfs”

　　resturl地址为API Server所在主机可以访问到的Heketi服务的某个地址

 

6、定义PVC及测试Pod

[root@k8s-master01 gfs]# kubectl create -f pod-use-pvc.yaml 
pod/pod-use-pvc created
persistentvolumeclaim/pvc-gluster-heketi created
[root@k8s-master01 gfs]# cat pod-use-pvc.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-use-pvc
spec:
  containers:
  - name: pod-use-pvc
    image: busybox
    command:
    - sleep
    - "3600"
    volumeMounts:
    - name: gluster-volume
      mountPath: "/pv-data"
      readOnly: false
  volumes:
  - name: gluster-volume
    persistentVolumeClaim:
      claimName: pvc-gluster-heketi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-gluster-heketi
spec:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: "gluster-heketi"
  resources:
    requests:
      storage: 1Gi

　　PVC定义一旦生成，系统便触发Heketi进行相应的操作，主要为在GlusterFS集群上创建brick，再创建并启动一个volume

　　创建的pv及pvc如下

[root@k8s-master01 gfs]# kubectl get pv,pvc  | grep gluster

persistentvolume/pvc-4a8033e8-e7f7-11e8-9a09-000c293bfe27   1Gi        RWO            Delete           Bound       default/pvc-gluster-heketi                                gluster-heketi                          5m
persistentvolumeclaim/pvc-gluster-heketi   Bound     pvc-4a8033e8-e7f7-11e8-9a09-000c293bfe27   1Gi        RWO            gluster-heketi   5m

 

7、测试数据

　　进入到pod并创建文件

[root@k8s-master01 /]# kubectl exec -ti pod-use-pvc -- /bin/sh
/ # cd /pv-data/
/pv-data # mkdir {1..10}
/pv-data # ls
{1..10}

　　宿主机挂载测试

# 查看volume
[root@k8s-master01 /]# heketi-cli topology info

Cluster Id: 5dec5676c731498c2bdf996e110a3e5e

    File:  true
    Block: true

    Volumes:

    Name: vol_56d636b452d31a9d4cb523d752ad0891
    Size: 1
    Id: 56d636b452d31a9d4cb523d752ad0891
    Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
    Mount: 192.168.20.31:vol_56d636b452d31a9d4cb523d752ad0891
    Mount Options: backup-volfile-servers=192.168.20.30,192.168.20.20
    Durability Type: replicate
    Replica: 3
    Snapshot: Enabled
...
...
# 或者使用volume list查看

　[root@k8s-master01 mnt]# heketi-cli volume list
  Id:56d636b452d31a9d4cb523d752ad0891 Cluster:5dec5676c731498c2bdf996e110a3e5e Name:vol_56d636b452d31a9d4cb523d752ad0891
  Id:828dc2dfaa00b7213e831b91c6213ae4 Cluster:5dec5676c731498c2bdf996e110a3e5e Name:heketidbstorage
  [root@k8s-master01 mnt]#

　　vol_56d636b452d31a9d4cb523d752ad0891为volume Name，Mount: 192.168.20.31:vol_56d636b452d31a9d4cb523d752ad0891，挂载方式。

　　挂载查看数据

[root@k8s-master01 /]# mount -t glusterfs 192.168.20.31:vol_56d636b452d31a9d4cb523d752ad0891  /mnt/
[root@k8s-master01 /]# cd /mnt/
[root@k8s-master01 mnt]# ls
{1..10}

 

8、测试Deployments

[root@k8s-master01 gfs]# cat nginx-gluster.yaml 
apiVersion: extensions/v1beta1 
kind: Deployment 
metadata: 
  name: nginx-gfs
spec: 
  replicas: 2
  template: 
    metadata: 
      labels: 
        name: nginx 
    spec: 
      containers: 
        - name: nginx 
          image: nginx 
          imagePullPolicy: IfNotPresent
          ports: 
            - containerPort: 80
          volumeMounts:
            - name: nginx-gfs-html
              mountPath: "/usr/share/nginx/html"
            - name: nginx-gfs-conf
              mountPath: "/etc/nginx/conf.d"
      volumes:
      - name: nginx-gfs-html
        persistentVolumeClaim:
          claimName: glusterfs-nginx-html
      - name: nginx-gfs-conf
        persistentVolumeClaim:
          claimName: glusterfs-nginx-conf
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: glusterfs-nginx-html
spec:
  accessModes: [ "ReadWriteMany" ]
  storageClassName: "gluster-heketi"
  resources:
    requests:
      storage: 0.5Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: glusterfs-nginx-conf
spec:
  accessModes: [ "ReadWriteMany" ]
  storageClassName: "gluster-heketi"
  resources:
    requests:
      storage: 0.1Gi
[root@k8s-master01 gfs]# kubectl get po,pvc,pv | grep nginx
pod/nginx-gfs-77c758ccc-2hwl6          1/1       Running             0          4m
pod/nginx-gfs-77c758ccc-kxzfz          0/1       ContainerCreating   0          3m

persistentvolumeclaim/glusterfs-nginx-conf   Bound     pvc-f40c5d4b-e800-11e8-8a89-000c293ad492   1Gi        RWX            gluster-heketi   2m
persistentvolumeclaim/glusterfs-nginx-html   Bound     pvc-f40914f8-e800-11e8-8a89-000c293ad492   1Gi        RWX            gluster-heketi   2m

persistentvolume/pvc-f40914f8-e800-11e8-8a89-000c293ad492   1Gi        RWX            Delete           Bound       default/glusterfs-nginx-html                              gluster-heketi                          4m
persistentvolume/pvc-f40c5d4b-e800-11e8-8a89-000c293ad492   1Gi        RWX            Delete           Bound       default/glusterfs-nginx-conf                              gluster-heketi                          4m

　　查看挂载情况

[root@k8s-master01 gfs]# kubectl exec -ti nginx-gfs-77c758ccc-2hwl6 -- df -Th
Filesystem                                         Type            Size  Used Avail Use% Mounted on
overlay                                            overlay          86G  6.6G   80G   8% /
tmpfs                                              tmpfs           7.8G     0  7.8G   0% /dev
tmpfs                                              tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root                            xfs              86G  6.6G   80G   8% /etc/hosts
shm                                                tmpfs            64M     0   64M   0% /dev/shm
192.168.20.20:vol_b9c68075c6f20438b46db892d15ed45a fuse.glusterfs 1014M   43M  972M   5% /etc/nginx/conf.d
192.168.20.20:vol_32146a51be9f980c14bc86c34f67ebd5 fuse.glusterfs 1014M   43M  972M   5% /usr/share/nginx/html
tmpfs                                              tmpfs           7.8G   12K  7.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs

　　挂载并创建index.html

[root@k8s-master01 gfs]# mount -t glusterfs 192.168.20.20:vol_32146a51be9f980c14bc86c34f67ebd5 /mnt/
[root@k8s-master01 gfs]# cd /mnt/
[root@k8s-master01 mnt]# ls
[root@k8s-master01 mnt]# echo "test" > index.html
[root@k8s-master01 mnt]# kubectl exec -ti nginx-gfs-77c758ccc-2hwl6 -- cat /usr/share/nginx/html/index.html
test

　　扩容nginx

[root@k8s-master01 ~]# kubectl get deploy
NAME        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
heketi      1         1         1            1           14h
nginx-gfs   2         2         2            2           23m
[root@k8s-master01 ~]# kubectl scale deploy nginx-gfs --replicas 3
deployment.extensions/nginx-gfs scaled
[root@k8s-master01 ~]# kubectl get po
NAME                               READY     STATUS      RESTARTS   AGE
glusterfs-5npwn                    1/1       Running     0          18h
glusterfs-8zfzq                    1/1       Running     0          17h
glusterfs-bd5dx                    1/1       Running     0          18h
heketi-5cb5f55d9f-5mtqt            1/1       Running     0          14h
nginx-gfs-77c758ccc-2hwl6          1/1       Running     0          11m
nginx-gfs-77c758ccc-6fphl          1/1       Running     0          8m
nginx-gfs-77c758ccc-kxzfz          1/1       Running     0          10m

　　查看文件内容

　　[root@k8s-master01 ~]# kubectl exec -ti nginx-gfs-77c758ccc-6fphl — cat /usr/share/nginx/html/index.html
　　test

 

9、扩容GlusterFS

9.1添加磁盘至已存在的node节点

　　基于上述节点，假设在k8s-node02上增加磁盘

　　查看k8s-node02部署的pod name及IP

[root@k8s-master01 ~]# kubectl get po -o wide -l glusterfs-node
NAME              READY     STATUS    RESTARTS   AGE       IP              NODE
glusterfs-5npwn   1/1       Running   0          20h       192.168.20.31   k8s-node02
glusterfs-8zfzq   1/1       Running   0          20h       192.168.20.20   k8s-master01
glusterfs-bd5dx   1/1       Running   0          20h       192.168.20.30   k8s-node01

　　在node02上确认新添加的盘符

Disk /dev/sdc: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

　　使用heketi-cli查看cluster ID和所有node ID

[root@k8s-master01 ~]# heketi-cli cluster info
Error: Cluster id missing
[root@k8s-master01 ~]# heketi-cli cluster list
Clusters:
Id:5dec5676c731498c2bdf996e110a3e5e [file][block]
[root@k8s-master01 ~]# heketi-cli cluster info 5dec5676c731498c2bdf996e110a3e5e
Cluster id: 5dec5676c731498c2bdf996e110a3e5e
Nodes:
0f00835397868d3591f45432e432ba38
d38819746cab7d567ba5f5f4fea45d91
fb181b0cef571e9af7d84d2ecf534585
Volumes:
32146a51be9f980c14bc86c34f67ebd5
56d636b452d31a9d4cb523d752ad0891
828dc2dfaa00b7213e831b91c6213ae4
b9c68075c6f20438b46db892d15ed45a
Block: true

File: true

　　找到对应的k8s-node02的node ID

[root@k8s-master01 ~]# heketi-cli node info 0f00835397868d3591f45432e432ba38
Node Id: 0f00835397868d3591f45432e432ba38
State: online
Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
Zone: 1
Management Hostname: k8s-node02
Storage Hostname: 192.168.20.31
Devices:
Id:82af8e5f2fb2e1396f7c9e9f7698a178   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):25      Free (GiB):14      Bricks:4

　　添加磁盘至GFS集群的node02

[root@k8s-master01 ~]# heketi-cli device add --name=/dev/sdc --node=0f00835397868d3591f45432e432ba38
Device added successfully

　　查看结果

[root@k8s-master01 ~]# heketi-cli node info 0f00835397868d3591f45432e432ba38
Node Id: 0f00835397868d3591f45432e432ba38
State: online
Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
Zone: 1
Management Hostname: k8s-node02
Storage Hostname: 192.168.20.31
Devices:
Id:5539e74bc2955e7c70b3a20e72c04615   Name:/dev/sdc            State:online    Size (GiB):39      Used (GiB):0       Free (GiB):39      Bricks:0       
Id:82af8e5f2fb2e1396f7c9e9f7698a178   Name:/dev/sdb            State:online    Size (GiB):39      Used (GiB):25      Free (GiB):14      Bricks:4

 

9.2 添加新节点

　　假设将k8s-master03，IP为192.168.20.22的加入glusterfs集群，并将该节点的/dev/sdc加入到集群

　　加标签，之后会自动创建pod

[root@k8s-master01 kubernetes]# kubectl label node k8s-master03 storagenode=glusterfs
node/k8s-master03 labeled
[root@k8s-master01 kubernetes]# kubectl  get pod -owide -l glusterfs-node
NAME              READY     STATUS              RESTARTS   AGE       IP              NODE
glusterfs-5npwn   1/1       Running             0          21h       192.168.20.31   k8s-node02
glusterfs-8zfzq   1/1       Running             0          21h       192.168.20.20   k8s-master01
glusterfs-96w74   0/1       ContainerCreating   0          2m        192.168.20.22   k8s-master03
glusterfs-bd5dx   1/1       Running             0          21h       192.168.20.30   k8s-node01

　　在任意节点执行peer probe

[root@k8s-master01 kubernetes]# kubectl exec -ti glusterfs-5npwn -- gluster peer probe 192.168.20.22
peer probe: success.

　　将新节点加入到glusterfs集群中

[root@k8s-master01 kubernetes]# heketi-cli cluster list
Clusters:
Id:5dec5676c731498c2bdf996e110a3e5e [file][block]
[root@k8s-master01 kubernetes]# heketi-cli node add --zone=1 --cluster=5dec5676c731498c2bdf996e110a3e5e --management-host-name=k8s-master03 --storage-host-name=192.168.20.22
Node information:
Id: 150bc8c458a70310c6137e840619758c
State: online
Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
Zone: 1
Management Hostname k8s-master03
Storage Hostname 192.168.20.22

　　将新节点的磁盘加入到集群中

[root@k8s-master01 kubernetes]# heketi-cli device add --name=/dev/sdc --node=150bc8c458a70310c6137e840619758c
Device added successfully

　　验证

[root@k8s-master01 kubernetes]# heketi-cli node list
Id:0f00835397868d3591f45432e432ba38    Cluster:5dec5676c731498c2bdf996e110a3e5e
Id:150bc8c458a70310c6137e840619758c    Cluster:5dec5676c731498c2bdf996e110a3e5e
Id:d38819746cab7d567ba5f5f4fea45d91    Cluster:5dec5676c731498c2bdf996e110a3e5e
Id:fb181b0cef571e9af7d84d2ecf534585    Cluster:5dec5676c731498c2bdf996e110a3e5e
[root@k8s-master01 kubernetes]# heketi-cli node info 150bc8c458a70310c6137e840619758c
Node Id: 150bc8c458a70310c6137e840619758c
State: online
Cluster Id: 5dec5676c731498c2bdf996e110a3e5e
Zone: 1
Management Hostname: k8s-master03
Storage Hostname: 192.168.20.22
Devices:
Id:2d5210c19858fb7ea3f805e6f582ecce   Name:/dev/sdc            State:online    Size (GiB):39      Used (GiB):0       Free (GiB):39      Bricks:0

 

发表于 2018-11-14 15:42 杜先生的博客 阅读(…) 评论(…) 编辑 收藏

 

版权声明：本文为dukuan原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://www.cnblogs.com/dukuan/p/9954094.html

kubernetes实战(九)：k8s集群动态存储管理GlusterFS及容器化GlusterFS扩容的更多相关文章

  1. 图解 kubernetes scheduler 架构设计系列-初步了解

    资源调度基础 scheudler是kubernetes中的核心组件，负责为用户声明的pod资源选择合适的nod […]...

  2. Kubernetes 多集群在开源项目 KubeSphere 的应用

    Kubernetes 多集群使用场景 随着容器的普及和 Kubernetes 的日渐成熟，企业内部运行多个 K […]...

  3. 基于 Clusternet 与 OCM 打造新一代开放的多集群管理平台

    背景 随着 5G、物联网设备的爆炸性增长以及智能终端不断增强的计算能力，带来了前所未有的数据量，传统的中心集中 […]...

  4. kubernetes 1.17.2结合ceph13.2.8 实现jenkins部署并用traefik2.1代理

    注：关于ceph、kubernetes集群的部署在此不声明，相信搜到本篇博文，你一定对ceph、kuberne […]...

  5. 腾讯云TKE-PV使用COS存储案例：容器目录权限问题

    一文帮你解决容器目录权限难题！ 背景 在TKE的集群中创建工作负载并把某一个对应的cos桶的根目录挂载到/da […]...

  6. 为什么 kubernetes 天然适合微服务

    本文由  网易云 发布。   作者：刘超，网易云首席解决方案架构师 最近总在思考，为什么在支撑容器平台和微服务 […]...

  7. Health Check – 每天5分钟玩转 Docker 容器技术（142）

    强大的自愈能力是 Kubernetes 这类容器编排引擎的一个重要特性。自愈的默认实现方式是自动重启发生故障的 […]...

  8. kubernetes Auto Install Guide

    1.概念&架构 Kubernetes is an open-source system for aut […]...

随机推荐

  1. 【译文】不是所有的 bug 都值得修复的

    原文作者：KRISTINE PINEDO 译者：白乐航 欢迎访问网易云社区，了解更多网易技术产品运营经验。   […]...

  2. Java知多少（29）覆盖和重载

    在类继承中，子类可以修改从父类继承来的方法，也就是说子类能创建一个与父类方法有不同功能的方法，但具有相同的名称 […]...

  3. 几种编程语言及其介绍 – 颜子琦

    几种编程语言及其介绍 机器语言：纯粹的机器代码 。汇编语言：8086汇编、Win32汇编、.NET的汇编 高级 […]...

  4. 最全的Sublime text 常用插件整理

    Sublime text 常用插件 记录一些自己在使用sublime时常用的一些插件 cssrem 、Subl […]...

  5. 关于CCS软件的Graph功能使用详解

    　　装载自： 　　　　　　http://blog.csdn.net/HJ199404182515/articl […]...

  6. 首创诠释docker的Formulas: Windows 7 + Tiny Linux 4.19 + XFS + Vmware Workstation = super machine (docker从零开始时记，Follow me and you go)

    首创诠释docker的Formulas: Windows 7 + Tiny Linux 4.19 + XFS […]...

  7. 实用工具网站分享

    持续更新中~~~ 在线工具网站分享 免费PPT模板下载网站: PPT超级市场 http://ppt.sotar […]...

  8. JDK容器类Map源码解读

    java.util.Map接口是JDK1.2开始提供的一个基于键值对的散列表接口，其设计的初衷是为了替换JDK […]...

热门专题

Java中字符串相加和字符串常量相加区别
airpods
使用正则表达式构造定制的HTML5输入框
开源学习管理系统（LMS）的比较
python多线程与多进程及其区别
3D》
将本地文件上传至服务器
java集合类源码分析之List（一）
【转】电路板绘制经验积累
让图片再小一点点
键盘除了fn键都失效了，键盘失灵怎么办？
e:Exception
查看端口是被哪个应用/进程占用
液晶显示
小白工作记录
FileInputFormat详解额
苹果MacBook笔记本电脑物超所值
三种SQLite数据库管理工具(IDE)的介绍及区别
微服
python知识点总结01(不定时更新)
Python+Selenium显示等待操作代码
展开目录

目录导航