NFS做Kubernetes的后端存储

使用NFS作为默认SC：

• 安装NFS服务
• 配置共享存储路径开放权限
• 搭建NFS服务器和NFS-Client
• 在Master节点上面执行命令 vi /etc/exports，创建 exports 文件，启动 nfs 服务，创建共享目录。在其他的node节点上安装nfs服务并且开启服务

安装NFS服务

在所有节点安装NFS服务配置共享目录。

$ yum install -y nfs-utils
$ echo "/nfs/data/ *(insecure,rw,sync,no_root_squash)" > /etc/exports
$ mkdir -p /nfs/data
$ systemctl enable rpcbind && systemctl start rpcbind
$ systemctl enable nfs-server && systemctl start nfs-server

#检查配置是否生效
exportfs

# 输出结果如下所示
/nfs/data  <work>
$ showmount -e 10.10.10.128
  Export list for 10.10.10.128
  /nfs/data *

• 在Node节点上配置NFS-Client(node1和node2操作)
• 安装客户端工具,挂载 nfs 服务器上的共享目录到本机路径 /root/nfsmount。

$ yum install -y nfs-utils
$ mkdir /root/nfsmount
$ systemctl enable rpcbind && systemctl start rpcbind
$ systemctl enable nfs-server && systemctl start nfs-server
$ mount -t nfs 10.10.10.128:/nfs/data /root/nfsmount

• 测试Pod直接挂载了

$ vim pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: vol-nfs
  namespace: default
spec:
  volumes:
  - name: html
    nfs:
      path: /nfs/data   #1000G
      server: 自己的nfs服务器地址
  containers:
  - name: myapp
    image: nginx
    volumeMounts:
    - name: html
      mountPath: /usr/share/nginx/html/
$ kubectl apply -f pod.yaml 
$ echo  ABCD > /nfs/data/index.html 
$ curl 
ABCD

设置动态供应

创建provisioner（NFS环境前面已经搭好）

字段名称	填入内容	备注
名称	nfs-storage	自定义存储类名称
NFS Server	10.10.10.128	NFS服务的IP地址
NFS Path	/nfs/data	NFS服务所共享的路径

• 在Master节点上面操作
• 先创建授权

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
   name: nfs-provisioner-runner
rules:
   -  apiGroups: [""]
      resources: ["persistentvolumes"]
      verbs: ["get", "list", "watch", "create", "delete"]
   -  apiGroups: [""]
      resources: ["persistentvolumeclaims"]
      verbs: ["get", "list", "watch", "update"]
   -  apiGroups: ["storage.k8s.io"]
      resources: ["storageclasses"]
      verbs: ["get", "list", "watch"]
   -  apiGroups: [""]
      resources: ["events"]
      verbs: ["watch", "create", "update", "patch"]
   -  apiGroups: [""]
      resources: ["services", "endpoints"]
      verbs: ["get","create","list", "watch","update"]
   -  apiGroups: ["extensions"]
      resources: ["podsecuritypolicies"]
      resourceNames: ["nfs-provisioner"]
      verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
#创建nfs-client的授权
kind: Deployment
apiVersion: apps/v1
metadata:
   name: nfs-client-provisioner
spec:
   replicas: 1
   strategy:
     type: Recreate
   selector:
     matchLabels:
        app: nfs-client-provisioner
   template:
      metadata:
         labels:
            app: nfs-client-provisioner
      spec:
         serviceAccount: nfs-provisioner
         containers:
            -  name: nfs-client-provisioner
               image: lizhenliang/nfs-client-provisioner
               volumeMounts:
                 -  name: nfs-client-root
                    mountPath:  /persistentvolumes
               env:
                 -  name: PROVISIONER_NAME #供应者的名字
                    value: storage.pri/nfs #名字虽然可以随便起，以后引用要一致
                 -  name: NFS_SERVER
                    value: ip
                 -  name: NFS_PATH
                    value: /nfs/data
         volumes:
           - name: nfs-client-root
             nfs:
               server: ip
               path: /nfs/data
##这个镜像中volume的mountPath默认为/persistentvolumes，不能修改，否则运行时会报错

在Master上创建provisioner

• 创建存储类
• 改变默认sc

“reclaim policy”有三种方式：Retain、Recycle、Deleted。

Retain

• 保护被PVC释放的PV及其上数据，并将PV状态改成”released”，不将被其它PVC绑定。集群管理员手动通过如下步骤释放存储资源
• 手动删除PV，但与其相关的后端存储资源如(AWS EBS, GCE PD, Azure Disk, or Cinder volume)仍然存在。
• 手动清空后端存储volume上的数据。
• 手动删除后端存储volume，或者重复使用后端volume，为其创建新的PV。

Delete

• 删除被PVC释放的PV及其后端存储volume。对于动态PV其”reclaim policy”继承自其”storage class”，
• 默认是Delete。集群管理员负责将”storage class”的”reclaim policy”设置成用户期望的形式，否则需要用户手动为创建后的动态PV编辑”reclaim policy”

Recycle

• 保留PV，但清空其上数据，已废弃

$  vi storageclass-nfs.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: storage-nfs
provisioner: storage.pri/nfs
reclaimPolicy: Delete

$ kubectl apply -f storageclass-nfs.yaml

#改变系统默认sc
https://kubernetes.io/zh/docs/tasks/administer-cluster/change-default-storage-class/#%e4%b8%ba%e4%bb%80%e4%b9%88%e8%a6%81%e6%94%b9%e5%8f%98%e9%bb%98%e8%ae%a4-storage-class

$ kubectl patch storageclass storage-nfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

验证nfs动态供应

创建PVC

$ vi  pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-claim-01
 # annotations:
 #   volume.beta.kubernetes.io/storage-class: "storage-nfs"
spec:
  storageClassName: storage-nfs  #这个class一定注意要和sc的名字一样
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

使用PVC

$ vi testpod.yaml
kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: busybox
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: pvc-claim-01

K8s 1.20x版本nfs动态存储报错

persistentvolume-controller waiting for a volume to be created, either by external provisioner "qgg-nfs-storage" or manually created by system administrator waiting for a volume to be created, either by external provisioner "storage.

• 修改apiserver的配置

$ cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
···
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --feature-gates=RemoveSelfLink=false # 添加这个配置

• 重启apiserver

$ systemctl  restart kubelet