📦 kubernetes版本 v1.18.10 Rook版本 v1.6.10
🏢 官方文档: https://rook.github.io/docs/rook/ (版本在 Prerequisites中)
本博客写于2024年，如果要是问: 为什么2024年还在使用
k8s1.18版本？是因为我最近在看《深入刨析Kubernetes》这本书，这本书使用的就是此版本，我为了减少学习误差也使用了此版本
🏆 使用ROOK对接NFS
如果你要是在问我
Rook对接NFS不是多此一举吗？为什么不直接对接Ceph？作者记忆力有限之前学的Ceph知识早就抛掷脑后，又卡在大四上学期比较忙碌的时候，又需要用到这样一个特别方便的存储编排器，故此
我的 K8s集群架构
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 16d v1.18.10
k8s-node01 Ready <none> 16d v1.18.10
1️⃣ 使用 git clone Rook源
执行如下命令即可
1.common.yaml文件主要定义了一些 CRD文件
2.operator.yaml从字面上看就知道时 operator了
3.psp.yaml为pod安全规则
4.rbac.yaml也是字面意思了，是 rbac规则
$ git clone --single-branch --branch v1.6.11 https://github.com/rook/rook.git
cd rook/cluster/examples/kubernetes/nfs
kubectl create -f common.yaml
kubectl create -f operator.yaml
kubectl apply -f rbac.yaml
kubectl apply -f psp.yaml
等待 rook-nfs-operator启动
[root@k8s-master nfs]# kubectl get pod -n rook-nfs-system
NAME READY STATUS RESTARTS AGE
rook-nfs-operator-8549fbd4d6-kpc46 1/1 Running 0 58m
2️⃣ 创建 NFSServer对接的 PV
下面的使用 nfs对接 nfsserver，在 pod绑定 pvc时候会有报错
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/fdc6ee3a-a30e-4a84-b3c6-5247d87b88e3/volumes/kubernetes.io~nfs/pvc-2032a746-ac6b-485d-9083-5bbc6289846a --scope -- mount -t nfs 10.99.222.217:/rook-nfs-pvc/default-rook-nfs-pv-claim-pvc-2032a746-ac6b-485d-9083-5bbc6289846a /var/lib/kubelet/pods/fdc6ee3a-a30e-4a84-b3c6-5247d87b88e3/volumes/kubernetes.io~nfs/pvc-2032a746-ac6b-485d-9083-5bbc6289846a
Output: Running scope as unit: run-ref12c7c1622f40c2b2c96203eb5f2af9.scope
mount.nfs: access denied by server while mounting 10.99.222.217:/rook-nfs-pvc/default-rook-nfs-pv-claim-pvc-2032a746-ac6b-485d-9083-5bbc6289846a
我百思不得其解，排查一下午没有排查出来，我只好放弃这个方法，直接用 hostpath来创建 pv
apiVersion: v1
kind: PersistentVolume
metadata:
name: rook-nfs-pv
namespace: rook-nfs
spec:
capacity: ##设置pv的大小
storage: 40Gi
claimRef:
name: rook-nfs-pvc
namespace: rook-nfs
accessModes: ##设置pv的访问模式
- ReadWriteMany ##单节点读写
volumeMode: Filesystem ##文件系统是filesystem还是block
persistentVolumeReclaimPolicy: Recycle ##回收策略
nfs: ##nfs设置
path: /nfs-default-claim
server: k8s-node01
对接关系为 ServerPV----ServerPVC----NFSSserver----RookPv----RookPVC
1.Rook-NFSServer需要绑定一个 PVC，而此 PVC需要绑定一个 PV
2.使用动态 ROOK为 storageclass的 provisioner为 nfs.rook.io/rook-nfs-provisioner的 pvc的绑定 pv时候,需要通过 NFSSserver从它绑定的 pvc(ServerPVC)的存储空间（也就说 nfs）中创建一个 RookPv来和它指定的 pvc绑定
[root@k8s-master nfs]# cat pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: rook-nfs-pv
namespace: rook-nfs
spec:
capacity: ##设置pv的大小
storage: 20Gi
accessModes: ##设置pv的访问模式
- ReadWriteMany ##单节点读写
volumeMode: Filesystem ##文件系统是filesystem还是block
persistentVolumeReclaimPolicy: Recycle ##回收策略
nfs: ##nfs设置
path: /nfs/pv1
server: k8s-node01
##查看
[root@k8s-master nfs]# kubectl get pv -n rook-nfs
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
rook-nfs-pv 5Gi RWO Recycle Available nfs 77s
3️⃣ 创建 ROOK-NFSServer
详情可以参考:
https://rook.github.io/docs/rook/v1.6/nfs-crd.html
---
# A default storageclass must be present
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-default-claim
namespace: rook-nfs
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
---
apiVersion: nfs.rook.io/v1alpha1
kind: NFSServer
metadata:
name: rook-nfs
namespace: rook-nfs
spec:
replicas: 1
exports:
- name: share1
server:
accessMode: ReadWrite
squash: "none"
# A Persistent Volume Claim must be created before creating NFS CRD instance.
persistentVolumeClaim:
claimName: nfs-default-claim
# A key/value list of annotations
annotations:
rook: nfs
创建后查看
[root@k8s-master ~]# kubectl get pvc -n rook-nfs
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-default-claim Bound rook-nfs-pv 20Gi RWX 32m
[root@k8s-master ~]# kubectl get pv -n rook-nfs
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
rook-nfs-pv 20Gi RWX Recycle Bound rook-nfs/nfs-default-claim 41m
4️⃣ 创建 ROOK的 NFS StoragClass
[root@k8s-master nfs]# cat sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
labels:
app: rook-nfs
name: rook-nfs-share1
parameters:
exportName: share1
nfsServerName: rook-nfs
nfsServerNamespace: rook-nfs
provisioner: nfs.rook.io/rook-nfs-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
创建完成后查看
[root@k8s-master nfs]# kubectl get storageclasses.storage.k8s.io
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-nfs-share1 nfs.rook.io/rook-nfs-provisioner Delete Immediate false 68m
🍪 实验
创建一个 pvc，并 storageClass设置为 rook-nfs-share1
[root@k8s-master nfs]# cat pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rook-nfs-pv-claim
spec:
storageClassName: "rook-nfs-share1"
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Mi
创建后查看，它居然自动绑定了一个 pv，没错这个 pv就是 rook帮你创建出来的，这样在多 statefulset环境下好用多了，不用为每个 volumeTemplate创建 pv
[root@k8s-master nfs]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
rook-nfs-pv-claim Bound pvc-5d03ccbb-8c0a-4c5b-a47b-8eef366b17ad 1Mi RWX rook-nfs-share1 3s
[root@k8s-master nfs]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-5d03ccbb-8c0a-4c5b-a47b-8eef366b17ad 1Mi RWX Delete Bound default/rook-nfs-pv-claim rook-nfs-share1 4s
