前提：需要内网和阿里云账号的vpc做互通
插件地址是电梯直达，但是用这个仓库中的部署文件没办法在私有云直接部署，会有报错。需要做一些修改。
环境：k8s1.18.8 alibaba-cloud-csi-driver-v1.1.1(部署时的最新版本)
错误
如果是直接用github上原部署文件来部署，csi-plugin和csi-provisioner会报以下错误。报错提示没有nodeid
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
$ kubectl logs -n kube-system csi-provisioner-7bc58d984d-f6sgr -c csi-provisioner -f
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:02:10 --:--:-- 0curl: (7) Failed connect to 100.100.100.200:80; Connection timed out
Running nas plugin....
time="2021-01-31T08:43:01Z" level=info msg="Multi CSI Driver Name: nas, nodeID: , endPoints: unix://var/lib/kubelet/csi-provisioner/driverplugin.csi.alibabacloud.com-replace/csi.sock"
time="2021-01-31T08:43:01Z" level=info msg="CSI Driver Branch: 'master', Version: 'v1.16.9.43-f36bb540-aliyun', Build time: '2020-10-12-21:54:17'\n"
time="2021-01-31T08:43:01Z" level=info msg="Create Stroage Path: /var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/controller"
time="2021-01-31T08:43:01Z" level=info msg="Create Stroage Path: /var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/node"
time="2021-01-31T08:43:01Z" level=info msg="CSI is running status."
time="2021-01-31T08:43:01Z" level=info msg="Metric listening on address: /healthz"
time="2021-01-31T08:43:01Z" level=info msg="Driver: nasplugin.csi.alibabacloud.com version: 1.0.0"
time="2021-01-31T08:43:31Z" level=info msg="Use node id : "
E0131 08:43:31.295445 11 driver.go:46] NodeID missing
I0131 08:43:31.295452 11 driver.go:93] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x16de344]
goroutine 75 [running]:
github.com/kubernetes-csi/drivers/pkg/csi-common.(*CSIDriver).AddVolumeCapabilityAccessModes(0x0, 0xc000113f14, 0x1, 0x1, 0x0, 0x0, 0x0)
/home/regressionTest/go/pkg/mod/github.com/kubernetes-csi/drivers@v1.0.2/pkg/csi-common/driver.go:96 +0x1c4
github.com/kubernetes-sigs/alibaba-cloud-csi-driver/pkg/nas.NewDriver(0x0, 0x0, 0xc000702140, 0x4e, 0x0)
/home/regressionTest/go/src/github.com/kubernetes-sigs/alibaba-cloud-csi-driver/pkg/nas/nas.go:78 +0x1af
main.main.func1(0xc000136224, 0xc000702140, 0x4e)
/home/regressionTest/go/src/github.com/kubernetes-sigs/alibaba-cloud-csi-driver/main.go:180 +0x79
created by main.main
/home/regressionTest/go/src/github.com/kubernetes-sigs/alibaba-cloud-csi-driver/main.go:178 +0xfe5
|
以上错误应该是csi-plugin需要nodeid最终注入到每个node的annotation中作为区分csi-node，对于非阿里云机器，需要在在csi-plugin手动传入hostname做为nodeid
相应调整并部署
下载及解压
1
2
|
$ wget https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/archive/v1.1.1.tar.gz
$ tar -zxvf alibaba-cloud-csi-driver-1.1.1.tar.gz
|
修改对应的yaml文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
|
$ cd alibaba-cloud-csi-driver-1.1.1/deploy/nas/
# 注意nas文件夹有两套配置，私有云用dedicated的这一套（专门提工单问的）
$ vim nas-plugin-dedicated.yaml
......
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.16.9.43-f36bb540-aliyun
imagePullPolicy: "Always"
args:
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=2"
- "--driver=nas"
- "--nodeid=$(KUBE_NODE_NAME)" #增加这一行
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
......
$ vim nas-provisioner-dedicated.yaml
......
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.16.9.43-f36bb540-aliyun
imagePullPolicy: "Always"
args:
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=2"
- "--driver=nas"
- "--nodeid=$(KUBE_NODE_NAME)" #添加这一行
env:
# 这个文件里没有下面这个KUBE_NODE_NAME的变量 也需要加上
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName #到这里
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/csi-provisioner/driverplugin.csi.alibabacloud.com-replace/csi.sock
......
|
安装部署
1
2
3
4
|
$ kubectl apply -f rbac.yaml
$ kubectl apply -f nas/nas-plugin-dedicated.yaml
$ kubectl apply -f nas/nas-provisioner-dedicated.yaml
# 注意：一定要确保创建的plugin和provisioner没有问题后再进行下面的操作
|
- 部署storageclass、pvc和deployment进行测试
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
$ vim sc-test.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: test-sc
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
driver: flexvolume
server: xxxxxxx.nas.aliyuncs.com:/test-1/ #这里使用你阿里云上nas地址
archiveOnDelete: "true"
reclaimPolicy: Delete
provisioner: nasplugin.csi.alibabacloud.com
volumeBindingMode: Immediate
$ kubectl apply -f sc-test.yaml
storageclass.storage.k8s.io/test-sc created
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
test-sc nasplugin.csi.alibabacloud.com Delete Immediate false 50s
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
$ vim pvc-test.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-test
spec:
storageClassName: test-sc
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500Mi
$ kubectl apply -f pvc-test.yaml
persistentvolumeclaim/pvc-test created
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-test Bound nas-075602b2-7723-4f8a-8b3d-aeb852772cd6 500Mi RWX test-sc 4s
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
$ vim deployment-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-nas
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
volumeMounts:
- name: nas-pvc
mountPath: "/data"
volumes:
- name: nas-pvc
persistentVolumeClaim:
claimName: pvc-test
$ kubectl apply -f deployment-test.yaml
deployment.apps/deployment-nas created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deployment-nas-57b549759c-k2nw8 1/1 Running 0 38s
|
至此部署测试成功，想要再详细测试的话可以往pod的/data目录放个文件，然后删除pod，看新创建的pod里还有没有放的文件。
参考链接
阿里云文档
Author
dylan
LastMod
2021-02-15
License
如需转载请注明文章作者和出处。谢谢！
