GRPC error - Unmatched '"' - PersistentStorage not granted

I am trying to setup a gitlab auto devops with Kubernetes on ubuntu 18.04.

The kubernets cluster is up and running, and connected to gitlab. gitlab requires some prerequisites, like prometheus and ingress. They require persistent storage which I’d like to satisfy the freenas provisining.

Installing democratic-csi worked fine using the https://jonathangazeley.com/2021/01/05/using-truenas-to-provide-persistent-storage-for-kubernetes recipe. I fail when validating test-claim-nfs.yaml. After the persistentvolumeclaim/test-claim-nfs I can see my nfs volumes and shares being created succesfully. I can mount the NFS volume on my kubernetes nodes.

Assigment of the PersistentVolume fails, though. See enclosed log. GRPC internal unmatched '"' , looks like an internal error to me.

Does the log give enough of a hint? Any other information required?

Kubernets 1.20.2 helm.sh/chart: democratic-csi-0.7.0 TrueNAS-12.0-U1.1

I0213 22:25:30.381616       1 controller.go:1335] provision "democratic-csi/test-claim-nfs" class "freenas-nfs-csi": started
W0213 22:25:30.381755       1 controller.go:570] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0213 22:25:30.381970       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"democratic-csi", Name:"test-claim-nfs", UID:"123e98cd-1507-49f8-b9bc-a88f68ac420b", APIVersion:"v1", ResourceVersion:"13934661", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "democratic-csi/test-claim-nfs"
I0213 22:25:30.384568       1 controller.go:731] CreateVolumeRequest name:"pvc-123e98cd-1507-49f8-b9bc-a88f68ac420b" capacity_range:<required_bytes:1073741824 > volume_capabilities: access_mode: > parameters: parameters: parameters: parameters: 
I0213 22:25:30.384698       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I0213 22:25:30.384704       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-123e98cd-1507-49f8-b9bc-a88f68ac420b","parameters":{"csi.storage.k8s.io/pv/name":"pvc-123e98cd-1507-49f8-b9bc-a88f68ac420b","csi.storage.k8s.io/pvc/name":"test-claim-nfs","csi.storage.k8s.io/pvc/namespace":"democratic-csi","fsType":"nfs"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"nfs","mount_flags":["noatime","nfsvers=3"]}},"access_mode":{"mode":1}}]}
I0213 22:25:31.294562       1 connection.go:185] GRPC response: {}
I0213 22:25:31.294673       1 connection.go:186] GRPC error: rpc error: code = Internal desc = Error: Unmatched '"'.
I0213 22:25:31.294722       1 controller.go:752] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = Error: Unmatched '"'.
I0213 22:25:31.294776       1 controller.go:1102] Final error received, removing PVC 123e98cd-1507-49f8-b9bc-a88f68ac420b from claims in progress
W0213 22:25:31.294784       1 controller.go:961] Retrying syncing claim "123e98cd-1507-49f8-b9bc-a88f68ac420b", failure 4
E0213 22:25:31.294821       1 controller.go:984] error syncing claim "123e98cd-1507-49f8-b9bc-a88f68ac420b": failed to provision volume with StorageClass "freenas-nfs-csi": rpc error: code = Internal desc = Error: Unmatched '"'.
I0213 22:25:31.295131       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"democratic-csi", Name:"test-claim-nfs", UID:"123e98cd-1507-49f8-b9bc-a88f68ac420b", APIVersion:"v1", ResourceVersion:"13934661", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "freenas-nfs-csi": rpc error: code = Internal desc = Error: Unmatched '"'.

[baloan]

I am impressed. Changing the TrueNAS root shell to sh resolved the issue.

[travisghansen]

I should say -U2 has a bunch of fixes that generally make the experience better.