Persistent local storage provisioner not configured properly after deployment

Description

After deployment of OKD 3.11 platform, local storage provisioner is not being properly configured (details below).

If it’s just misconfiguration issue, sorry in advance.

Version

$ ~/openshift-ansible$ ansible --version
ansible 2.6.2
  config file = /home/openshift/openshift-ansible/ansible.cfg
  configured module search path = [u'/home/openshift/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]

$ ~/openshift-ansible$ git describe
openshift-ansible-3.11.45-1-4-gc7ef5eb2e

Steps To Reproduce

1. Deploy OKD platform with hosts.cfg inventory file attached below

Expected Results

• storage classes listed in openshift_persistentlocalstorage_classes created after deployment
• local-volume-provisioner DaemonSet configured to use image set up in openshift_persistentlocalstorage_provisionner_image variable
• local-volume-provisioner-config config map configured properly (according to OKD docs )

# Example local-volume-provisioner-config definition
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-volume-provisioner-config
  labels:
    infrastructure: platform
data:
    storageClassMap: |
        local-hdd:
            hostDir: /mnt/local-storage/local-hdd
            mountDir: /mnt/local-storage/local-hdd

Observed Results

• local-volume-provisioner-config config map do not have proper structure (according to docs )
• local-volume-provisioner DeamonSet configured with default uay.io/external_storage/local-volume-provisioner:v1.0.1 although openshift_persistentlocalstorage_provisionner_image has been set up to newer version
• no storage classes are being created

# local-volume-provisioner-config config map create after deployment
apiVersion: v1
data:
  local-hdd: '{ "hostDir": "/mnt/local-storage/local-hdd", "mountDir" : "/mnt/local-storage/local-hdd"
    }'
kind: ConfigMap
metadata:
  creationTimestamp: 2018-11-28T14:29:05Z
  name: local-volume-provisioner-config
  namespace: local-storage
  resourceVersion: "2331"
  selfLink: /api/v1/namespaces/local-storage/configmaps/local-volume-provisioner-config
  uid: f59059bd-f319-11e8-ae62-42010a840026

# Provisioners logs errors straight after deployment:
# oc local-volume-provisioner-2297f 
I1128 14:44:16.768596       1 discovery.go:141] Found new volume at host path "/mnt/local-storage/local-hdd/small_1" with capacity 5358223360, creating Local PV "local-pv-68b38aa8"
E1128 14:44:16.769868       1 discovery.go:153] Error creating PV "local-pv-68b38aa8" for volume at "/mnt/local-storage/local-hdd/small_1": PersistentVolume "local-pv-68b38aa8" is invalid: metadata.annotations: Required value: Local volume requires node affinity

# Provisioners logs errors straight after changing image to version to 2.2.0:
# oc logs local-volume-provisioner-24psr 
I1128 14:47:10.207714       1 main.go:60] Loaded configuration: {StorageClassConfig:map[] NodeLabelsForPV:[] UseAlphaAPI:false UseJobForCleaning:false MinResyncPeriod:{Duration:5m0s} UseNodeNameOnly:false}
I1128 14:47:10.207832       1 main.go:61] Ready to run...
W1128 14:47:10.207841       1 main.go:76] JOB_CONTAINER_IMAGE environment variable not set.
I1128 14:47:10.208056       1 common.go:352] Creating client using in-cluster config
I1128 14:47:10.224929       1 main.go:82] Starting controller
I1128 14:47:10.224985       1 main.go:96] Starting metrics server at :8080
I1128 14:47:10.225477       1 controller.go:45] Initializing volume cache
I1128 14:47:10.429780       1 controller.go:108] Controller started

# No PVs are being created because of wrong local-volume-provisioner-config config map
# After changing config map to example from Expected behavior section all PVs are being created properly

Additional Information

Provide any additional information which may help us diagnose the issue.

• Operating System:
  • Ansible control-host: Ubuntu 18.04.1 LTS
  • OKD nodes: CentOS Linux release 7.5.1804 (Core)

# Configuration of inventory file related to Persistent local storage (full inventory file in attachments)
openshift_persistentlocalstorage_enabled=True
openshift_persistentlocalstorage_classes=['local-hdd']
openshift_persistentlocalstorage_path=/mnt/local-storage
openshift_persistentlocalstorage_provisionner_image=quay.io/external_storage/local-volume-provisioner:v2.2.0

# After changing version of image in DeamonSet, creating storage class manually and applying config-map from Expected results section,  provisioner is working as expected
# oc logs local-volume-provisioner-47gm9
I1128 14:53:59.186275       1 discovery.go:298] Found new volume of volumeMode "Filesystem" at host path "/mnt/local-storage/local-hdd/large_1" with capacity 32196526080, creating Local PV "local-pv-21f259b0"
I1128 14:53:59.200721       1 discovery.go:326] Created PV "local-pv-21f259b0" for volume at "/mnt/local-storage/local-hdd/large_1"
I1128 14:53:59.200853       1 discovery.go:298] Found new volume of volumeMode "Filesystem" at host path "/mnt/local-storage/local-hdd/large_2" with capacity 32196526080, creating Local PV "local-pv-410529eb"
I1128 14:53:59.201208       1 cache.go:55] Added pv "local-pv-21f259b0" to cache
I1128 14:53:59.208146       1 discovery.go:326] Created PV "local-pv-410529eb" for volume at "/mnt/local-storage/local-hdd/large_2"
...

hosts.cfg

[leoluk]

I also stumbled upon this - looks like persistent local volumes are completely broken in 1.11.

The v1 provisioner does not know about the >=1.10 nodeAffinity API, and the v2 API is unsupported by openshift-ansible. The proper fix will be to support the v2 config. Here’s the helm template:

https://github.com/kubernetes-incubator/external-storage/blob/master/local-volume/helm/provisioner/templates/provisioner.yaml

Example PV with node affinity (if you want to create it manually):

https://github.com/kubernetes-incubator/external-storage/blob/master/local-volume/examples/manual-pv.yaml

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.