Initial GlusterFS native setup fails with Invalid JWT token: signature is invalid

Description

On Gluster install into existing cluster, Verify Heketi fails with a “signature invalid” error.

Version

Please put the following version information in the code block indicated below.

• Your ansible version per ansible --version

2.7.2

If you’re operating from a git clone:

• The output of git describe

openshift-ansible-4.0.0-0.84.0

Steps To Reproduce

1. Create new OKD 3.11 cluster on Fedora 29 using deploy_cluster.yml
2. Attempt to add native glusterfs to this cluster using openshift_storage_glusterfs/config.yml

Expected Results

Install completed.

Observed Results

Install fails with the following error:

TASK [openshift_storage_glusterfs : Verify heketi service] ****************************************************************************************************************************************
Friday 07 December 2018  19:33:00 +0000 (0:00:00.127)       0:00:55.337 *******
fatal: [ip-172-31-1-111.us-west-2.compute.internal]: FAILED! => {"changed": false, "cmd": ["oc", "--config=/tmp/openshift-glusterfs-ansible-FOV42Y/admin.kubeconfig", "rsh", "--namespace=glusterfs", "deploy-heketi-storage-1-bknvx", "heketi-cli", "-s", "http://localhost:8080", "--user", "admin", "--secret", "q4CpOzGTKWT2IF9JKERYzI26ZwnKjnHnYDi6XBHqV3k=", "cluster", "list"], "delta": "0:00:00.327708", "end": "2018-12-07 19:33:00.906016", "msg": "non-zero return code", "rc": 255, "start": "2018-12-07 19:33:00.578308", "stderr": "Error: Invalid JWT token: signature is invalid (client and server secrets may not match)\ncommand terminated with exit code 255", "stderr_lines": ["Error: Invalid JWT token: signature is invalid (client and server secrets may not match)", "command terminated with exit code 255"], "stdout": "", "stdout_lines": []}

Additional Information

I tried the fix described for #10708 but it did not solve the issue.

I also tried adding import_task: get_heketi_key to the start of tasks/heketi_set_cli, and it also did not make a difference.

[giamma]

@fmq delete the ‘app-storage’ project which contains the glusterfs pods:

oc delete project app-storage

[fmq]

@prasenforu What worked for me ( actually deleting the project app-storage as suggested by @giamma also works ) was to execute the GlusterFS uninstall playbook ( playbooks/openshift-glusterfs/uninstall.yml ) with openshift_storage_glusterfs_wipe=true (data WILL be lost).

one more thing check /etc/fstab for any Gluster created disk that where left behind after the uninstall since entires don’t get erased and this causes machine not to boot any more.