Frequent and inconsistent token expired error

Summary

In about 9 out of 10 times, a serverless command results in a The access token has expired. Make sure you can access your cluster and try again. When I just run the same serverless deploy -v over and over again, eventually it works.

I’m trying to encourage my team to use serverless + kubeless, but this error is preventing me from being productive. This may be related to #89, but running a kubectl command to refresh the token doesn’t seem to have any affect. My only option is to submit the same command again and again…

Error

$ SLS_DEBUG="*" serverless deploy -v
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command info
Serverless: Load command logs
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Packaging service...
Serverless: Excluding development dependencies...

  Error --------------------------------------------------

  The access token has expired. Make sure you can access your cluster and try again

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: The access token has expired. Make sure you can access your cluster and try again
    at getToken (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/serverless-kubeless/lib/helpers.js:113:13)
    at Object.getConnectionOptions (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/serverless-kubeless/lib/helpers.js:147:17)
    at new CRD (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/serverless-kubeless/lib/crd.js:30:17)
    at deployFunction (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/serverless-kubeless/lib/deploy.js:416:24)
    at _.each (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/serverless-kubeless/lib/deploy.js:555:9)
    at arrayEach (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/lodash/lodash.js:516:11)
    at Function.forEach (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/lodash/lodash.js:9344:14)
    at BbPromise (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/serverless-kubeless/lib/deploy.js:552:7)
    at deploy (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/serverless-kubeless/lib/deploy.js:551:10)
    at BbPromise.then (/home/mpalumbo7/code/example-repo/functions/hello-world/node_modules/serverless-kubeless/deploy/kubelessDeploy.js:144:19)
From previous event:
    at PluginManager.invoke (/home/mpalumbo7/.npm-global/lib/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.run (/home/mpalumbo7/.npm-global/lib/node_modules/serverless/lib/classes/PluginManager.js:422:17)
    at variables.populateService.then.then (/home/mpalumbo7/.npm-global/lib/node_modules/serverless/lib/Serverless.js:111:33)
    at processImmediate (timers.js:632:19)
    at process.topLevelDomainCallback (domain.js:120:23)
From previous event:
    at Serverless.run (/home/mpalumbo7/.npm-global/lib/node_modules/serverless/lib/Serverless.js:98:6)
    at serverless.init.then (/home/mpalumbo7/.npm-global/lib/node_modules/serverless/bin/serverless:43:28)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           11.6.0
     Serverless Version:     1.36.3

Thoughts

Looking at the code in ./lib/helpers.js I wonder if it has something to do with using the azure auth-provider?

Line 111 begins:

 // Access tokens may expire so we better check the expire date
    const expiry = moment(userInfo.user['auth-provider'].config.expiry);
    if (expiry < moment()) {
      throw new Error(
        'The access token has expired. Make sure you can access your cluster and try again'
      );
    }

But my redacted kubeconfig shows that I don’t have a expiry field?

users:
- name: my-cluster-admin
  user:
    auth-provider:
      config:
        access-token: <access-token>
        apiserver-id: <apiserver-id>
        client-id: <client-id>
        expires-in: "3600"
        expires-on: "1548412445"
        refresh-token: <token>
        tenant-id: <tenant-id>
      name: azure

[mpalumbo7]

Opened PR #171 with your suggested changes. It works on my environment both with and without an expired token.

The token was valid during the whole deployment, it was just the const expiry = moment(undefined) followed by expiry < moment() would occassionaly return true when there was more than millisecond delay between executing lines.

[andresmgot]

So it seems that we are dealing with two problems. Regarding the missing expiry key in the configuration I would just omit the validation if the expiry info is not available:

+ if(userInfo.user['auth-provider'].config.expiry) {
    // Access tokens may expire so we better check the expire date
    const expiry = moment(userInfo.user['auth-provider'].config.expiry);
    if (expiry < moment()) {
      throw new Error(
        'The access token has expired. Make sure you can access your cluster and try again'
      );
    }
+ }

Then, the second problem is that we are not checking if the ConfigMap that we receive is a valid ConfigMap. To check that I would use the code instead of the status. In the file config.js when we are retrieving the resource we should throw an error if the API returns a failure:

            .on('end', () => {
-              const res = Buffer.concat(data).toString();
-              this.configMag = JSON.parse(res);
-              resolve();
+              const res = JSON.parse(Buffer.concat(data).toString());
+              if(res.code && res.code !== 200) {
+                reject(`Unable to parse ConfigMap. Received ${res.message}`)
+              } else {
+                this.configMap = res;
+                resolve();
+              }
            });

What is not clear to me is if you are still finding the issue that the token expires while you are deploying your functions. Is that still happening after removing the check?