"Forbidden!Configured service account doesn't have access" error when open workspace in TLS mode

Describe the bug

In Multiuser Eclipse Che of 7.3.2 version it was impossible to open workspace on OCP 3.11:

Che version

• latest
• nightly
• other: 7.3.2 RC , 7.3.1

It wasn’t reproduced with Eclipse Che 7.5.0-SNAPSHOT

Steps to reproduce

1. Download chectl:next
2. Login to OCP 3.11
3. Install Eclipse Che by command:

chectl server:start -p openshift -a operator --domain=https://console.ocp311.crw:8443 --che-operator-cr-yaml=org_v1_che_cr.yaml -n che-tls

org_v1_che_cr.yaml: https://gist.github.com/dmytro-ndp/8d5fcdc1dfd6dd88bb0d98b0bf7d9adb

1. Go to installed Eclipse Che and try to create and open default workspaces. See error:

Error: Failed to start the workspace: "Failure executing: POST at: https://172.30.0.1/apis/project.openshift.io/v1/projectrequests . Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. projectrequests.project.openshift.io is forbidden: User "system:serviceaccount:che:che" cannot create projectrequests.project.openshift.io at the cluster scope: no RBAC policy matched. The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password.

che server log

2019-11-15 13:28:39,136[nio-8080-exec-1]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 560]    - Workspace 'admin/wksp-o3um' with id 'workspaceozmb5drcndo4ez88' created by user 'admin'
2019-11-15 13:28:42,263[nio-8080-exec-7]  [ERROR] [o.e.c.w.i.o.p.OpenShiftProject 123]  - Unable to create new OpenShift project due to lack of permissions.HINT: When using workspace project name placeholders, os-oauth or service account with more lenient permissions (cluster-admin) must be used.
2019-11-15 13:28:42,267[nio-8080-exec-7]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 447]   - Failure executing: POST at: https://172.30.0.1/apis/project.openshift.io/v1/projectrequests . Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. projectrequests.project.openshift.io is forbidden: User "system:serviceaccount:che:che" cannot create projectrequests.project.openshift.io at the cluster scope: no RBAC policy matched. The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password.
org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInfrastructureException: Failure executing: POST at: https://172.30.0.1/apis/project.openshift.io/v1/projectrequests . Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. projectrequests.project.openshift.io is forbidden: User "system:serviceaccount:che:che" cannot create projectrequests.project.openshift.io at the cluster scope: no RBAC policy matched. The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password.
    at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.create(OpenShiftProject.java:127)
    at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.prepare(OpenShiftProject.java:92)
    at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProjectFactory.create(OpenShiftProjectFactory.java:92)
    at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:69)
    at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:31)
    at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startAsync(WorkspaceRuntimes.java:419)
    at org.eclipse.che.api.workspace.server.WorkspaceManager.startAsync(WorkspaceManager.java:443)
    at org.eclipse.che.api.workspace.server.WorkspaceManager.startWorkspace(WorkspaceManager.java:366)
    at org.eclipse.che.multiuser.resource.api.workspace.LimitsCheckingWorkspaceManager.startWorkspace(LimitsCheckingWorkspaceManager.java:132)
at org.eclipse.che.api.workspace.server.WorkspaceService.startById(WorkspaceService.java:432)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
    at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
    at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
    at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:297)
    at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:233)
    at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
    at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
    at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
    at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
    at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
    at org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:53)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitalizationFilter.doFilter(KeycloakEnvironmentInitalizationFilter.java:151)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.eclipse.che.multiuser.keycloak.server.KeycloakAuthenticationFilter.doFilter(KeycloakAuthenticationFilter.java:70)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:108)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.30.0.1/apis/project.openshift.io/v1/projectrequests . Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. projectrequests.project.openshift.io is forbidden: User "system:serviceaccount:che:che" cannot create projectrequests.project.openshift.io at the cluster scope: no RBAC policy matched.
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:476)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:413)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:227)
    at io.fabric8.openshift.client.dsl.internal.ProjectRequestsOperationImpl.create(ProjectRequestsOperationImpl.java:74)
    at io.fabric8.openshift.client.dsl.internal.ProjectRequestsOperationImpl$1.apply(ProjectRequestsOperationImpl.java:91)
    at io.fabric8.openshift.client.dsl.internal.ProjectRequestsOperationImpl$1.apply(ProjectRequestsOperationImpl.java:87)
    at io.fabric8.openshift.api.model.DoneableProjectRequest.done(DoneableProjectRequest.java:27)
    at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.create(OpenShiftProject.java:120)
    ... 59 common frames omitted
2019-11-15 13:31:48,825[nio-8080-exec-9]  [ERROR] [o.e.c.w.i.o.p.OpenShiftProject 123]  - Unable to create new OpenShift project due to lack of permissions.HINT: When using workspace project name placeholders, os-oauth or service account with more lenient permissions (cluster-admin) must be used.
 

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: OCP 3.11 / OCP 4.2

Installation method

• chectl:next
• che-operator
• minishift-addon
• I don’t know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

[rhopp]

I’ve just deployed Che using chectl 7.3.2 on OCP 4.2.4 ./chectl/bin/chectl server:start -a operator -p openshift --che-operator-cr-yaml=custom-resource.yaml -n rhopp-che732 where custom-resource.yaml is from the original post from Dmytro.

Workspace started fine for me (I’ve tried the Java-maven one)

[dmytro-ndp]

Looks like that problem caused by wrong version of chectl:next and operator image which were used. So, is closing the issue as being not reproduced with proper installers.